Hacker News Reader

Comment by hug | Hacker News Reader

Yep. It's worse when working with vendors, and even worse when working with vendors in a Windows-centric environment.

"Our application needs to run as domain admin". No, mate, it absolutely does not. "Well it definitely needs local admin". No, mate, it doesn't even need that. What it usually needs is write-access to a single folder or regkey somewhere that has default ACLs on it that only allow local admins, but trying to have that conversation is a waste of time.

Of course as soon as you say something like that, all support bets are off. Depending on the business you're working for you either suck it up so you have a 'support agreement' in place, or you do it properly and never talk to the vendor again. (Or option #3, you rebuild the app in its default "holy shit this is how my business gets owned" mode, place a support call, and have them fix that instance while you watch so you can replicate their efforts elsewhere.)

Most of the same set of vendors get their panties in a twist when you tell them their database is on RDS, too, but that's a whole different argument to have with these people.

View on HN

I've had to support an app that:

1) only ran in MSIE

2) required the domain policy to permit VBScript

3) required permission to launch + invoke local COM/applications

4) served over http://

All just so they could direct MSWord to directly print the server-side generated Sales Order template.

I audibly gasped and asked if they knew this put our systems at extreme risk, and the tech talking me through it basically shrugged "it's how it works"

Thankfully that webapp 'upgraded' years ago so that's no longer how it's done, but when it's the only way to get orders into your business from one of your top 10 customers, you just setup a workstation to do it :(

> Yep. It's worse when working with vendors

20 years in the vendor side, and I can count on one hand the number of times a customer followed the recommended process to create a limited user for one of our AD services. Every other time - just run it as an admin service account, and if they’re feeling motivated we might get a dedicated admin account created.

It’s very frustrating…

Cant you just read the files needes then drop privileges ?

I don't believe you can do that in Windows; your process is bound to the SID it started with.

There is `AdjustTokenPrivileges` and `CreateRestrictedToken` which can accomplish something like that.

> It's worse when working with vendors

Well this works the other way around too. I found this especially true when delivering software which is considered an interim solution.

More than once I got root privileges on customer machines or seen my software run as root because the project owner found it too cumbersome to deal with his own employers security regulations. Or just wasn't given the budget to deal with it properly. Sometimes it's easier to get an exception than to follow protocol.

This is coming back to haunt you if something goes wrong or an interim solution wasn't so interim after all. If someone responsible for auditing permissions asks at this later stage it was always the vendors fault.

Some software refuses to run as root. It's far from perfect, and usually easily circumventable, but it might push a lazy admin to at least create a user for your thing or similar.

I hate that as a solution. I've had strange issues with software that I couldn't debug, but when I ran it as root in a test VM, it worked instantly. So I immediately knew it was a permission issue and started digging through the docs for a list of files it needed access to. I then fixed the permissions for it and ran it without root in production.

If the software limited me to not running as root, I would have probably spent far longer debugging other issues and digging through 100s of lines of strace output.

I think all software should come with the --räikkönen[0] --here-there-be-dragons --yes-i-mean-it set of switches to enable you to do boneheaded stuff when needed.

And using them in prod should be grounds for dismissal, or at least pretty severe legal liability if things go wrong as a result.

[0] https://m.youtube.com/watch?v=F3d_Cu5Mzbk

I really like that idea, actually. Have a blanket "no safety checks whatsoever" flag, but in the ToS it invalidates all warranty and support requirements if in use. This would also make it really obvious in court who was negligent in case of a security breach, since something like "running as root is bad practice" is a lot less clear to a non-technical judge than "they explicitly enabled a flag called --i-know-this-is-insecure-do-it-anyways", even though the issue (running as root) was the same.

I honestly wonder who can use these applications these days?

Large companies have stopped giving administrator access years ago. The vendor could corrupt their way into some middle manager all they want, they have zero power over procurement and IT, users won't get administrator access to use the app.

If the app is really unusable, which it is, that must be preventing a whole lot of sales.

Microsoft provides toolkits to help with this. I have spent easily 100+ hours churning through a catalogue of applications that 'need local admin' and have only found one case where it was true, and it was due to a bug that the latest software version fixed. It is almost always a false truth due to cost I feel. I primarily analyzed applications that were for PLCS and other industrial equipment.

You can find more information about this here: https://docs.microsoft.com/en-us/windows/deployment/planning...

I run a VM that stays up to date with the latest Windows version that I install the toolkit on for testing.

(side tangent, the user of this application insisted they had the latest version, and with it being behind some login that required some form related to ITAR or something to register, we just had to trust him until I had him download it 'just in case.')

Great timing. I'm looking for solutions to run Siemens Simatic PDM without being a local admin. Installing it with temporary admin rights seems to silently fail.

I'm also trying to use libpcap and USBPcap without local admin. They request UAC every time I open Wireshark to start a new capture session. (Needed for debugging raw Ethernet and serial stacks)

If you have any tips for either of these I am all ears!

Use npcap and check the settings for the admin-not-required button (forgot the exact name). But generally, accessing raw anything needs elevated rights and should need elevated rights. For a normal user to be able to sniff network and usb traffic is usually a huge security hole and therefore correctly locked away. Not at all comparable to the other cases someone discussed above about some crappy app needing admin to write c:\windows\crappyapp.ico...

> Large companies have stopped giving administrator access years ago.

This is untrue for low-end enterprise. Orgs with up to 10,000 seats run weird shitty local-admin-required products all over the place, usually due to legacy.

In consulting, I would regularly run into core LOB apps that not only required local admin but "couldn't" run as a Windows service and required a user to be logged on with the UI window open in order to function.

You spend hours using the associated Microsoft toolkit & various shims to make the product function without, but there's always another business who just does it the lazy way.

I've worked for two of the FAANG companies together about 15 years, and I've always had local administrator access and root access to (1) my personal client machines (laptop, desktop on all platforms) (2) my personal development machines on the corporate network (3) my teams' production machines.

I'm a software developer and wouldn't have it any other way. Software developers need root/admin access if you expect any development velocity and ability to solve problems in production with expedience. There are too many diagnostic operations that depend on having root/admin, and especially fixes to errors.

One exception might be if the production fleet is handled by a separate operations team – but I'm not in favor of this model, which has its own negative impact on velocity/delivery speed due to the coordination involved, and the tension between "software developers want to ship" and "operations team wants software to remain unchanging and stable".

I'm not in favor of operations teams until the software team is overwhelmed with support requests and their velocity is slowing down. In that case, I'd start with beefing up the support team; and then finally at tremendous scale with high operational burden, hire an operations team / SRE team to operate the system.

The solution is to make software responsible for owning/operating their own systems until they are extremely large scale (Amazon S3 scale) – as scale so large that rare spontaneous faults would eat up a significant percentage of the overall software team's time. I'm not opposed to having frontline customer support in front of the developers, potentially multiple tiers (since a lot of customers need to Read The Fine Manual); but once the escalation hits a developer, to expeditiously solve the problem they need root/admin.

Your software developers with root/admin access should still be logged such that central security team can observe what's going on on every machine at the company, via configuration of SSHD, and regular inspection of that config – e.g. a shell that logs off-box command execution in real-time by default, either for all commands or commands root; and this will capture what they're doing unless they go very far out of the way to conceal and obfuscate their activities. Even then, the initial steps to begin the concealment will show up in the logs, unless very well obfuscated.)

Lastly, if your team runs public web servers, then you need to listen on ports like 80 and 443 while requiring root. Well-written servers drop root immediately after opening the socket, but custom written servers may run on those ports and not drop capabilities and run in a container. For example, Faceook Messenger Platform's onboarding instructions and example code –https://developers.facebook.com/docs/messenger-platform/gett... – recommends binding a TCP port >1024:

``` app.listen(process.env.PORT || 1337, () => console.log('webhook is listening')); ```

Aside: Nice and professional port number there. If I'd written that documentation I would have used 8081 or 8443, and included a discussion of the implications of running a web server on HTTPS (443) or HTTP (80).

I understand that having software developers having root/admin on their machine is very different than non-tech jobs, but for any jobs in the engineering family it's essential. The builders and operators of a system should always have root access to it, because (1) it's needed for development throughput (2) they can trivially get root access anyway through a variety of different means, such as incorporating a grant of root access into the code they're shipping, with minor obfuscation.

At my last company I was also involved in a major effort to make easier to not run software as root. However, for "typical" services running on single-purpose machines or Virtual Machines that don't do anything but receive or process commands from any other software, and don't run any other software (besides company Infra), I think the risk is low. "root" and "nobody" are marginally different on a machine that runs a single application because an attacker who breaks in can access all of its data anyway.

It's true that with root an attacker can implant rootkits and more firmly maintain a grasp of the server, but competent FAANG companies (1) require MFA in order to log in to servers besides the initial hardware client in front of the user (2) will audit machines for unexpected changes like what I'm describing above. I'm guessing only highly sophisticated attackers like APTs would be able to conceal from those scans, and there will always be the record of initial penetration to review. Incident response will quickly fan out to all machines the pwned machines has communicated with. One of the FAANGs had complete network logs for machines like this and would have been able to inspect the network traffic.

Lastly, sophisticated corps employ defense-in-depth and multiple layers of redundancy like (1) you can't attempt to log into machines on the network without both being on the VPN and having authenticated certificates from the enterprise root authority, and (2) Bastion machines which can log commands being sent to machines even if the machines themselves don't, and so on.

I disagree with most of this, to be perfectly honest.

For one, devs don't need root in prod in the vast majority of shops, which aren't FAANG scale. Assuming a nice sane environment where your deploys to pre-prod mirror your prod environment, I can't think of many cases where having root on prod makes much difference. It's the inverse of your statement, where places that aren't massive should find it feasible to 1:1 their prod environment for the devs to noodle about in.

For two, statements like this one make me wary of a blanket policy of devs getting root:

> if your team runs public web servers, then you need to listen on ports like 80 and 443 while requiring root.

Have you heard of setcap and CAP_NET_BIND_SERVICE, perhaps? I know this is only one example of many where you may "need" root, but in a lot of cases devs just don't have the ops knowledge to know what they actually need.

Yes, I am aware of these capabilities. However, you as the developer still need to be able to write software that runs as root on the box, both to (likely) install new software version and to drop the capability either shortly before executing the primary application (preferred); or the application drops it shortly after launching (seems to be more common – e.g. Postfix Architecture [1] – probably because the App knows what minimal permissions it needs and this is less work for the user).

Meanwhile, as a developer you still need the ability to deploy applications unless some kind of "deployment layer" handles depositing new software versions onto the machine.

I'm also concerned about the lack of root for being able to inspect what's going wrong with software on the machine (`ps` and `lsof` and socket-specific equivalents). How do I take a heap dump or stack trace from erroneous software on a prod machine without root? How can I step through the software in a debugger? (You could argue that I shouldn't do this but it's a tradeoff between velocity and security, the right answer which may depend on the business we're in.)

Containers are kind-of this but I wouldn't use containers as security sandbox, since they're not designed for it. User namespaces get close, but I prefer to rely on the hypervisor for this kind of security boundary, not namespaces, cgroups, or containers – because the kernel API is massive and security vulnerabilities are found in it regularly. (Root exploits less often.)

How would you build an operate a server in production without root? I honestly don't have experience. All of my experience has been running installation scripts as root, and having personal root access for inspecting/debugging (potentially even live debugger attaching and stepping); unless the `sudo` commands were very carefully managed I wouldn't be able to use the majority of my standard toolkit for fixing problems, and I wouldn't be able to fix the problem if I found them.

Am I running out of file descriptors because the limit is set too low? OK, bump up the limit. That's a code change but I can initiate an emergency recovery by locally changing it on all machines in the fleet.

We should also acknowledge that there are different kinds of software systems. If I was working at Coinbase or on Payments at Stripe I might have a different perspective. I previously worked at AWS and now currently work on WhatsApp so I am still incredibly security-conscious, but the level of security required by a product is a negotiable product specification, not a hard-and-fast-rule like the necessary strength of a bridge in civil engineering.

[1] Postfix Architecture: http://www.postfix.org/OVERVIEW.html

>We should also acknowledge that there are different kinds of software systems. If I was working at Coinbase or on Payments at Stripe I might have a different perspective.

It all boils down to this at the end of the day. There is clearly some level of tradeoff between control and velocity, and there are valid business cases for setting the dial just about anywhere on the spectrum.

As a developer, you should want as little power on as few servers as needed to do your work. With the unlimited access to do whatever you want, you also get lots of responsibility, which means you need to develop your security-consciousness to near paranoid levels, and that's not fun.

If it's possible for you to log into a server and just become root (hopefully there's at least sudo so you don't log in directly as root), you have to make sure your personal laptop or workstation is encrypted, your SSH key is passphrase-protected and you use an SSH agent (or even an external physical device) to store your private key. Certainly you should not be using passwords to log in anywhere. You should do these things anyway, but the more direct power you have, the more important it becomes.

This is especially true if you have programmatic access to a public cloud somewhere: you do not want any more power than necessary for any longer than necessary because administrator access to eg. an AWS account means that when it leaks your infra can be automatically compromised in less time that you can react in and you can't even put firewalls in front because it's accessible from the internet.

Leave the operations that require real power to CI systems and automated processes where you have some testing and review in the middle. Get your logs and memory dumps shipped out of the system into tools that actually help you make use of them; if you need to debug a live system as root, that should be an exceptional situation that raises all kinds of alarms so that it can be formally accepted.

I'm a guy whose job involves maintaining hundreds of systems and I'm always happy when I can do it with less power.

> (hopefully there's at least sudo so you don't log in directly as root

Correct, at both FAANG I logged in as myself and had passwordless sudo. (Passwordless sudo being preferable to minimize the number of machines that have user password hashes at the company, since these can be brute-forced when employees set bad passwords).

> your SSH key is passphrase-protected and you use an SSH agent

I don't have long-lived SSH keys. The FAANGs where I've work use short-lived (~24h) SSH client certificates, issued by Single Sign On and a server-trusted SSH Certificate Authority, where the SSO authenticates the user via an MFA (typically FIDO2 or variants with a Yubikey press). (Yes, you can indeed employ CAs with SSH, and generate short-lived certificates [1]). I don't generally type my password after signing into my client machine, or possibly once more to authenticate into the corporate web framework (w/ MFA).

After that, every SSH action requires a individual MFA approval. It doesn't require a password or any SSH key protected by a password – I think long-lived SSH keys, like you'd protect by a password, are an anti-pattern because when employees leave the company or change teams you have to remember to revoke them or update their keys. With the SSH CA approach, you simply change which certs it's willing to issue to an employee for which servers based on policy; and then people who leave the company can't access that system altogether. You get the right behavior by default without anyone remembering to take action during off-boarding.

No long-lived SSH keys with passwords. Short-lived SSH certificates don't need their own passwords.

As a developer operating systems on a suite of servers, it's simply a business need for me to have SSH into the (virtual machines that I own) running on those servers (not necessarily the underlying hypervisors – those are an entirely different beast).

If you were in the cloud it's equivalent to being able to obtain SSH certificate that allows you to log onto e.g. EC2 instances. Or if I'm a team that's operating Bare Metal instances, I can log into those.

I don't accept that root is less than necessary to do my job. When things go wrong there are too many directions that debugging can go that you cannot explore without root. Like I mentioned, I can't take a machine out of service, and attach a debugger to it, and invoke the failing functionality. I can't change machine parameters that may fix the problem like memory limits. You can't even examine the contents of a number of crucial logs without it.

I agree with the principle of least access and employ it in my personal systems as well. At work, I think POLA results in you having root to every machine you're responsible for.

[1] https://engineering.fb.com/2016/09/12/security/scalable-and-... (Disclosure: the FAANG I currently work for is Facebook)

SSH certificates are a good solution, though I think you could still use longer-lived SSH identities as part of the authentication process to the system that signs your certificate with MFA on top. I personally really just don't like passwords for authentication at all, and SSH is super convenient to use on top of being secure.

Having a solution like that in place goes a long way towards reducing the immediate power you have available at any one time, so I don't think we disagree much at all.

That is the whole point of container based distributions like Bottlerocket.

Also even for Windows, in cloud deployment the only admin access we tend to have in production are dashboards, no one gets to touch the actual machine.

In enterprise consulting projects, in security conscious customers, I never got Admin account directly, only temporarily via tooling that would request for it with a couple of hours duration, and installing software required IT tickets.

> For one, devs don't need root in prod in the vast majority of shops, which aren't FAANG scale. Assuming a nice sane environment where your deploys to pre-prod mirror your prod environment, I can't think of many cases where having root on prod makes much difference. It's the inverse of your statement, where places that aren't massive should find it feasible to 1:1 their prod environment for the devs to noodle about in.

You can and should mirror the environment, but it's rarely going to be perfect (particularly in terms of traffic), and while it's often worth putting time into better test environments, being able to just hop onto a prod machine and properly trace your programs' behaviour (which often means you need BPF probes or similar) is a gamechanger for fixing production problems quickly.

> Have you heard of setcap and CAP_NET_BIND_SERVICE, perhaps? I know this is only one example of many where you may "need" root, but in a lot of cases devs just don't have the ops knowledge to know what they actually need.

We absolutely don't. Would learning it actually be a good use of time though? Linux user accounts are not generally seen as a hard security boundary, local root exploits are ten-a-penny, and the user account that's running the service necessarily has full access to the actually important stuff (user data) anyway. Particularly in this age of disposable VMs for single processes, I'd think the cost-benefit favours running everything as root, giving developers root, and structuring your security model so that your servers don't trust each other.

> if your team runs public web servers, then you need to listen on ports like 80 and 443 while requiring root

How common is this case, really? I'd expect most public-facing services, even really high-scale/'special' stuff, to really be instantiated inside a private network (VPC or suchlike, scoped down to just that service if you're lucky) and not internet-accessible, fronted by public-facing load balancers listening on 80/443 that the service team doesn't need or want to control. AWS's ALBs and various equivalents definitely allow for that. In such a situation there's no need for the internal port that your service hosts listen on to match the external port (though having root on those boxes may be desirable for many other reasons).

(of course, if your public web servers are the load balancers themselves, that is different...)

Even in single machine setups, nginx or apache run on ports 80/443 and reverse proxy to the application servers running on unprivileged ports with unprivileged user ids. It's even easier than having to deploy applications as root (ssh often is not configured to allow root logins.)

I'd even argue that you always always want to use some common webserver or proxy (nginx, apache, haproxy, ...) on 80/443 proxying to the application running on 127.0.0.1:8080. Why? Multiple reasons:

- Unified TLS config and certificate deployment to your chosen common webserver. Never again having to fiddle with yet another crappy vendor configuration not knowing about anything beyond DES, Java crapping itself with larger certificates, vendors not patching TLS library problems for ages.

- Unified log format: Vendors never stick to a proper format, but everyone can read and parse apache logs. Also, easier handling through unified output to syslog/logstash/whatever.

- Unified authentication: No need to wait for all your vendors to support your orgs chosen SSO scheme. Just let your common webserver handle things and pass on some 'X-Authenticated-User: joe'. Most vendors do handle those.

- Additional authentication: Afraid of pre-auth exploits in a vendor's crappy application or whatever SQL injection in the login form some internal department concocted? Just have the proxy do an additional authentication step (e.g. negotiate/kerberos/AD auth), then unauthenticated users cannot harm you and authenticated evil users will be plainly visible in the logs.

- Fixing crap: Vendor changed the URL scheme of some webapp again? All the links from the internal wiki are broken now? Have the proxy do some regex-rewriting.

As for the various grandparents talking about whatever they did at FAANG: I don't believe it, because the above concept is (almost) the same as what e.g. Google is doing with https://cloud.google.com/beyondcorp

I'd say you don't even need to run your local proxy on 80/443 unless it's your public-facing endpoint. Client -> load balancer:80 -> [proxy:8080 -> application:8085] or something is fine.

> As for the various grandparents talking about whatever they did at FAANG: I don't believe it

I promise you there's a lot of stuff that by-design lives in its own private network (VPC, etc.), and that won't change even with efforts to reduce reliance on corporate intranets. Your laptop gets treated differently from some application server in a cloud provider's datacenter. The latter probably has no reason ever to reach out over the Internet, it has no business even accepting connections from anyone but the (very small) set of allowed clients (probably just your load-balancer layer, plus whatever management stuff). So why allow it?

The only argument I've seen for this is that by knowing your hosts are exposed to the internet, you're forced to build for that high-threat environment, but that doesn't seem to outweigh the additional layer of defense-in-depth.

This is not to say that anyone should use that private networking layer as their only security control. So: build as if you're on the public internet, but then don't be, to the maximum extent possible.

An aside for your aside, in regards of port number: nobody reads documentation for fun. Even a little of tongue-in-cheek like your "unprofessional" example above makes the drudgery bearable ("heh, that's elite in leetspeak, ha ha"). It shows that it's written by humans, for humans, as opposed to oracle-style soulless corporate suck. You may consider it unprofessional, I consider it humanizing.

I once found an easter egg 'kilroy was here' type paragraph buried deep in an otherwise dry slash 'uninteresting to most people' document, so I reached out and told the person who'd left it that I found it. It absolutely made their day. I wouldn't do this very often, but it was smoothly integrated with the doc and anyone skimming would have missed it.

I was responsible for an in house tool, it required local admin to write to the event log I think? I don't even think that was a useful feature, but I never explained why or removed it. Merely demonstrated that debugging crashed out without admin rights.

I managed to have local admin on my box by default for as long as I worked there and it was great.

Downloading and installing a new VS using my own msdn. Halycon days

What is so great about this software that people put up with this kind of crap?

Probably non-technical managers making technical decisions. They run off or fire anyone who has a clue (seen as obstructionists). After that, it's to hell in a hand basket.

Reply on news.ycombinator.com