Hacker News Reader

Personally, I'm not a fan of the answers that amount to a cloud-hosted thin client. I use these at work, they're absolute technological marvels, but they suck.

The real answer is a zero trust network that implements:

- multi factor auth

- deployment approval gates

- end to end service encryption

- ALE for secrets and keys

- password managers

- WireGuard tunneling or equivalent

- read only production environments by default; major levers to pull in order to write

- fully partitioned environments, all of which partitioned away from the corporate network of laptops, printers, and security cameras

> - read only production environments by default; major levers to pull in order to write

Yes. In general, it's a good idea to split state management from business logic.

In the simplest thing, that means that eg you have a database that's separate from the rest of your site. But the principle applies more generally.

Useful for keeping things simple.

To go further: if you want to log something, you send it to a log server that is super simple and can only write to one location. So if someone takes over your business logic service, they can't write arbitrarily.

You've already included the answer - "using their machines as remote terminals, doing most their actual work on some remote server".

The developer uses MFA (TOTP, Push Notification, Yubikey etc) into a virtual desktop inside the organisation (Citrix, VMWare Horizon, etc).

From there, the developer can SSH / whatever into their development environment - which is hosted "inside" the corporate network, or their cloud provider, via internal links.

All code, and dev boxes live "inside" the corporate network, and only keypresses, mouse movement, and screen diffs are sent back and forth.

Most remote access packages can prevent clipboard, USB device, file transfer etc.

If you need a password manager for work purposes, then it lives on the corporate managed network - not your remote laptop/desktop - and to be really paranoid - you only ever "copy/paste" those passwords - you don't type them in.

If you really want to lock it down further, give the remote workers dedicated corporate equipment that they only use to access the remote desktops, so you can prevent some things like screen capturing, and really lock down the software to prevent things like keylogging software/malware.

You also should have the entire development environment segregated from the "business" corporate network as well.

It's only really an issue if you want to have offline developers - in which case I don't have any thoughts ready to hand - (but would expect it to be a very locked down machine, possibly with an even more locked down VM inside it).

As someone who regularly uses multiple layers of Virtual Desktop -> Virtual Desktop -> Remote Desktop, provided the network can handle it (on both your local network, and the corporate network), it works surprisingly well.

> You also should have the entire development environment segregated from the "business" corporate network as well.

This is the key point, otherwise you'll also just get a linux machine with developer root access, but EVEN CLOSER to the corporate network.

For most development work, this would probably cause a serious productivity drop, but it definitely makes sense for the portion of the work that involves accessing critical production resources. For DevOps roles, that could well be the majority.

I'm all in for VM based privilege separation, but that won't protect you from infected endpoint. Assuming this was a targeted attack, folks that achieved RCE on DevOp engineer's machine could have waited for her to authenticate and then inject keystrokes into VM, SSH, VNC, Remote Desktop, Citrix or whatever remote management system they're using.

Honestly, this HN thread is full of bad advice and factually incorrect patronizing. Okta-style system asking to accept every single permission would not have protected from an attack, because Okta caches and reuses authentication tokens. Clipboard snooping / keylogger detection wouldn't have worked because none of these solutions are robust against targeted attacks.

The only thing I can think of which would have (and should have) helped is alert SOC / incident reponse team. Good luck finding one though.

Glad to see someone else with the same reaction, because a lot of this advice is... interesting, like people who are worried about keyloggers but think the clipboard is safe.

This is both a misunderstanding of the problem and an attempt to solve an administrative problem using technology... which cannot really solve it.

Developers have nothing to do with this. It's a common practice in companies that have "expensive" production environment (eg. VMs rented from AWS) that developers never get any kind of access to production environment. Ever. At all. No need to tie developers' hand by putting them behind a ton of unnecessary firewalls. They have no need for the sensitive information and shouldn't be burdened by protecting it.

The few people who do have access to company's "expensive" production environment are / should be very few people, most likely in the infra / DevOps department. These people do need to follow special protocol for communicating with the "expensive" environment, which, likely, doesn't happen all that often. Depends on the product, of course, but unlikely to be more than once a day, or even once a week.

----

PS. In many, many years of being in infra / system / automation I had never typed any passwords for any important services I had to use. They are usually difficult to type due to having all kinds of Unicode characters I wouldn't know how to reproduce w/o a little research. It's also very rare that they end up in system clipboard, since I usually end up using something like vi+tmux over SSH in Emacs' ascii-term to copy the password from somewhere and paste it somewhere else. So, stuff like AWS keys would have to be stolen by taking screenshots of my screen or something like that...

I mean, why on Earth would anyone deploy to production environment from their personal laptop? Normally, deployment is made from some sort of a testing / staging environment where the system was being tested / archived before shipping it to the next stop... It sounds like some kind of emergency / unplanned situation where a DevOps had to log into the remote system from their laptop.

Are you misunderstanding the term "DevOps"? You build it, you run it. If a DevOps team only runs things other developers have build, it is not a DevOps team.

No. I'm not.

There seem to be two competing definitions

In this case, DevOps shouldn't be rearchitecting, developing, or changing a password management's solution, crypto, architechture, or design in any way. Not in the slightest.

In my experience, I find that working with Virtual Desktops is the most frustrating user experience as a developer you could have. I prefer working in containerized environments which are more efficient and do not require the same amount of configuration processes as a Virtual Desktop.

You should check some solutions out.

My company operates in a Windows centric industry and our software team uses it as well.

It turns out you don't need administrative privileges for a lot of dev work (installing and running vs code, python, node, many databases, etc...).

My experience is that sudo apt-get install is a Linux Distro thing, most programs don't need special permissions as long they are installed in user scope.

So, answering your question, our devs are like regular users: when they need to install something that needs privileges they call IT. Surprisingly, that rarely happens.

Privilege escalation on Windows is super easy though, every red teamer I know has a bunch of ready to use exploits (most of them public) up their sleeve. And it is virtually impossible to get a good baseline of a developer's machine, so I'm pretty sure every SOC out there is simply allowlisting huge swaths of your software.

You can sorta kinda harden these systems, but that would only work against common malware. And you generally can't isolate senior engineers in their own little DMZ, so any RAT on their machines usually leads to catastrophic consequences.

Privilege escalation is a red herring. Everything you need to compromise production from a developer PC is either available for a regular user or not available at all.

Anyone with this level of access should know not to run random github projects or npm install in the most sensitive context. The choice is between easy or secure. You can't have both and that's a reality one has to accept. It's not that difficult to spin up a VM when you want to fuck around and isolate it from sensitive data.

Especially as a "DevOps engineer", gatekeeping and providing least privilege access is in the job description. I understand getting lazy and relaxing the rules in some contexts but not when running a password manager on this scale, unacceptable.

Don't do development on any system that has access to production. Develop on dev lane resources.

Production access should only be allowed from a locked down system with no open ports and a very small whitelist set of software. Operations for said system should be simple. Deploy version x with necessary provisioning. Backup system. Restore system. View monitoring and logs.

You must minimize the surface area connected to production.

On the network side the system with the keys for production is also firewalled off from general Internet access. So potential malware can't phone home.

Give you a MacBook and a cloud Linux dev machine (or as many as you want).

Secure access to dev environment with multiple easy factors - MBP biometrics, yubikey, etc

Enforce strong password and FDE on the laptop.

Obviously doesn’t need to be a MacBook, but you stated an aversion to overly locked down windows laptops.

In general, just enforce strong multifactor creds when accessing internal resources.

You can also enforce device very requirements, so a stole device or yubi can’t be used either.

Then do what you want on your Linux instances in the cloud. They’re protected with their own controls.