Hacker News Reader

Comment by nathan_compton | Hacker News Reader

I never, ever, do development outside of a podman container these days. Basically if I am going to run some code from somewhere and I haven't read it, it goes in a container.

I know its not foolproof, but I can't believe how often people run code they haven't read where it can make a huge mess, steal secrets, etc. I'll probably get owned someday, I'm sure, but this feels like a bare minimum.

View on HN

Probably because it’s fine 99.99% of the time and humans aren’t intuitively good at handling risk that functions like that. Besides, security is something handed off to specialists to free the devs up to focus on building things in most companies. We’re not going to change that no matter how much it represents some ideal.

It's like seatbelts/car seats.

Indeed!

> if I am going to run some code from somewhere and I haven't read it, it goes in a container

How does this work? Every single npm package has tons of dependency tree nodes

Everything runs in the container and cannot escape it. Its like a sandbox.

You have to make sure you're not putting any secrets in the container environment.

All right then, keep your secrets.

>You have to make sure you're not putting any secrets in the container environment.

How does this work exactly? containers still need env vars and access to databases and cloud environments. Without these the container is just useless isolated pod.

Maybe don't use JavaScript on the backend.

Not who you asked, but I have a similar setup. I can run everything I need for local development in that image (db, message queue emulator, cache, other services). So, setting things like environment variables or running postgres work the same as they do outside the container.

The image itself isn't the same image that the app gets deployed in, but is a portable dev environment with everything needed to build and run my apps baked in.

This comes with some nice side effects like being able to instantly spin up clean work environments on my laptop, someone elses, or a remote vm.

This really depends on your setup. If possible, I have local development containers as much as possible. nginx, postgres, redis, etc. I have several containers, each only has access to what it needs. We have an isolated cloud environment for development, in its own aws account.

Its not going to stop attacks, but it will limit blast radius a lot.

You are just reducing the blast radius with use of podman; you will likely need secrets for your app to work, which will be exposed regardless of the podman approach.

Most people don’t have NPM keys in their application containers.

If you're developing in a container then you would have to be doing it without doing something like say, mounting your home directory into it.

The reality here is this is the sort of attack SELinux should be good at stopping (it's not because no one uses SELinux, the policies most commonly used don't confine the user profile in a useful way, and a whole bunch of tools love ambient credentials in environment variables).

No it is not.

I didn't read this as separate containers.

How are you doing this in practice? These are npm packages. I don't see how could reasonably pull in Posthog's SDK in a container.

What do you mean? You can drop into bash in a container and run any arbitrary command, so `npm install foo` works just fine. Why would posthog's SDK be a special case?

I think the issue is more about what else has to go into or be connected to that container. Posthog isn't really useful if it's air-gapped. You're going to give it keys to access all kinds of juicy databases and analytics, and those NPM tokens, AWS/GCP/Azure credentials, and environment variables are exactly what it exfiltrates.

I don't run much on the root OS of my dev machine, basically everything is in a container or VM of some kind, but that's more so that I can reproduce my environment by copying a VMDK than in an effort to limit what the container can do to itself and data it has access to. Yeah, even with root access to a VM guest, an attacker they won't get my password manager, personal credit card, socials, etc. that I only use from the host OS... But they'll get everything that the container contains or has access to, which is often a lot of data!

You're severely limiting the blast radius. This malware works by exfiltrating secrets during installation, if I understood it correctly. If you would properly containerize your app and limit permissions to what is absolutely required, you could be compromised and still suffer little to no consequences.

Of course, this is not a real defense on its own, its just good practice to limit blast radius, much like not giving everybody admin rights.

> Upon execution, the malware downloads and runs TruffleHog to scan the local machine, stealing sensitive information such as NPM Tokens, AWS/GCP/Azure credentials, and environment variables.

Even a properly containerized app will still have these things, because you need things like environment variables (that contain passwords, api keys, etc) for your app to function.

Sure, but only the container is affected and it is always your responsibility to grant as little access as possible to the various credentials you may need to supply that environment. AFAICT with this worm, if you don't supply write-level GitHub credentials to the container (and you shouldn't!) and you install infected packages, the exploit goes no further.

> if you don't supply write-level GitHub credentials to the container (and you shouldn't!)

Really? What if your application needs to write to Github?

You could create/run thin proxies for every external service that handle the auth side, and run each in a separate container. Orchestrate everything with docker-compose. Need to connect to cloud services for local development? Have a container with a proxy that transparently handles authentication. Now only that container has the secrets for talking to that service.

That's a lot of work though, and increases the difference between your local dev environment and prod.

Because you need your application code to interact with Posthog's code. But if they're running in separate containers...how are you doing that. Surely you are not writing an api layer for every npm dependency you use.

This is smart and a good first step. Everyone can't be trusted to do the security dance flawlessly, though. We need sane defaults. Least privilege by default for 3rd-party code. Deno's headed in the right direction with this. But I think the solution needs to exist deeper in the stack. The surge in popularity of `curl -fsSL https://my-cool-ai-starup.ai/install.sh | bash` style installers is particularly concerning to me in this regard.

I ssh into a second local user and do development there instead with tmux.

I send mail to a demon which runs MsBuild and mails the output back to me.

You could still leak API keys

The same podman that had three new CVE breakouts not even two weeks ago?

Containers do not contain.

Security is oft done in layers. Podman is just another layer that might fail. There's no such thing as the perfect solution.

Another effective strategy I learned of recently that seems like it would have avoided this is to wait months before using new versions of packages.

Most attacks on popular packages last at most a few months before detection.

Using Podman over Docker is probably an even safer bet in that regard. But QEMU or something for an extra layer of safety and paranoia is probably the next best thing.

Would it have prevented this attack? It would still have published the secrets from your container to github.

the whole point of doing such things in ephemeral containers is to not have your secrets accessible from said container

Reply on news.ycombinator.com