I think it's clear that some customers env vars got exposed, so that can only mean unencrypted, right?
If you spin up an EC2 instance with an ftp server and check the "Encrypt my EBS volume" checkbox, all those files are 'encrypted at rest', but if your ftp password is 'admin/admin', your files will be exposed in plaintext quite quickly.
Vercel's backend is of course able to decrypt them too (or else it couldn't run your app for you), and so the attacker was able to view them, and presumably some other control on the backend made it so the sensitive ones can end up in your app, but can't be seen in whatever employee-only interface the attacker was viewing.
For non-sensitive environment variables, they also show you the value in the dashboard so you can check and edit them later.
Things like 'NODE_ENV=production' vs 'NODE_ENV=development' is probably something the user wants to see, so that's another argument for letting the backend decrypt and display those values even ignoring the "running your app" part.
You're welcome to add an input that goes straight to '/dev/null' if you want, but it's not exactly a useful feature.
Piping to /dev/null is of course pointless.
What you really want is the /dev/null as a Service Enterprise plan for $500/month with its High Availability devnull Cluster ;)
What's best practice to handle env vars? How do poeple handle them "securely" without it just being security theater? What tools and workflows are people using?
However I do feel now like my sensitive things are better off deployed on a VPS where someone would need a ssh exploit to come at me.
Notice how their tutorial says "run 'dotenvx run -- yourapp'". If you did 'dotenvx run -- env', all your secrets would be printed right there in plaintext, at runtime, since they're just encrypted at rest.
The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend (the '.env.keys' file by default in dotenvx) used to show them in the frontend and decrypt them for running apps.
Same for sops.
> The equivalent in vercel would be encrypted in the database (the encrypted '.env' file), with a decryption key in the backend
The encrypted .env file is actually committed to source code, and the decryption key is placed in Vercel's environment variables dashboard. The attacker only gained access to the latter here if using dotenvx so they can't get your secrets. Unless they also gained access to the codebase in which they have terabytes of data to go through and match up private keys from the database with encrypted .env files from the source code exfiltration - much more effort for attackers.
There is no silver bullet, but Dotenvx splits your secrets into two separate locations.
1. The private decryption key - which lives on Vercel in this example 2. The encrypted .env file which lives in your source code pushed to Vercel
Attackers only got access to the first (as far as I know was reported). So your secrets would be safe in this attack if using Dotenvx. (A private key is useless without its corresponding encrypted .env file. Attackers need both.)
The whitepaper goes into the problem and solution in more detail: https://dotenvx.com/whitepaper.pdf
The point of encryption is often times about what other software or hardware attacks are minimized or eliminated.
However, if someone figures out access to a running system, theres really no way to both allow an app to run and keep everything encrypted. It certainly is possible, like the way keepass encrypts items in memory, but if an attacker has root on a server, they just wait for it to be accessed if not outright find the key that encrypted it.
This is to say, 99.9% of the apps and these platforms arn't secure against this type of low level intrusion.
It always comes back round to "you can't have your cake and eat it".
You can, theoretically, decompile the system memory dump and try to mine the credentials out of the credential server's heap, but that exploit is exponentially more difficult to do that a simple `cat /proc/1234/environ`.
You can do service attestation securely, even for networked services.
Various certifications require this, I guess because they were written before hyper scalers and the assumed attack vector was that someone would literally steal a hard drive.
A running machine is not “at rest”, just like you can read files on your encrypted Mac HDD, the running program has decrypted access to the hard drive.
(And modern Linux is unusable without root access, thanks to Docker and other fast-and-loose approaches.)
Because I never do, unless I'm down in the depths of /var/lib/docker doing stuff I shouldn't.