Hacker News Reader

You had a mandate for 5-7 nines uptime and you couldn't tolerate a problem on a single node?

yup, that sounds strange. if the environment is that advanced, then there shouldn't be a problem of taking the node in question out of the service pool and let the devs loose on it (assuming there's no info that they aren't supposed to see). when they're done, just recycle it.

Where did they say it was a single node issue?

i think they mean, in case he mess up a single node, it should still be fine

I see, that makes more sense, thanks for the clarification.

I’d still rather not have to debug arbitrary mutations to the env or file system in a production container though.

It seems to me that shelling into a prod container/VM is discouraged not because you might cause it to fail, but because you might produce undefined behavior while claiming it is healthy (more like Byzantine faults).

For example if you unset a single env var by mistake, then 1/N of your requests will potentially fail. Debugging this kind of issue is a nightmare.

Not to mention that developers can often run arbitrary SQL from a prod shell when the app is backed by a DB.

> For example if you unset a single env var by mistake, then 1/N of your requests will potentially fail.

Or far worse, not fail.

Reply on news.ycombinator.com