Some notes on running new software in production | Hacker News Reader

Some notes on running new software in production

jvns.ca

84 pointseurg11 comments

The wisdom that comes from running things in production makes open-ended questions about those gotchas useful interview questions.

'looks like you used kubernetes for a major project recently. What were some of the unexpected warts?'

Someone with experience running a service on k8s will have at least a few stories, and how someone describes their own responses to those hurdles can shed light on their personality.

Agreed, though it saddens me how many of those hurdles will be poor rediscovery of previously well documented problems in older systems. :(

I tend to like these sorts of questions, really shows you how much a person understands about a system. I also like it when its asked of me, though I always wonder how much a rant about the things I've experienced setting up Nomad/Consul/Vault would be productive.

Don't make me relive those stories!

> Envoy’s active healthchecking means that you services get healthchecked by every client. This is mostly okay but (again) services with many clients can get overwhelmed by it.

Man that's brutal. Is that normal in service-mesh frameworks?

Try working at a place where all fixes are tested in production...

Can you describe what you guys do and your infrastructure? Is there a need to test in production or has it just not been important enough to prevent it?

Everybody has a test environment, it's rare to also have a production environment.

Your statement doesn't match up at all with what they said nor even make sense. What's the point of a business if you only have a test environment but rarely have a production environment?

You seem to be assuming to know what their development process and infrastructure is like.

The joke is that if you don't have a testing environment, production is your testing environment, which in effect means that you don't have a production environment.

Sorry, cannot describe infrastructure (not a throwaway account).

Prod and Test are out of sync in terms of mission critical software, Test is not powerful enough to handle Prod's tasks, and there is no formal procedure to use Test for testing. So Test is never actually used.

When a bug is found in Prod, it's fixed on Prod, while it's live.

By fix, of course I mean an iteration on and combinations of:

qc: we found bug

dev: it's not a bug

qc: it is a bug: [insert evidence]

dev: it's fixed now

qc: no it's not

dev: how about now?

qc: no

dev: and now?

qc: yes but now this other thing is broken

dev: here's another attempt, please check again

qc: Prod is down

dev: try it now

qc: it's still broken, along with the other broken part, and now these three other parts are broken too

ceo: it's fine, close this ticket

Reply on news.ycombinator.com