Comment by ritchiea | Hacker News Reader

Comment by ritchiea | Hacker News Reader

Parent Full thread

This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental.

I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev never really cared that we didn't have tests, or a careful deploy process. He just got angry when things went wrong. And even gave one more dev with even less experience than I had access to the code base.

It was a mess and the only thing I was "learning" was "don't touch any code you don't have to because who knows what you might break" which is obviously a terrible way to learn as a junior dev (forget "move fast and break things" we "move slowly and work in fear!"). So I quit and moved on, it was one of the better decisions I've ever made.

As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage".

Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger team-mates, a recent hire, "don't be afraid to break things!"

This is correct and is the way it should be. So how come the programmers are always politically gunning for the keys to the production server cabinet, where you do have to be afraid to break things?

That's where raises and bonuses come from.

Are they? Where I work (a bank) we were more than happy to move to read-only access to prod servers, and pass by a support team when we need to deploy things.

Heh. Another way I would say that is that "discretion is the better part of valor" And, to rip-off Hitchhiker's Guide to the Galaxy, "cowardice is the better part of discretion"...in that if fear makes you judiciously check your backups and write tests, then that's not a bad thing at all.

I believe developers need the courage to do what we feel is necessary and the paranoia to quadruple check our work and assumptions.

Paranoia is often useful, but with a good enviorment, and tools it's rarely need. I find bad assumptions often cause the worst problems. Break things regularly and you end up with fewer assumptions about the code base / production environment which is a vary good thing.

I don't trust tools unless I wrote them myself, in which case I know there will be bugs in them.

This is the right thing to encourage but I just would like to add always have a backup.

"Don't be afraid to break things as long as you have a backup".

It might be a simple version of the previous code, database copy or even the entire application. Do not forget to backup. If everything fails, we can quickly restore the previous working version.

Use source control. You can always revert.

I would go one step farther....

Every production deployment should involve blowing away the prior instance, rebuilding from scratch, and restarting the service; you are effectively doing a near-full "restore" for every deployment, which forces you to have everything fully backed up and accessible...

Any failure to maintain good business continuity practices will manifest early for a product / employee / team, which allows you to prevent larger failures...

Spoken like a man who has maintained applications but never databases.

In the world where data needs to be maintained, this is not necessarily an option. In the bank where I work, we deploy new code without taking any outage (provide a new set of stored procedures in the database, then deploy a second set of middleware, then a new set of front-end servers, test it, then begin starting new user sessions on the new system; when all old user sessions have completed the old version can be turned off). Taking down the database AT ALL would require user outages. Restoring the database from backup is VERY HARD and would take hours (LOTS of hours).

That being said, we do NOT test our disaster-recovery and restore procedures well enough.

My own rule now is:

The bug is in the part that I though it was so obvious, I missed some check.

Hmm, is it courage like "real men test in production environment" ?

slowly and work in fear businesses usually collapse. the technical problem eventually becomes a business problem and no one realizes it can be solved at the source level.

it first builds inefficieny at the technical level then at the business level, finally it causes issues at the cultural level, and thats when the smart people start leaving.

Reply on news.ycombinator.com