Hacker News Reader

Comment by 62951413 | Hacker News Reader

Cases like that make me wonder why there's no wider adoption in out industry of some trivial ideas such as (my first inspiration would be the aviation industry):

* checklists/runbooks with sample commands so that you copy/paste instead of manually typing or relying on your shell's history

* explicit scripts, named in a verbose and clear way, to be executed for dangerous steps (e.g. to delete something or deploy to prod) instead of allowing typing random commands

* for really critical operations, a mandatory code review/a second pair of eyes in some way (the blog seems to mention something of this nature)

View on HN

We do this in all of my projects. It’s a pita and tedious but on big enough projects with big enough clients (governments) things like deleting a prod database means CNN shows up on your front lawn.

Checklists with copy paste commands are used some places, but it really requires that the team maintaining the process feels like they are getting value out of them for the checklists to be kept up to date.

One place I worked we had detailed step by step checklists with ready to copy paste commands for patching server. The Windows process was forever drifting and getting out of date while the Linux process was being kept up to date.

The difference was that the Windows admins were all morning people who enjoyed showing up to work at 7am or earlier if they could justify it and just doing it from memory. While the Linux team were all night owls who preferred to roll out of bed 5min before the update schedule, remote in, execute the process with as much automation as possible in the hope of getting some more sleep before having to got to work.

Basically the Windows process was only ever actually followed when someone outside of the Windows admins had to cover for them and the Linux process was used every time by the Linux team.

>checklists/runbooks The last couple places I've worked have called these Standard Operating Procedures (SOPs). Effectively just the manual version of something that can be automated

From the article, it sounds like it would have helped to formulate the correct command in a non-production environment before moving to production

Because they do exist, have for 20 years, they just aren't used because that sounds like work that isn't a minimum viable product, or a waste of time to most cowboy "developers" / startups.

Testing / separation of dev environments is almost black magic.

Because we have no industry standards for ops. There are certainly IT standards, like ITSM, but the day to day product operations are ignored. And even if there were industry standards, nobody would implement them until they were forced to, same as the rest of the standards. Product people don't want anyone restricting them, and their priority isn't to ensure operations are reliable.

Reply on news.ycombinator.com