Hacker News Reader

Comment by rb808 | Hacker News Reader

> Much harder, and much longer: you couldn't release to production until someone with authority signed off on it, and that might take days.

Wow lucky guy, its only a few days for you.

View on HN

Lucky you, it's only one sign off!

At JP Morgan (major US bank) the most brutal ticketing system (ITSM) required approval from 6 people in average. Could take a whole week easily just to get in touch with each of them and beg for a change to be approved. Thankfully there are very few systems that require this kind of ticket for access. (and there is a bug in the ticketing system anyway so could get a valid ticket in 15 minutes when truly needed twice a year).

The direct effect is that all systems relying on that for access control are abandoned and rotting because it's impossible to do preventative maintenance.

Well, I suppose JP Morgan probably saw Knight Capital Group's mistake.

I worked at a place that had seasonal "no deploy" policies and had some software that controlled very expensive equipment. It was amazing the number of processes needed to update certain things.

It is a true balancing act. I do wonder if any actual courses exist that talk to business people about software life cycle, what is needed for a live system, and how to judge such things?

My reading is that Knight had a manual, high-touch release process. It’s hard to imagine enough bureaucracy to make that safe. Companies with proper CI/CD may deploy with what seems like reckless abandon, yet are essentially immune to the particular mistake of accidentally forgetting a server.

The part of the investment bank that deals in trading system is fully CI/CD. Developers deploy 10 000 times a week (measured during the coronavirus change freeze so probably below usual).

I guess I should be the one writing about software life cycle? Do you have any particular questions in mind? That will give me a starting point for a next blog article.

A long time ago, I once had a deployment rejected by a change management team because I used the wrong form - the right form was identical to the form I had used apart from the title. They insisted I resubmit the whole thing and wait until the next time they reviewed changes.....

I think I almost cried - after that I simply ignored them and deployed stuff when I wanted... :-)

I think I almost cried - after that I simply ignored them and deployed stuff when I wanted... :-)

Yeah, great. I'm sure nothing ever happened and if an outside audit showed up, the organization would have failed. Never mind if something bad happened, its not only you getting axed.

The systems that needed that level of auditing were carefully controlled and I wasn't daft enough to modify those without CAB approval (in fact, I couldn't modify them).

One of the problems I have with these kind of processes is that it often applies the same level of process to all systems when some systems really don't need that level of control.

This is a really tricky thing to get right. If you put too many controls in place, people route around and you lose control. If you place too few in place, you lose control. If you try to make the rules flexible, people don't understand the rules, and you lose control.

Maybe it would work better if you educate people instead of insisting on controlling them?

Maybe, but one thing I've found is the normal developer often doesn't have visibility into the financial or legal side of the company that often mandate things. Liability is often a problem and they don't often come for the developer when problems arise.

Arguably, this is the intended outcome. The form exists so that the change management team can supply evidence of process compliance to its auditors, but when the scope of the change management processes is driven arbitrarily it can be ridiculously hard to quantify risk of change which slows decision making.

However, if a rogue engineer might decide to flout the policy and deploy stuff whenever they wanted, then the business stands to profit from the gains more quickly and with less hassle. If the risk doesn't pay off, the responsible engineer can be fired for cause, itself demonstrating that the business is in control.

You may not want to take on uncompensated personal risk by operating outside these lazily-defined change management policies.

It was about 10 years ago and a former employer that I left 5 years ago.

> I think I almost cried

The difference between capitalism (as a whole) and communism (as a whole) is that one of them has one less layer of bureaucracy.

Reply on news.ycombinator.com