Hacker News Reader

Comment by EwanToo | Hacker News Reader

I can see this going down in IT folklore, "Remember to do test your backups and roll back plans, or you might end up like Ulster Bank"

View on HN

I work with some people who used to work for RBS IT, they tell me that the department was as bureaucratic as they come (presumably to reduce risk) but once all the hoops had been jumped through, it came down to single team who would put code into production, and they were often quite poor at it.

When I think about the scale of this mess, and the hundreds of millions that RBS is going to loose in compensation and lost customers, I hope that RBS (and in house corporate IT in general) starts to appreciate the need for automated deployment and fault tolerant IT systems rather than yet more process as the answer (I'm looking at you ITIL).

ITIL is not a problem, in fact it is a good thing that has happened to IT. However, the problem is the non-IT people running the ITIL. They end up sidelining the developers, or the developers badmouthing the processes, which causes all sorts of separation and issues. When people follow processes appropriately, things work out well and on time.

ITIL is a problem. It's extremely rare that an organization has problems because nobody's ever thought about how to do things better rather than having broken internal structure or incentives – ITIL can't help with either of those but the huge amounts of time and money it soaks up will make it much harder to actually do anything about them.

Or perhaps:

"Don't outsource all of your IT ops team or you might end up like RBS/Ulster Bank"

If you're an IT guy then that one is perhaps a more important battle to win.

In theory a separate company who's core competency is running IT systems should be less likely to mess this up, not more so.

However, retail banks are rapidly becoming sort of API hosting companies for the processing of financial transactions. Their core competency IS running IT systems.

If none of RBS's physical branches opened tomorrow, the UK would probably tick along just fine, with some inconvenience caused to people who deal mostly in cash. But fail to process payments for 24 hours, and the whole UK has problems. If this had gone any longer than it did, it's not hard to envisage a run on the bank and a domino effect to other banks. It's really quite scary how badly we depend on these IT systems, waaay more than Google or AWS.

In theory a separate company who's core competency is running IT systems should be less likely to mess this up, not more so.

That's true. Rather than "Don't outsource" the warning should be "Don't fire all the existing staff who understand your systems".

They didn't. They fired half of the team.

"to do a NatWest"

(Unfortuntely it slightly rhymes with 'best', but meh)

I couldn't help thinking of the backronym: "Really Bad Shit".

I think it would work better with some kind fo rhyming scheme.

Reply on news.ycombinator.com