London Stock Exchange: What really went wrong | Hacker News Reader

London Stock Exchange: What really went wrong

computerworlduk.com

27 pointsDMPenfold200814 comments

Short summary of the article. The outage was caused by the introduction of new requirements by LSE for end-of-day price reporting. The data vendors were the ones who had the responsibility to adhere to these new rules, by making changes to their systems. Unfortunately, the "big bang" approach to the switch-over proved fatal.

From the article:

“I am astonished they did not run the systems in parallel,” said a source at one of the major data vendors. “At least for a few weeks or months as necessary.”

The LSE argues that it gave plenty of proper testing and preparation time, and that the vendors should have been ready. It has been working with them to resolve the issues.

> “I am astonished they did not run the systems in parallel,” said a source at one of the major data vendors. “At least for a few weeks or months as necessary.”

I worked at a wall street trading firm for some years in the past. Our group was primarily working on replacing older mainframe applications. After the normal back and forth with development and QA, they would be run in parallel with the systems they were replacing for a minimum of 3 months. Development was not allowed to install, monitor, "fix" or otherwise touch them during that time. We either got a "go" or "no go" periodically, and we had to fix whatever the issues were or they never saw the light of day.

I haven't followed the story that well, and only skim read this article, but there was a four-ish hour outage in the morning last week. I think the LSE came up OK for about half an hour but then went down (although it might not have come up at all; I can't remember). It seems unlikely that that was caused by data vendors not reporting end of day close prices correctly.

Why am I reminded of Douglas Adams? "All the planning charts have been on display at your local planning department in Alpha Centauri for 50 of your Earth years, so you've had plenty of time to lodge any formal complaint and it's far too late to start making a fuss about it now..."

I seriously doubt the plans were that far away. The transition was not announced the day before after all.

Remember when there were some issues with LSE's Windows-based solution? You had Linux/FOSS fanboys all over saying that this was obviously the fault of Windows and what not and that Linux would fare much better.

I'm wondering what sort of response they will have to this and why they weren't as eager to report this to social news-sites as they were with the Windows-story.

Maybe this will finally make them realize that shitty (design-)decisions can be made on any platform and for complex software the underlying OS/kernel doesn't really matter all that much. We can only hope.

It was a counter reaction. The previous system's advertisement campaign was very strong and I remember people implying it was a better solution than a Linux/FOSS alternative. Microsoft had a huge campaign and it was one of their main examples on what their new .Net platform was capable of doing.

Also this article has nothing to do with downtime but with issues on closing prices.

The LSE acquired a mostly unknown company in Sri Lanka to code this system.

And on top of that, the LSE is extremely opaque and old school about reporting what was wrong with their systems every time something went belly up. They never have a post-mortem and regularly blame "human error" without giving details.

These seem to indicate systemic problems with the LSE. That makes me wonder if it was really justified to blame Windows Server and SQL Server the last time around.

I don't know. In this case the outsourcer seems to be a lowest-bidder offshorer. Last time it was Microsoft. I don't know if there's anything that can be learnt from this.

That makes me wonder if it was really justified to blame Windows Server and SQL Server the last time around

Maybe, maybe not. But Microsoft probably should have thought twice about featuring TradElect as a star .NET deployment given the generally poor development practices that seem to have been at work here.

What happened with the previous system was that it dragged on for years and years, with many expensive delays and poor technological decisions. By contrast, the Linux system is up and running much faster.

So yeah, you can try to spin it as 'Linux is just as bad as Microsoft', but if one takes 5 years and is a complete disaster, and the other one takes 2 years and has a few hiccups, I know which one I prefer.

Disclaimer: as far as I could care, Windows and Linux can both go and die in a fire, so you know I'm neutral :D

-----

The root cause of the problem the previous one had if I recall correctly was that they had some CTO come in who didn't know anything except the Microsoft way of doing things. So, naturally, they ended up picking Microsoft for everything. Problem is, the Microsoft sales reps promise you the world and whisper sweet nothings in your ear until you give them your virtue, but once they have ravished you they roll over and go straight to sleep.

>Microsoft sales reps promise you the world and whisper sweet nothings in your ear until you give them your virtue, but once they have ravished you they roll over and go straight to sleep.

Isn't that true of any reps? Why single out Microsoft? And this deal was with Accenture IIRC.

Haven't you dealt with IBM or Oracle reps?

You're not making a persuasive case to me that the programming language and OS, rather than the developement methods, management style, and expertise of the coding team were the factors that determined sucess or failure.

> So yeah, you can try to spin it as 'Linux is just as bad as Microsoft', but if one takes 5 years and is a complete disaster, and the other one takes 2 years and has a few hiccups, I know which one I prefer.

The point being made is that the problems in both the previous and current systems were probably nothing to do with the underlying OS/platform, but because of some incompetency in the design or implementation of this specific project.

Reply on news.ycombinator.com