LSE Halts | Hacker News Reader

LSE Halts

londonstockexchange.com

101 pointsashishb4u69 comments

http://online.wsj.com/article/SB10001424052748704150604576165753761876730.html?mod=googlenews_wsj

The problem is that the LSE can't hire top tier talent. There are plenty of people who are experts in building hugely concurrent and fail-safe exchange software, and they pretty much all work for investment banks.

The LSE can't afford to compete with $1000/day salaries so they end up hiring second-tier developers. And unfortunately second-tier developers are a lot lot worse than first-tier developers.

I don't think it's about talent, but more of a problem of proper planning and execution. It seems things had been rushed to meet the tight demands, causing the chaos.

When LSE acquired MilleniumIT, to redo the platform, they claimed to have developed world fastest trading platform[1] and had enough talent under their belt developing trading systems around the globe for last decade.

[1] http://www.wallstreetandtech.com/exchanges/showArticle.jhtml...

None of the exchanges have the required talent. Being the best exchange technology company is like being the best team in a minor league. If you're good at extreme low latency stuff you're going to be working for a bank.

At a bank being great at low-latency can mean tens of millions of dollars of profit for your trading desk and a huge bonus for your personally.

At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back.

> At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back.

Exchanges pay out bigger bonus' than you'd imagine. 7 figures for a programmer is high but no where out of the ordinary. Says a guy who has worked for an exchange.

I seriously doubt any exchange is regularly paying developers multi-million dollar bonuses.

Which exchange did you work for, I don't see one on your linkedin profile?

Exchanges often use (relatively) esoteric hardware + real-time OSes, such as the Tandem NonStop series. Experience with such a platform + domain knowledge about the finance industry is a rare combination, so I find it plausible that a couple of core expert programmers on the stock exchange's system get compensated quite well. High six figures + 80% bonus == seven figure, I guess?

> I seriously doubt any exchange is regularly paying developers multi-million dollar bonuses.

I'd agree with that. Having said that, I doubt many investment banks are regularly paying developers multi-million dollar bonuses.

The people who get those are called quants or traders, even if they have programming skills, just my experience.

> Which exchange did you work for? I'd rather not say, if you want to know, contact me via email. Not sure how you got my linked in info as it's not attached to my profile.

None of the exchanges have the required talent.

Ahem, based on what evidence exactly did you reach that conclusion ?

TBH, from all the exchanges/brokers I've seen and dealt with in the past, there are a few that are pretty solid - Deutsche Borse is one of them. There's EuroMTS as well.

Essentially, most exchanges that deal with many market makers have got an extremely challenging technical problem to solve - scaling and latency. One that most investment banks can't even begin to solve.

I'm not saying there aren't any good talent in banks - there are plenty. I'm just saying that there's some really smart, talented people in exchanges.

That's very likely. Any programmer with even an entry level position at the LSE would get head hunted by multiple agents in the first month. And the investment banks do pay £700/day for talent.

I know at least two guys who are getting paid closer to £1000 a day to work in London for a bank. The money is pretty much insane imho, and the guys tells me the 'talent' is just like anywhere else - ie most people are meerly compentent (but still getting paid £500 plus a day). Even graduates with zero experience are taken on at £300 a day or more.

He seems to better than competent, and so has managed to negotiate a much higher rate.

Does anyone know if this halt is related to the recent switch over to the new Linux based platform? (http://news.ycombinator.com/item?id=686116)

it is a new platform, hard to tell but I wouldnt go so far as to say linux is to blame, its more likely to be exchange software, they've gone from native to fix 5, so its using a new protocol, new os, new software stack and most likely new hardware stack

I don't think famoreira was trying to blame Linux. --That's just the part of the new system that you can reference easily.

Yes, I didn't mean to blame Linux. I was just referring to the system in the way it was made famous (replacing a Microsoft based system with a Linux one).

I don't think that "linux is to blame" or even that any choice of programming languages or other tools are to blame.

It's a large system with extra-ordinarily tight performance requirements, and a lot of integration with external systems, and by nature hard to phase in gradually. That kind of system is very hard to get right first time regardless of your toolset. I'd be interested to hear more about thier methods to deal with these requirements and risks, less about thier platform choice. IIRC, both this and the previous failed system were outsourced, which doesn't exactly fill me with confidence.

I believe this is due to a glitch in the Application and not in Linux per se.

"And as a final note, solely for the seriously nerdy techies, there's been some talk about the failure being due to F# error."

http://ftalphaville.ft.com/marketslive/

I've no idea where they got this from, although I mentioned it on Twitter as a joke: https://twitter.com/tim_g_robinson/status/41086872876892160

F# on linux?

http://www.mono-project.com/Release_Notes_Mono_2.10#Language...

Via mono. If that's true... that would be an interesting news for the project (not because of the bug itself of course...).

If the LSE were using F# we'd have heard a lot more about it already. For instance, from Microsoft.

Maybe not if they were using F# on mono. Their PR would have to work very hard to put this in a way that doesn't say "LSE is using our language on a foreign platform using a completely alternative implementation, which has nothing to do with us apart some ECMA specs - and they're still successful without MS involvement. Yes, we just confirmed that you can trust your highly critical .NET environment to the alternative versions of runtime."

OK, I wouldn't expect an official press release like this from Microsoft head office.

But the team in Microsoft Research are (a) keen to promote their language (people like Don and Brian often help early adopters personally) and (b) supportive of Mono (they've always released a .zip file for Mono at the same time as their Windows installers).

They have publicly stated that the system is written in C++ on Suse Linux.

http://www.computerworlduk.com/news/open-source/3260727/lond...

I wouldn't believe 100% in those statements.

True enough. If the statement turns out to be not entirely correct, that would be news.

I suspect that if there is any F# code, it's in a 3rd party system that talks to the stock exchange. There are lots of those due to the nature of an exchange, though they should all talk over the standard FIX protocol.

> I wouldn't believe 100% in those statements.

Why not? What makes you think this is a lie?

They systematically cover the problems and have zero transparency. Never a post-mortem. Like back in January.

using a immature language (f#) over unproved software stack for an international multi billion dollar low latency transactions when there are dozens of proven highly stable functional alternatives that run well on a linux platform? I can hardly believe it

That's modern enterprise computing for you.

I don't believe it at all. Mainly because of the public statements that the new LSE trading system was written in C++.

On the other hand, it has to integrate with lots of enternal trading systems, one of which could easily be written in F#.

I doubt they'd be using Mono - if it really is an F# issue it maybe indicates it's a failure in some of the legacy system they are replacing.

As far as I know, the previous system didn't use F#.

"written in C#" http://www.computerworlduk.com/news/open-source/3260727/lond...

I still don't understand how they choose languages with garbage collection for systems that should be real time or similar.

It's not F# or Java, it's C++. http://news.ycombinator.com/item?id=2261896

It could be some left over system from the previous .Net platform causing a knock-on effect.

It could be anything. Why do you want it to be that?

I don't want it to be that. I'm saying the rumour sounds plausible.

Not so plausible to me, considering that the old system was in C#.

I'm pretty sure I've read somewhere that they where using F# for some parts.

Can we have a link please?

That seems unlikely. F# only recently moved from being an experimental language from Microsoft Research to a supported .NET language. The previous system is old enough to predate that.

Agreed. and Anderson consulting using an experimental, non-obvious language? I don't see it.

GC isn't a problem for realtime systems, if configured correctly. For example, have the GC running incrementally in a separate thread.

Close, you do need to stop the world every once in a while to GC some of the harder stuff.

But it's true, there's flavors of Java that are certified for hard realtime. If you stop the GC for 100ms every 2s regardless if you need it or not, that's pretty nice and predictable behavior for a realtime system.

100ms is an eternity. You can do GC with 1ms pauses.

http://www.infoq.com/articles/azul_gc_in_detail, for one example.

Erlang is not "real time" in the hard, embedded systems sense, but it does pretty well and has GC.

Here's a bit more information: http://www.erlang.org/faq/academic.html#id55755

I don't know what sorts of 'real time' properties trading systems are required to have - can anyone talk about it with a bit more in-depth knowledge?

I don't know much, but for high-frequency trading, fast is never fast enough. response times are measured in microseconds.

http://richg74.wordpress.com/2010/10/25/speedy-new-lse-tradi...

http://www.h-online.com/open/news/item/Linux-boosts-Stock-Ex...

Speed isn't all. The system should be based in algorithms with manageable worst case time.

I don't buy Erlang camp's reports.

A GC will always trash the cache, and that's the worst issue right now in hardware. (If you take out HDD I/O.)

You may be interested in the Metronome project, which aims to provide a GC suitable for realtime use: http://domino.research.ibm.com/comm/research_projects.nsf/pa...

An academic paper about the project, Design and Implemention of a Comprehensive Real-Time Java Virtual Machine: http://domino.research.ibm.com/comm/research_people.nsf/page...

that link doesn't contain the text 'F#' any where on the page.

It's been archived to http://ftalphaville.ft.com/blog/2011/02/25/498596/markets-li...

Thanks:)

"Continuous trading has now resumed in all order driven trading services."

Sri Lanka? http://uk.reuters.com/article/2009/09/16/uk-lse-it-sb-idUKTR... (via /B @ftalphaville comments)

Nothing official yet. (http://www.ft.com/cms/s/0/9bf8033a-40bb-11e0-9a37-00144feabd...)

What if all these systems were open sourced? Including all the load tests.

At this size, some obfuscation might be good. If you find a bug and crash some browser, it's not a big deal. If you find some problem in LSE that can be triggered from outside, you're dealing with realworld $M transactions.

Also, unless you have a huge farm of servers it would be useless to you - and basically anyone really.

> What if all these systems were open sourced?

LSE considers its software to be a competitive advantage.

.....er sure

When they make it work, it could be ;-)

Where is the fail over? If ever a system needed 99.9999...

Unfortunately sometimes fail over isn't a solution. Until they've at least narrowed down the problem, a fail over system may make things worse.

Personally, I'm curious about the testing that was done - given the complaints and problems that have been reported so far, did they even try to run the new and old systems simultaneously for any period of time to verify that the various components could all keep up properly? Or at the very least, did they try to test the system at full load for a decent period of time?

Failover in this scenario would require instant, 100% replication of a near-zero latency system.

I'm no expert, but I'm fairly sure having trades 'disappear' or any other loss of consistency would be worse than downtime in this case.

this is the official place for updates, although its not very responsive at the moment,probably due to load http://liveservice.londonstockexchangegroup.com/en/incidents...

It's been archived - here's the link to this incident in the archive:

http://liveservice.londonstockexchangegroup.com/en/incidents...

I wonder if it's HFT related? A group of rogue algorithms gone exponential? There are certainly people worried about that...

Reply on news.ycombinator.com