Bloomberg Terminals Suffer Widespread Failures
nytimes.com
nytimes.com
Bloomberg's tickering plant is still working and their emsx(trading system) is up and running, albeit a bit slow right now.
I think it was on a stackoverflow podcast that Joel or Jeff said that it makes you wonder about people who complain about down time if they've ever built something of their own.
Google's been down, ditto for facebook, twitter and Azure. If you build a big service, something will eventually fail, Bloomberg is no different, if anything they are much better than Reuters or CapIQ for availability.
I saw this quote from another comment here:
> but the widespread reliance on the $10k+ per year terminal in the financial services industry means that this thing is ripe for disruption.
I completely disagree, Bloomberg isn't some slow lethargic big company. They iterate relentlessly, their help desk is the best I've ever had the pleasure of dealing with.
If I've got a problem with the trading system, their first level help will get me to a technical specialist in that area in under 30 seconds. If they can't solve it then I'll talk to a developer in 30 minutes.
Bloomberg is the leader in teh financial space, not only because their tech is the best in teh business, but because they listen to their customer better than their competitors, they release more frequently, and when they want to enter an area( say risk management) they go out and buy someone who is already a leader in that field.
In short they never stop running, if anything they are increasing the distance between them and their "peers".
As one example, they just posted a message to every Bloomberg terminal user telling us that the issue was an internal networking issue that has been solved.
EDIT to answer this question:
> How do they protect the ticker tape thing from attacks, like say, a DDOS?
2 things:
1) This question really doesn't make sense. How do you DDOS a one way line? They send you quotes for the tickers that you request. You can't really DDOS them, unless you do something like request tickers over and over again in a tight loop, in which case their api will rate limit you.
2) its a closed system. Only terminal users can access it and only through the bloomberg terminal or their api, so they control all the access points.
The first three, at least, are free services and availability isn't a priority for many of their customers. Bloomberg users have different requirements.
Boeing 747's are complex systems; it's not acceptable if one fails occasionally.
Yet they have failed. Reality isn't always acceptable.
You're right that 747s should not just fall out of the sky. However, parts of 747s fail constantly. In a machine that complex, they are guaranteed to have some downtime. A well manufactured 747 will fail cleanly and safely and recover smoothly.
Probably a much bigger issue for everyone who doesn't have exchange connectivity, but you'd also hope they have some sort of fallback for exactly this scenario. I.e. reuters or something.
Any insight on what happened today? I'm surprised there's no failover mechanism that tries to route traffic some other way in case of a communication failure.
Unfortunately BBG requires a bit more scrutiny because when their network goes down entire countries cannot finance themselves -- they had to postpone some debt sales in the UK today I read.
HN in particular seems to have a "_____ is down" on the front page every few days. It's always filled with comments like this guy[1] saying how their architecture is clearly broken or how any fifteen year old could write something better.
Bloomberg runs some of the most reliable technology in the world, and is one of the best at running distributed architecture, so the fact that a problem like this wasn't more contained is surprising, if not shocking.
But reports from inside the company suggested
that a spilled can of Coke in one of the server
rooms had been responsible for knocking out
systems across two continents.
Fuckin please.
As for a few seconds delay, that's probably how they're interacting with them, not how they could use them. Both have both very different UIs and very different Excel plugins (assuming someone has both on their desk, from experience, they probably use the Excel plugins a lot/mission critical functionality).
If I wanted to play around with a Bloomberg for a few hours in North America, just to see what the fuss is about, how could I go about doing this?
At that level of cost there is an expectation of almost perfect up time and multiple redundancy - the only reason people pay that much is they believe they can get more value than that out of it..
I'm not a huge fan of Javascript and it's proliferation of frameworks but some of that stuff is objectively both good and useful.
Back a long time ago the applicable pejorative was "yuppie". I haven't heard that one recently. I suspect that many people reading here would have to look it up in Wikipedia.
Edit: BTW at the time I was partial to calling people "yuppie scum". Sigh. It's so hard to keep up with the changing putdowns! :)
That said, there might be a market for a web framework in PL/I there is a whole lot of IBM big iron out there :-P
Would be an amazing business that, supposing you could get buy in from a load of other firms.
Sure, C++/C/Fortran/some home grown template library is what it's built on. What would they gain by rewriting it some node framework of the week or whatever other shiny distraction? If they did do that and that caused downtime, there would be the same naysayers here complaining about that instead.
As another commentator mentioned, chip away. That is how Bloomberg made it as a company. By being better and more sane at fixed income in a very new market.
Find a niche and chip away at that. Data is expensive. Personally, I think making a niche in something coming to becoming mainstream (like R) which is already there in some boutique areas is where it could be.
But don't quote me. I'm still doing COBOL>
Is it a legitimate compliance or practical implementation problem, or is it just the network effect?
I enjoyed the article and I have squirrelled this quote from a trader away for future use - the need to meet and exchange information is why we used to have stock exchanges of course.
http://www.businessinsider.com/reason-for-the-global-bloombe...
But if you want to cripple the financial system, Bloomberg's a good single point of failure to attack. That and Seamless, which also had a hiccup today.
That crash was totally not strategic at all. /s