Finance world in the dark as Bloomberg terminals go offline | Hacker News Reader

Finance world in the dark as Bloomberg terminals go offline

telegraph.co.uk

45 pointsjacquesm44 comments

I don't understand how Bloomberg could be down worldwide.

Worldwide???

99% of what Bloomberg terminals are useful for can easily be replicated in multiple data centers. Why not, at a minimum, three separate regions, e.g. Asia, Europe, America?

These terminals lease for $24,000 a year. No discount for multiple terminals. I wouldn't be surprised if a single big investment bank pays $50 million a year to Bloomberg for its terminals. Michael Bloomberg is a billionaire. Surely the company can throw a few crumbs into some redundant data centers and availability zones.

And if it's down because of a botched worldwide software update, the people responsible should be immediately walked out the door and told "you're too f*ing stupid to work here any more".

IMO.

Sorry to be so dramatic. But the financial world depends on these terminals (for better or for worse). They have a right to expect unrealistic reliability.

No need to say sorry. Truth might be bitter but is still truth. It's going to be a little doom's day today in the finance world worldwide.

They're really lucky this happened on a slow news Friday; especially with everything going on with Greece right now.

Support people at Bloomberg must have been really busy today and phones may be constantly ringing. Wish we get post mortem report later on.

The service does have unrealistic reliability. An event like this is exceedingly rare.

Terminals can lease for way more than $24,000 depending on additional data options, and an investment bank can easily spend more than $50m on them.

There will also be Reuters, Datastream and various other terminals with not quite substitutable functionality kicking around the office, so they're not totally 'in the dark' its just a bit less convenient and some trades could be missed.

Bloomberg has just paid a lot for their experience of what mistakes to not make in the future.

Walking them out the door would be a fantastic waste of money.

That's the way a smart business would think. I wonder how smart Bloomberg is? It's not uncommon for the goals of employees ]assign blame, cover ass, grab more power, short term profit] to be misaligned with the long term goals of the business.

There is a lot of assumption going on (that a human messed up). Even if that was the case I've never known it to be a place that would have knee jerk reactions like that. The company is more interested in saying "failures will happen, how do we prevent them from becoming major issues?".

That sounds like a smart company.

Anytime humans are involved mistakes will be made. Where I work when something goes down it's a total panic and later it always has to be someone's fault. Sure lends itself to lots of CYA and working scared.

Former bloomberg employee here. While the bloomberg service usually runs very well, its built mostley on spaghetti code that has been copy pasta'ed since the 80s and 90s. It might have improved since I worked there, but there were some serious deficiencies in development methodology.

Heh, interesting tidbit. I've seen them advertising for positions in London on HN and they gave a good impression.

In light of your nickname it's not surprising then that you write 'former'.

I don't know when you worked there, but that is not how things are run. Source: I work there.

This is the same BBG where the huge hairball of C++ wrapped around C wrapped around Fortran makes up the crustry codebase? Where the interview process is still shitty pointer and char array questions? Where they implement their own borked version of the STL? Does anyone internally follow Lakos's books there? (if so or if not, why?)

My interview at BB involved zero shitty pointer and character array questions. It was challenging but included virtually no trivia whatsoever. And that was in 2004.

I agree, the interviews I did with them recently didn't involve pointer/char array questions. Most of the questions were a lot more interesting compared to interviews I had with other companies at the time.

Disclosure: I start at BBG in June.

Do you remember any questions? I'm going to be applying there soon and I'd like to know.

My interview at BB actually was 90% about pointers and character arrays, 10% about the keyword static in C. I got the job worked a great 3-4 years there. Also, that was 13 (!) years ago.

My interview in 2003 was mostly legit, though one senior manager's 30-minute segment was all BS of exactly this nature. (His initials were GJ and in the four and a half years I worked there, I discovered he was the closest real-world embodiment of the Dilbert pointy-haired boss I've ever encountered)

The most memorable part was:

GJ: Name three examples of defensive programming.

Me: I'm sorry, I'm not familiar with the term; could you define it?

GJ: You know, defensive programming.

Me: I've not heard the term, but maybe I know it by a different name; could you describe it?

GJ: No, let's move on.

Later, I looked it up. He just meant things like `if (p == NULL) return False;`. I knew the concept well, and could've spoken about it intelligently and effortlessly, except I had never heard that particular name for it.

Oh, I just remembered another exchange between us:

GJ: What's the difference between a segmentation fault and a bus error?

Me: [perfect explanation]

GJ, seeming disappointed that he hadn't stumped me: Oh, someone already asked you this, huh?

Me: No, I know this, because once when I was in college--

GJ: Come on, you looked this up before, didn't you?

I wasn't sure how to respond -- I mean, yes, technically I did look it up, once upon a time. How the hell else would I know, derivation from basic axioms?

Anyway, I got the job, but I wouldn't be surprised if this other person was interviewed by the same guy and it left a bad taste in their mouth, too. As a senior manager, he probably interviewed 500 candidates a year.

Not GJ, but yes, a senior manager who asked similar questions. The getting angry at the answer bit is interesting - I gave an answer to one question that was obviously even better than the canned answer he was expecting, he actively ground his teeth a moment or two, like he was being shown up in front of the junior colleague, and at first refused to discuss the question until I pressed the point.

I have friends still working at BBG, so I get to hear the current interview stories, and it seems things have not really improved.

My favorite BBG anecdote was when I was on the hiring side. I rated a candidate "hire at top priority" but he was rejected by another manager.

When I asked why, the manager told me it was because the candidate wrote "xml" on his resume and not "XML" and that any knowledgable programmer would know to capitalize it correctly so he clearly didn't know his stuff.

</seriously>

This only happened because he wrote his resume in Emacs or vi. A real programmer would have used MS-Word and it would have autocorrected.

Haha, that’s incredible :)

I didn't know a programmer is expected to have "XML" (or "xml" on his cv). It's something that you can pick up very quickly if you don't know it.

Let's all chime in with Bloomberg interview experiences! My experience included two timed online tests full of "shitty pointer and array" (and C++ constructor initialization order and C undefined behavior and...) questions and two phone screens of the same before they brought me onsite.

The onsite interviews were 10% system design, 10% more "shitty C++" questions, 20% actual coding, 20% analytical puzzles ("you have two eggs and a skyscraper...") and 40% grilling on networks, such as enumerating all the error codes that socket system calls can return. While I've written tons of low level networking code, I didn't have the depth of networking experience they sought. (And I didn't do well on the analytical puzzles.) It was an interesting challenge though, and my only complaint was the job description didn't indicate what they really wanted.

2005 to 2007, but things might have changed since then.

One of the "fond" memories was during our training where the guy basically said that when you compile your function (or app) the first time, you run the command and then get a cup of coffee cause it gonna take at least 20 mins, due to cyclic dependencies in the codebase.

Hi Andrew!

What I remember from my time there (2003-2007) was how quickly everyone was able to ship code despite (or perhaps because of) no global style guide, unit test mandates, or even rules about copy-pasting code. Each team was able to do things the way they thought best, thanks to an incredibly resilient, flexible production infrastructure that could tolerate, isolate, route around, and quickly repair any newly-introduced bug.

The code path (Bloomberg four-letter function, to those familiar with the service) running through the bug could nearly instantly, and if I'm not mistaken, in a totally automated way, be flipped over to a hot backup instance running the previous week's code, and a patch or rollback could be released to the live code almost as soon as it was written.

Most tech companies see the problems caused by coding too fast and respond by doing things to slow people down. Bloomberg cured the ills of excess programming velocity by turning the speed up even further. Like Facebook's "Move fast and break things", but with an extra, "...and then move even faster to fix them", and in a way that (mostly) insulated customers from being exposed to the breakage.

The fact that it's front-page news when there's a hiccup shows that it's a man-bites-dog event. You'd be in trouble if it wasn't major news when there was a service incident.

That's very interesting, and definitely an infrastructure many companies should strive for and could learn from.

I'm wondering, however, not how problems were handled, but how were they detected - code throwing exceptions is the easy case, the real problem is code running almost correctly, but with slightly wrong outputs. I imagine there could be many different "sensibility" checks implemented, (e.g. no negative prices, no huge jumps price jumps), but in general it appears to be a very interesting and complex problem.

It probably depends on what group you work with. I interviewed there a few years ago following a who's hiring post by you (thanks!), and the interviewers seemed to know what they were doing. Didn't work out for me, but it was the group owning the feed, IIRC.

However, since then I've interviewed many candidates from Bloomberg, and while they were all very smart, their description of the work they did indicated systems that were at varying levels of WTF. In their defense this was most likely due to legacy reasons.

Also, it's funny how Bloomberg candidates inevitably use multicast in their system design interviews, which throws most "web company" interviewers off balance :-)

as recently as 2013 there were still libraries that had Fortran dependencies.

I wonder how many people appreciate the Irony that twitter is now the universal mechanism for people to report outages, and get updates on when critical services are down/up.

All those that were early enough to see twitter go through their growing pains probably.

And Bloomberg terminals are now the universal mechanism for people to monitor and report on how successful Twitter is as a company.

Is there an alternative to Bloomberg? I haven't ever heard of traders using anything else.

Reuters offers a terminal with similar capabilities.

As do CQG

Depends on exactly what but FactSet. Have used both, FactSet is more user friendly, Bloomberg is way faster once you get used to it.

Depending on what they trade, traders aren't going to be using anything else because of the messaging on Bloomberg. There are a lot of services popping up solely as a replacement for the messaging service due to that Bloomberg reporter really fucking up and letting people know that they snoop on things and the many bank collusion scandals.

Bloomberg's stickiness is primarily about the IM network. Everyone is on it. Familiarity is a close second, but nobody likes paying the fees and an alternative would be welcomed, particularly by clients burned by the chat log snooping scandal.

There was some press quite recently about an industry-backed initiative to develop an alternative IM platform (which is a necessary first step to displacing terminals), but nothing much has been said since.

FYI another discussion going on. Parent link there is to an NY Times article. https://news.ycombinator.com/item?id=9393884

IB still down for some people; think I heard 25% worldwide.

Just happy everything was up for me before getting to work.

It also looks like the markets are dropping sharply (SnP futures etc). I wonder if there is a connection?

Reply on news.ycombinator.com