Hacker News Reader

Comment by MBCook | Hacker News Reader

Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years, is obviously bad.

Deploying in such a way that all your servers are not running the same codebase is obviously bad.

Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad.

Not having anyone monitor things closely enough, including the hundreds of warning emails they got before the market opened, is obviously bad.

There is no hindsight necessary here. You could look at what they were doing and predict a catastrophe.

View on HN

I'm not sure I agree with "Deploying in such a way that all your servers are not running the same codebase is obviously bad." I have a lot of experience in large scale systems (although this incident with 8 machines does not qualify) and I would say there is _always_ a period of transition where versions X and Y are online in production simultaneously. How can it be otherwise? You'd need scheduled downtime to do it any other way.

I think the main problem here is nobody at this company pushed back on this stupid development plan of reusing a flag for a different purpose. There's no excuse for that (or maybe there is, they had run out of fields in some fixed-width message format or something dumb like that). Also apparently the use of the flag was not tied hermetically to the binary in production; when they rolled back the binary the flag was still there but it meant something different to the old software.

The correct way to roll this type of change out is for the new input (the "flag" in this case) to be totally inert for the old version of the software, and for the new version to have a config file or command line argument that disables it. So _first_ you start sending this new feature in the input, which is meaningless and ignored by the existing software, and then you roll out the new software to maybe 1% of your fleet, and see if it works. Then roll it out to maybe 10% and leave it that way for a week. Insist that your developers have created a a way to cross-check the correctness of the feature in the 10% test fleet (structured logging etc). If it looks good roll it to 100%. You now have three ways to disable it: turn it off in the input stream, turn it off in the new software with the config or argument, or roll back the software.

Doesn't look like these guys really knew what they were doing.

"How can it be otherwise? You'd need scheduled downtime to do it any other way."

Trading floor is only open a few hours every day, the functionality being rolled out required the markets to be open. Furthermore, since the changes were all for new functionality they rolled it out in stages days ahead of time (good move).

> How can it be otherwise? You'd need scheduled downtime to do it any other way.

Roll out the code in advance, and have the production machines switch to it at a defined, synchronized time?

I mean, imagine you only have one production machine. If you're willing to admit that you can have it switch from version X to version Y with no downtime, then synchronization is the only barrier to doing the same on n machines. Why would you need scheduled downtime?

Synchronization is non-trivial, but the question is mostly how fine you need the synchronization to be. E.g., if you are doing a live upgrade using e.g Erlang or Nginx, you can sort-of decide when new processes will be served by the new server, but existing processes and requests may linger with old code until much later.

But there's at least 30 minutes of downtime per week per market (usually per day), and the vast majority of downtimes coincide during the weekend - so this is all moot discussion and needlessly complex solution. If you can afford the downtime, switch midnight GMT between Saturday and Sunday, when all markets are closed.

You're right, yet nearly everybody operates on a daily restart schedule. I guess it's more intuitive, but it's wrong. Still, nobody ever got fired for bouncing systems once a day!

You're right. I understand a slow rollout, I was referring to the fact that they thought they deployed to all servers but didn't.

If you plan is for the software to be on all servers, it needs to be on all servers.

I wonder how you can have code sit on a server unused and then 8 years later have it be called? What language was this written in?

Why would the language matter? Dormant code is the same in any language.

Poster was probably thinking of directories of interpreted code (i.e. Python, PHP, Ruby), compared to compiled binaries (C++, C, C#, Java), since many compiled languages have dynamic deployment mechanisms like OSGi's hotswapping of individual modules.

It was a flag that was re-purposed. If the code is never deleted and the flag's use is discontinued then the code never runs. However, it seems that the flag was reused years later causing the old code to be inadvertently reactivated. It has nothing to do with the language it was written in.

It's most likely a giant C++ server or set of servers. It's not really that surprising that old code is in there. One way releases are handled in these kinds of places is to roll out new features on some kind of external flag system, perhaps shared memory flags or some other mechanism. This way if something goes wrong with that one feature, you can disable it by flipping the switch, rather than having to back out the software, which would likely cause downtime during market hours, a major problem.

What sometimes happens when you want to decommission features is you just turn the flags off rather than remove the code. There's an obvious allure to this as you have already tested the on/off functionality of the switch when you did the original roll-out so you can avoid having to test whether you have removed the code correctly. It sounds like in this case they removed the code and repurposed the switch that disabled said code (it may really be a shared memory system and they are running out of flags), but they fucked it up. The old code was still there on some servers and the switch was turned back in intent to enable the new feature it was re-purposed for, re-enabling this old code.

Reply on news.ycombinator.com