Hacker News Reader

Comment by benjaminwootton | Hacker News Reader

I'm surprised by the monolithic all-or-nothing deployments that they have off of the single binary.

I prefer to componentise applications and allow those components to be deployed, released, and rolled back separately.

I also don't agree with the 'rollback is for losers' message as hinted at in the article.

A fast dependable rollback (measured in the seconds) is significantly preferable to getting a developer to implement a fix to some issue under pressure and push it out in a rush. Much better to roll back, take stock, then implement the right fix under the tested process.

View on HN

This deployment of what's called "www" ("dub-dub-dub") is somewhat monolithic in the sense that it is large, but it is not really an "all-or-nothing" deployment. It is tested internally (for at least 2-3 days) by employees (engineers and non-engineers), and then on a subset of servers, a larger subset, and so on. Performance, error rates, interaction rates, and so forth are compared between the incumbent and new-hotness versions for a reasonably reliable indication of issues throughout the rollout. More on the www push at https://www.facebook.com/video/video.php?v=10100259101684977

Teams that run "services" (such as the one that powers the "typeahead" search bar at the top of Facebook) deploy their services separately, at whatever pace makes sense for them. These are almost never in tight lock-step with anything else, for the usual reasons.

If a new release goes out every day, how is that tested internally for 2-3 days? Do employees get split across versions or are incremental changes rolled out much more frequently?

Sounds like only the latter is being discussed: Facebook typically rolls out a minor update on every single business day. Major updates are issued once a week, generally on Tuesday afternoons.

Facebook is updating tens of thousands of servers with every push. "Rolling back" a release could take as long as a regular push and contribute to problems as the version in use diverges.

Instead, FB has an aggressive and flexible internal system for "ungating" features to groups based on different criteria. Usually a feature would be pushed out in a deactivated state, then a developer will slowly ramp up its exposure to actual traffic. This limits the ability for a push to insta-break the site and means they can come back around for the next day's push with tweaks, then increase the code's exposure.

Still though, the 1.5 GB binary takes 30 minutes to get out to the servers as per the article.

In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30.

I agree that you can and should have all of the gating and safety nets and checks in the world as you roll forward, but at the end of the day, you need the ultimate safety net of a very fast, very dependable rollback that you can run at will.

IMO reverting isn't for losers. It should be the first thing you do in case of errors, if your architecture supports it.

"In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30."

According to Rossi the servers keep the old version around after rollout, so rollback wouldn't require them to push out the old binary again but just to restart the old process. This will probably only take a couple of seconds making rollback really fast if needed.

According to the article reverting does NOT involve re-deploying. Each server maintains the previous version of the binary and if needed the release team can pull the switch to revert all the servers. I assume that takes seconds, not 30 mins.

In fact i worked at a dutch social network where we also used hiphop. The new compiled binary is pushed to all servers and then it is started, the old binary is stopped and a port handover is done. Thus deploying without downtime. The old binary is available on the system, making a rollback very fast. However old binaries are removed after a time. So you can only roll back to a previous version quickly.

"Facebook is updating tens of thousands of servers with every push. "Rolling back" a release would effectively take as long as a regular push and could contribute to problems if they're found." Absolutely not. A lot of places get around this with a simple symlink switch. You keep N older releases. So it might be /code/releases with various datestamped releases...and /code/releases/current points to the currently running one. Want to rollback? Point current to one older. Done.

Presumably the 1.5gb facebook binary is deployed as a daemon listening on port 80. Seems unlikely that they are cgi-ing a new process for each request.

CGI?

I'm torn. I understand your question, but yeah - what is that binary doing?

It is compiled PHP. Do they embed an http server? Do they talk fcgi with a well-known webserver, like - php would?

HPHP has a built in multi-threaded webserver.

Reply on news.ycombinator.com