Hacker News Reader

Comment by flyt | Hacker News Reader

Facebook is updating tens of thousands of servers with every push. "Rolling back" a release could take as long as a regular push and contribute to problems as the version in use diverges.

Instead, FB has an aggressive and flexible internal system for "ungating" features to groups based on different criteria. Usually a feature would be pushed out in a deactivated state, then a developer will slowly ramp up its exposure to actual traffic. This limits the ability for a push to insta-break the site and means they can come back around for the next day's push with tweaks, then increase the code's exposure.

View on HN

According to the article reverting does NOT involve re-deploying. Each server maintains the previous version of the binary and if needed the release team can pull the switch to revert all the servers. I assume that takes seconds, not 30 mins.

In fact i worked at a dutch social network where we also used hiphop. The new compiled binary is pushed to all servers and then it is started, the old binary is stopped and a port handover is done. Thus deploying without downtime. The old binary is available on the system, making a rollback very fast. However old binaries are removed after a time. So you can only roll back to a previous version quickly.

"Facebook is updating tens of thousands of servers with every push. "Rolling back" a release would effectively take as long as a regular push and could contribute to problems if they're found." Absolutely not. A lot of places get around this with a simple symlink switch. You keep N older releases. So it might be /code/releases with various datestamped releases...and /code/releases/current points to the currently running one. Want to rollback? Point current to one older. Done.

Presumably the 1.5gb facebook binary is deployed as a daemon listening on port 80. Seems unlikely that they are cgi-ing a new process for each request.

CGI?

I'm torn. I understand your question, but yeah - what is that binary doing?

It is compiled PHP. Do they embed an http server? Do they talk fcgi with a well-known webserver, like - php would?

HPHP has a built in multi-threaded webserver.

Still though, the 1.5 GB binary takes 30 minutes to get out to the servers as per the article.

In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30.

I agree that you can and should have all of the gating and safety nets and checks in the world as you roll forward, but at the end of the day, you need the ultimate safety net of a very fast, very dependable rollback that you can run at will.

IMO reverting isn't for losers. It should be the first thing you do in case of errors, if your architecture supports it.

"In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30."

According to Rossi the servers keep the old version around after rollout, so rollback wouldn't require them to push out the old binary again but just to restart the old process. This will probably only take a couple of seconds making rollback really fast if needed.

Reply on news.ycombinator.com