In apps like this the policy is to only add columns to tables and new tables, never to remove anything, so the database can be upgraded hot, and the old code can continue to run on it, only the new code will see any new columns/tables.
If it's just 1 column, and it doesn't store much data, then probably no. If it's a bunch of columns and it's taking up lots of disk, they they probably migrate what they need to a new table and drop the old one. Altering tables is usually not a good idea. In mysql it can stop replication and can mean hours or days of downtime depending on the situation and amount of data.
We have access to multiple colos of slightly over-capacity machines. It's not uncommon to migrate by switching all traffic over to one colo. Then upgrade the next colo to the new code and switch all traffic there. Finally upgrade the first colo and then start cross loading the traffic between colos again.
Since we are big enough to have at least a couple of colos per geographic region this means we can avoid rolling migrations which are pretty horrible when you get trapped in a failure mid-roll.
Basically, hardware is much cheaper than downtime, and very big web companies have lots more money. So we don't swap out old servers gradually: instead we buy and set up an entirely new set of servers, deploy to them several weeks in advance of the planned launch, and run QA against these production-level boxes. Then, when we are ready to "launch", all we're really doing is a relatively low-risk DNS change: all the potentially tricky deployment issues having been ironed out beforehand.
After a couple of weeks/months of operation on the new hardware, if there have been no major problems, the old boxes are decommissioned -- either re-imaged and put back into service to expand capacity, or more often taken out of service entirely (I've no idea what we do with old boxes when we stop using them, funnily enough).
Also, while the DNS is switching there would be some major database inconsistencies between the two server stacks. How do you deal with that?
Could you clarify what you mean by database inconsistencies between stacks?
What property do you work on? How is it more efficient to buy or reallocate entirely new servers than update your old ones?
How many times per year do you even update your application? The cost of getting the servers ready and installing packages would be prohibitive if you had to roll out, say, an urgent security fix.
When doing a major release seldo is spot on. Especially given the small amount of time it takes for hardware to get out of date it's often cheaper to provision new hardware prior to the major release and then deploy straight onto that. Mostly the old boxes would then get cycled back in to production, something less strenuous like staging, or retired.
For the really huge properties, the cross-colo upgrade pattern you (sh1mmer) describe is the way to go.
Also, cross-colo is not a factor of being big, it's a factor of having a BCP (business continuity plan). Every team at Yahoo is urged to have BCP for practically everything. I used to do cross-colo upgrades for a relatively small project once we had more than two frontend servers.
I image cheap way to fake multiple colos now is to use visualization, amazon style. Heck you could do exactly what Seldo was suggesting every time on EC2.
Also, the new Buzz site used all new hardware (and was a new product) and Games rolled out on new hardware when it was upgraded from HF2K to PHP. The old servers were reimaged and folded in with their PHP siblings once it was clear that the build was holding up properly.
When you cannot, I, personally, believe in partitioning traffic across concurrent versions. This can be done dynamically or statically -- really it depends on the nature of your system, in general something will stand out as obviously right for your situation.
To take an example, if your service is primarily user centric, you can partition the system by user and roll out accordingly. Let's say you have four interacting systems: a font end proxy which understands the partition boundaries, an appserver, a caching system, and a database -- pretty typical.
The front end proxy in this system is shared by all users (this need not always be true as you can do subdomain and dns games, but that is a different major headache), but everything behind it can be dedicated to the partition (this is not necessarily efficient, but it is easy).
Now, let's say we need to make a backwards-incompatible,coordinated change to the appserver and databases associated with the partition. As we cannot roll these atomically without downtime we pick an order, let's say appserver first. In this case we will wind up rolling two actual changes to the appserver and one to the databases.
The appserver will go from A (the initial) to an A' which is compatible with both A and B databases, then the databases will go from A to B, and the appservers from A' to B. You'll do this on one small partition and once done, let it bake for a while. After that, you'll roll the same across more. Typically going to exponentially more of the system (ie, 1 partition, 2 partitions, 4 partitions, 8 partitions, etc).
This means you have a, hopefully short lived, interim release of one or more components, which is probably grossly inefficient, but you wind up in a stable state when complete. The cost of doing this is not pleasant, as you basically triple QA time (final state, interim state, two upgrade transitions) and add a non-trivial chunk of development time (interim state). That said, this is why most folks just take the downtime until the cost of the downtime is greater than cost of extra development.
This is, of course, a pain in the ass to coordinate. It is easy to do with relatively small big systems (less than a few hundred servers, say, assuming you have good deployment automation), and probably the pain of coordinating is still less than the pain of baking component versioning into everything... for a while.
An alternate model, which requires significantly more up front investment, is to support this in a multi-tenant system where you don't (for upgrade purposes) dedicate a clone of the system to each partition. Instead you can bake version awareness into service discovery and dynamically route requests accordingly.
A very traditional (of the blue-suited variety) is to use an MQ system for all RPCs and tag versioning into the request, then select from the queue incorporating the version. This makes the upgrade almost trivial from a code-execution point of view, and can even help with data migration as you can queue up updates during the intermediate state database and play a catch-up game to flip over to the end state database. This is the subject for a blog post, though, rather than a comment, as it is kind of hairy :-)
Integration, or sytem testing had taken place over the previous 1-2 month period.
They have 2 large datacenters, let's call one "production" and the other one "disaster recovery" (abbreviated "DR"). Between these and the internet are some large routers (the Cisco ones that cost about what a house costs). At the beginning of "migration" the routers are switched to point from production to DR, so that all internet traffic points to DR.
At this time, the servers are being updated with the new code. This can take some time, especially if large databases need to be restored as part of the migration, or if the update scripts take a long time. There are a lot of servers involved, a ballpark is about 100 servers: some Sun, some WinTel, some IBM mainframes. Some mirrored, some clustered, some all by their little old lonesome. Some applications are Java, some .NET, and some Cobol[2].
Approximate timeline [3]:
People start dialing into the main conference call about 5:30PM eastern [4].
Switch to DR about 6pm eastern.
Code in production is migrated/installed, servers rebooted if necessary. Done about 9-11pm eastern.
Testing [5] starts and continues until about 3AM.
GO/NO-GO decision is made sometime between 3 and 4AM[6].
Rollback if necessary.
Switch routers to point to production at 6AM.
Preliminary postmortem report generally done by 2PM.
If no rollback, then repeat the following evening for DR.
Notes: 1 - They're a Fortune 100 company, I'm not telling who they are.
2 - That I'm aware of. It would not surprise me at all that there are a number of other "brands" of servers or programming languages involved.
3 - The actual timeline is usually a spreadsheet that's at a minimum, 50 pages long, plus about 10 more pages of first, second and third contacts in case something bursts into flames.
4 - It is common to have 100+ people monitoring the main conference line, and 1-2 dozen other conference lines used for individual components/products. One has to be awake and alert in case you're "called out" on the conference line.
5 - In general, because the main URL/URI/Hostnames are now pointing to DR, hosts files are changed so that configuration files don't get edited.
6 - Sometimes the decision gets delayed until almost 6AM if there are some problems.
http://www.cennydd.co.uk/2008/can-we-avoid-redesign-backlash...
"In a nutshell, a meaningless background was removed from a seller page. Pandemonium. After strong resistance the background was reinstated, to everyone’s satisfaction. In fact, the rebellious users were so placated that they failed to notice the designers slowly adjusting the background’s hex values over the next few months. The background got lighter and lighter until one day—pop!—it was gone."
Look under the title "eBay.com: Satisfying their Users with Gradual Evolution"
1) Up the code from our development environment to our staging server.
2) Personally test the shit out of it. We didn't have any QA people, so have fun finding your own bugs after looking at the same screen for X minutes/hours/days/weeks.
3) When all is good, have one of the senior developers run our sync script which would rsync the code off of our staging server to our master production server into a directory called something like: live_20081228.
4) Log into the master production server and run another sync script that would copy the code from the folder created in step #3 into the live directory. Sync script would then sync that directory to all of our other production servers.
5) If there's problems: 5a) Immediately roll back 5b) Pray that you still have a job.
The process, overall, was pretty terrible and allowed for a lot of problems to arise (which happened regularly).
When you can run multiple versions of code against the same dataset, upgrading the application layer one bit at a time becomes much easier.
"Developing Erlang at Yahoo" http://bayfp.blip.tv/file/1281369/ - Talks about the transition of Delicious from Perl to Erlang, including live migrations of the DB.
"Taking Large-Scale Applications Offline" http://www.youtube.com/watch?v=cePFlJ8sGj4 - in particular, the section on "Versioning and phased rollouts: what to do when you can't assume all servers have the same code"
In general, I'd say, most sites data is sharded or clustered. Amazon, I'm guessing, basically has many many different instances of their app running on different clusters all over the world (multiple clusters per datacenter). So they upgrade a cluster at a time, and their databases all synchronize with others of the same version.
Facebook's data is, obviously, sharded by network. So any given cluster runs x number of networks. Upgrades are then, again i'm guessing here, rolled out network by network. The data layer can be different from network to network.
Most of the time though, updates to large scale services aren't changing db schemas or making huge changes to the data layer, so it's as simple as updating the code and rebooting some app servers.
(Obviously I don't know exactly how they do it, cause I don't work at any of these places, but deploying it's that hard of a problem to solve)
They most definitely have multiple instances of the app running around the world, which synchronize data with each other, which is why you've never been unable to buy something. It's highly unlikely that all their instances would be down or overloaded at the same time.
Also, they most certainly don't have a single logical database. They use all kinds of things, including SimpleDB. You think product information is stored in the same database, or even in the same away as customer information?
>What I mean by that is that within Amazon, all small pieces of business functionality are run as a separate service. For example, this availability of a particular item is a service that is a piece of software running somewhere that encapsulates data, that manages that data.
The "secret sauce" is an invisible layer in front of the webservers. The client's connection terminates in this layer, and the loadbalancers establish new sessions with the web servers - the client never actually touches the server. This extra layer lets you do all sorts of clever stuff that was unimaginable back in the day when all we had was Squid and round-robin DNS.
From an amateur POV: But when usual servers come into play, I have had no clue abt big ones.
Infact a couple of months back, when my home server had a burst of traffic from my mobile app, it couldn't handle it (256kbps connection, 256MB ram with a 1.3 GHz P3 processor. well suited to serve a high bandwidth mobile app of 1000 users and also for constantly collecting data from APIs). But due to sudden traffic outburst, I had to string a friend's comp to mine.
My procedure: I first stopped updating data from APIs to my DB and kept it static. No updating the DB. Copied it to my friend's fast comp. I had my computer interfaced with a mobile which accepts input from the user mobile phones. That was the app's requirement. So I had an advantage of lining up requests right there in the mobile phone while I was switching the server. That trick should help if you are developing a mobile app on a very limited resource (a hobby project).