Introducing DataFabric

It is overwhelmingly likely that by the time you actually need to know about sharding, you won't be "new to this" anymore. :) It's a technique for sites with so much write traffic that a relational database running on one big machine with a lot of disks isn't enough to keep up. That is a lot of traffic, particularly if you're good at regular old SQL performance tuning, which is what you should learn first.

Anyway, I haven't studied sharding too closely myself because... IAGNI IAGNI IAGNI. But here's what I remember from spying on smarter people in hallways at conferences: Pro: You get to spread your database writes over multiple machines. Con: making an RDBMS do an efficient JOIN across machines, or enforce constraints across machines, or (god help us) execute an efficient atomic transaction across machines ranges from "annoying" to "incredibly effing difficult", which erodes much of the advantage of your RDBMS in the first place and causes people to start muttering about alternatives.

You are checking people in at a conference registration booth. You have people with last names A-M in line one and N-Z in line two. You have just sharded the registration forms by last name.

The Pro is that it's a lot faster to get people registered. The person at the desk has to seek between roughly half the records. The Con is that the person at the desk can't easily/elegantly get a record not in their grouping. For example, a couple comes up where they have different last names (John Jones and Jane Smith). You can't check them both in without bothering the person next to you and making the process slightly cludgy.

How does this apply to databases? Well, if you have a users table that is sharded, some users are in DB1 and others are in DB2. You can't simply join across the different servers. That's the big con to sharding. You lose the ability to join. There are things you can do, but they're cludgy.

The good thing to take away from this is that, unlike the conference check-in, databases are really good and fast. You don't need to shard as quickly as you do with humans. When you start getting a high level of concurrent activity, you can add replication. Once you've exhausted that, only then should you think about sharding. Without sharding, you can create large sites, but you'll get to a point where you need to start thinking about it - you're not at that point (not that I am). Twitter is at that point and handling it poorly. Really, if you get that big, you'll have the money to hire someone who knows what they're doing.