How about:
1) It's not just a DB problem.
1) It's not just a DB problem.
So not all scaling problems are database problems.
This is an argument in semantics.
You suggest storing all users physical addresses in memory across multiple machines. You also advocate "shard splitting" a la B-tree indexes which split when the node's population reaches 2n. OK, so when a split occurs, how do you maintain that massive table in memory. Do you lock it? Do you examine your existing queue of requests (which may soon point to the wrong node)? How do you handle new queue requests during "shard split maintenance"?
Calling these "database problems" is waving your hands. Why not just call them "computer problems"?
Regarding when shard split how you maintain the table, the easy answer is Terracotta. It is amazing for this and handles all that kind of stuff totally in the background. If you didnt have terracotta it would be a harder problem indeed. It handles locking and all of that magic for you. I highly recommend it. Regarding how you handle queue requests while the split is going on, what happens is that while the split is happening, the original shard continues to respond, but all new updates are sent to the old shard and the new one that is being created in the split, so during a split there is addition messaging going on. Once its done, the original shards range of users is cut back, but actually it is still storing all the old data. At that point you can delete the data that has been moved to the new shard - though this is not necessary.
Right. Just because we didn't write it doesn't mean it's magic.
There is still a lot of work to be done. It can get very complicated for high velocity apps. And it takes time.
So you're right. Bigger minds than the two of us are probably struggling with this (and lots of other stuff) as we write.
Now as you scale to 50 or so servers they will start to fail with some regularity and you can't reboot and or rebuild in five seconds so we need to have redundant systems ready. Splitting a shard probably takes longer than five seconds so you need to be proactive about such things. So we need an active rebalancing between shard so they can quickly pass around their load and fast failover incase things fail. These are DB problems but they are only problems because they system need to keep responding quickly as parts of the DB fail.
But this is still DB centric how do you load balance incoming and outgoing requests? (Routing each existing user to the same system every time and letting each system lock data when they want to change their password etc.) How do we recover from data corruption? How do we notice an outgoing or incoming mail server is down? You also have n^2 number of connections between systems as you grow can we keep that many sockets open? (More of an issue is we are segregating the network.) Etc.
PS: These are not hard problems by themselves but they are linked. And there are a lot more where they came from.
For example, I load tested my last rails app (load testing is my background-I have consulted for major pharma, tax, and other companies) and on a wimpy inherited server, processor maxed out first for the rails processes. I popped in a second CPU, then was able to get to the point where we could support 200 concurrent users, which was all we needed at the time.
I have seen limitations on bandwidth, firewall problems, load balancer misconfiguration, poorly written queries, memory usage,processor usage, thread deadlocks, and licensing limits (max database connections on a trial license) all be bottlenecks. At the larger scale, yes, database is more often the problem, but it's not clear twitter should be using a database in the way they are, anyway. Your statement goes too far.
Now consider a voting system you have 1,000 items and 1,000 users and you want to show how much people like them like each item? How well does that work when there are 100,000 users and 100,000 items?