Just-In-Time Scalability
startuplessonslearned.blogspot.com
startuplessonslearned.blogspot.com
From slide 13: "Learning: cross shard joins & transactions aren’t required"
Do you mean that there was never a time where you had to do a join or transaction between shards? Would this in turn imply that your data was embarrassingly parallel?
Also, just-in-time here seem like it refers to rapid architecture iterations. I was expecting something ec2/appengine-ish, maybe this is just a problem of terminology overloading.
IMVU has a pretty typical data profile for a social network + IM site. We worried all the time that we would have to build a really fancy distributed transaction engine. We started by just doing all of those transactions on the centralized master database. Since we were incrementally moving load off the master as-needed, we just did those tables last.
Turns out, that approach lasted more than three years, much longer than we ever anticipated given the rapid growth we experienced over that period. Of course, we did eventually have to tackle it, which we did by building a "durable message-passing" system.
As for terminology, you're completely right. We started using this term a few years ago, so it wasn't quite as overloaded as it is today. I am sometimes surprised how much more time and energy seems to be given over to thinking about the hardware, as opposed to the architectural, aspects of scaling.
If you mean that people are ga-ga over how many of Amazon's machines they could scale to but not how they can add layers of caching and database partitioning -- that's probably accurate. IMHO the key point is that Amazon has a generic framework. The architecture you choose to scale with probably ends up being a lot more ad-hoc, "driven" by your problem domain.
I'm fairly confident that if someone comes up with a rocket-ship approach to architecture that generalizes well for web startups, time and energy will be given over to thinking about it.
http://www.speakeasy.org/~ericries/Just-In-Time%20Scalabilit...