How to end the Fail Whales? With Blue Whales.
blogs.zdnet.com
blogs.zdnet.com
This was a very weird article:
> I had just completed a project for a large government agency that needed a system that could accomplish approximately 5,500 database row inserts and reads per second, or approximately 18 million transactions per week.
Not even close--5,500 a second comes to over 3 billion weekly transactions. 18 million transactions is nothing, that can be handled on a VPS.
> And as a customer requirement, we had to do it with COMMODITY HARDWARE.
Here's his idea of commodity hardware:
> The heart of the “beast”, as those of us referred to it, was comprised of 4 Unisys ES7000-based Oracle 10g RAC nodes, each using 256GB of RAM and sixteen dual-core Intel Xeon processors, and the most sophisticated, high-performance Cisco 10Gbps Ethernet switching that money could buy which was available between all the main clustered and load balanced tiers of the system — Web, J2EE Application Servers, Messaging Queue and Database.
Probably a million dollars in hardware and it barely was able to do 18m weekly transactions. Now for his advice to Twitter:
> No, there’s only one systems architecture that will solve the Fail Whales once and for all. As much as I hate coming off like an IBM commercial — cue the blue background and jazz music on Tech Broiler — the solution to the Fail Whale is a Blue Whale — an IBM System Z. A Mainframe. Or two. Preferably in a SYSPLEX configuration.
"Enterprise" people really grind my gears. Google started with servers in LEGO cases. Even today they just use tons of not very important hardware. The solution to scaling Twitter isn't throwing $2m to IBM, it's not trying to run it out of a RDBMS.
In the long run you won't be able to buy your way out of a shitty foundation.
When you fail for three years to stabilize one of the simplest imaginable architectures (pub/sub) then throwing some money at IBM to run your broken app off a big iron that "just doesn't care" doesn't seem unreasonable.
It could just buy them the time they need to cash out and be done.
Why is re-writing some of the system in Scala a debacle and or expensive? I haven't followed it too closely, but I got the impression that they have had success with it so far.
Well, when you have a track record of screwing up on a fairly simple problem for 3 years straight then a switch of programming language will not magically solve that for you.
At an industry event last fall, I suggested this very strategy to the newish VP of ops/engineering/scaling Twitter had just hired from the porn industry. My advice was Professor Frink-ish: "why not just buy one really big computer?" I was only half joking, but he wasn't impressed by my idea. Perhaps he remembered my name and that's why their recruiter has not called me back.
From what little I've heard about Twitter's development, I don't think they're mindlessly sticking to their initial model. They've been actively looking at ways to move beyond their initial RAD app. Now, I'm sure they had reasons to choose Scala for a layer, whether it's the best or not. Should they have gone with [insert-your-favorite-language-here] instead is largely a matter religious debate.
Any approach has inherent strengths and weaknesses. You can also write bad code in even the "best" languages. But to say they're simply throwing more hardware at inefficient code is probably a misrepresentation of their approach.
> No, there’s only one systems architecture that will solve the Fail Whales once and for all.
That is certainly one way to go. Another is to be clever. That's what startups do best... A large company goes to buy a main frame and Oracle licenses while the startup has a really clever idea and orders a bunch of servers from Dell.
Amen to that. The fact that we give Twitter such a hard time about their growing pains is reflective of our growing "reliance" on them. Clever approaches to "mission-critical" applications will include plenty of trial and error and iteration. A bank's not going to say "let's experiment with some other approaches."
I think that's the point of the article - if you need it to "just work", go with something that can provide that. Not to say they won't eventually get to a comparable point using unorthodox techniques, there will just be stumbles along the way.
It's a good PR move, though. By showing that mainframes work for Web 2.0 (even with a faulty argument) they'll probably score a corporate sale or two.