Hacker News Reader

Comment by cagenut | Hacker News Reader

This is cool and all, but its part of an overall trend about aws/ec2 thats starting to get on my nerves. They're spending their time and resources working their way up the stack instead of improving the lower layers of the stack for which you use them in the first place. Right now ~2+ms inter-node latencies, arbitrary 200+ms hangups on EBS, and the general lack of an SSD option make AWS a non-starter for things that need to scale without being architecture-astronaut computer science projects. For many workloads I have to colo instead of use AWS because of the age old "spend 10K on hardware or hire an FTE developer" tradeoff.

View on HN

Designing for fault tolerance and scaling out instead of individual node durability and scaling up is, in most cases, a sounder approach. Since the vast majority of applications map well to this case, it makes sense for Amazon to focus on features that assume/assist scale-out.

It's easier to design for scale-out than one might think; a great resource to get started is http://highscalability.com.

Regarding EBS, I haven't seen the hangup issue you've described. Any data?

As for low inter-node latencies, Amazon has an offering that specifically addresses that need: http://aws.amazon.com/ec2/hpc-applications/

Overall, I think Amazon is making a lot of inroads in areas with specific hardware demands. They just launched GPU compute options, and I expect we'll even see SSDs soon.

Scaling out vs. scaling up is the false dilemma, the answer is both at the same time solving the bottlenecks you have using the time and resources you have. The "vast majority" of applications do not map well to an n-wide map-reduce-esque design, the "vast majority" of applications bottleneck on an authoritative datastore's I/O capacity.

I'm the sysadmin (not the dba) for a website that does 30K http req/s at the edge, and our mysql cluster does about 3000 req/s during mid-day with most queries in the 1 - 5ms range.

Highscalability is a terrible place to get started, its a great place to share notes but if you get started there you will waste insane amounts of time on architecture astronaut nonsense.

What do you consider architecture astronaut nonsense?

On the spectrum of scaling an application AWS has definitely leaned hard towards scaling out. (where the other end is scaling up) This means it caters a little more to denormalized databases and load balanced web servers than it does to one giant mysql database server or a cluster of a few large database servers.

I agree with nethergoat in that if you need low latency communication between ec2 instances that you should be using their hpc instances with faster/optomized ethernet.

I however disagree with nethergoat and agree with cagenut and think that the vast-majority of applications still currently depend more on a scale up paradigm. That is slowly moving towards scale out as people/businesses see its benefits and AWS is catering to this trend.

Your claim is a false dilemma unless you can prove otherwise.

http://www.mysqlperformanceblog.com/2011/02/21/death-match-e...

tl;dr - "price is in the same order of magnitude, but performance is two to three orders of magnitude different."

Your claim was that Amazon, by working on features higher on the stack, is unable to work on improving lower performance. You need to prove that there is a lack of engineering resources at Amazon to work on both tasks at once.

Reply on news.ycombinator.com