Amazon announces Elastic MapReduce | Hacker News Reader

Amazon announces Elastic MapReduce

aws.amazon.com

148 pointsthamer61 comments

It is intersting to watch Amazon's strategy as they improve existing cloud computing services and add new services. The way they are going, it looks like Amazon will totally dominate the market.

I expect this will have huge importance to the web, and computing more generally, in the next 5 years or so. I wouldn't be surprised if this becomes as important, or even more important than their traditional business.

I feel like Amazon doesn't get nearly the mind-share they deserve for doing really awesome stuff. They've been up on the front of so many technology curves it's silly.

We kind of take for granted what they do since, you know, it's a store. But they're in search, they're heavy into data analysis, they've innovated in interfaces, payment processing, they've had community generated content since before we called it that, they've pulled off one of the most effective infrastructure platforms, they're redefining publishing and content distribution. And they have a great reputation for customer service.

Another thing I like about Amazon is that I have a pretty good idea of what they're collecting all of my data for and it's something that I'm pretty comfortable with.

I feel like they kind of get left out a lot of times when people are thinking about serious online innovation.

they're heavy into data analysis

That's what I've enjoyed about Amazon for years. I like going back to Amazon to see the books it recommends for me based on patterns it finds in my purchases and the purchases of other customers with overlaps with my purchases. And of course that increases revenue for Amazon, so that is smart all around.

How many companies do something totally amazing... twice?

Amazon has done it more than twice. They do it consistently and across their three core businesses.

I just mean in a general sense: e-commerce and now the cloud.

But they're both sides of the same coin for Amazon. They had to build up this infrastructure for their own use, and that's why they can do cloud hosting so efficiently. It's like when they had a9, which was a pretty cool search engine (I used it for a while); they've naturally got a really hard-core search engine in house that works on rich data just for their own mountain of data. It's the same for most of the other stuff that they've done.

I really think Amazon parallels Google in innovation, but they've somehow stayed much less sexy in the hacker world. I suppose it's that Amazon hasn't needed to make "we're smart people" one of their selling points.

I see what you mean about the infrastructure, and its a good point.

But as to love for Amazon... I dunno, most hackers I know have disdain for Google at this point, and much love for Amazon. But then... I am cloud obsessed, and this influences who I associate with :)

Amazon enables us. Google... not so much.

I completely agree. Just imagine Google announcing these same services... I would think that there'd be a massive amount of "Google Will Soon Host All the Internet" articles, and other such grandiose things. Like you said, Amazon is one of the largest innovators in tech, and they don't get remotely as much coverage as they should.

I guess the lesson is: A company's reputation, especially with the media, has a huge influence on the success of their products. So, try to be a media darling like Google or Apple.

Apple and Google have a long list of failed products that at least in apple's case was hyped by the media.

Here are some other facts about the two companies: Yesterday, APPL closed at 112.71, and GOOG at 362.74.

I still think getting positive publicity about a product makes that product more likely to succeed, and that there's not much harm in being a media darling.

“A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.” —John Gall

I like that this is one race where Google has to catch up to Amazon.

I don't agree. Though I think its great and technologically the next step forward this will never be a high margin business and will quickly turn into a race to the bottom as the switching costs are zero.

If you look at what they're doing, they've gone from offering raw storage and computers by the hour to offering DNA analysis as a service. They will keep building more sophisticated offerings. They're not racing to the bottom, they're racing to the top.

dna analysis is a niche, this is a service which by its nature is very easy to commoditize. To be useful it has to be invisible, and to be invisible it must have zero switch cost

DNA analysis is a niche, and they chose it because it is a commonly executed algorithm with a Hadoop implementation lying around. The point is that by including it they are showing that they will be offering more and more such algorithms for execution as implementations become available. First it was just machines, now they're running specific software and specific jobs on that software for you. The stack just keeps getting taller and nobody else's stack comes close. The more advanced the service you run, the more they are charging.

Amazon is winning by building more and more sophisticated services on top of their infrastructure to sell more and more use of the basic services.

I hate to sound repetitive, but everything about it is a race to the top, not the bottom.

how much of that stack is proprietary?

Every single technology they've built on here is open: 'PC hardware', Xen, Linux, Java, Hadoop, HTML, SOAP, Javascript, Web Services. The unique components are their grid technology and interface and that they have wrapped it all as a reliable, easy to use and all around great product. Thats why Amazon is so incredibly smart: they maximumize use of open source to create unique and valuable offerings that solve fundamental pain. Last week doing hadoop computes was a pain in the ass. This week its not. Value.

This is the most amazing thing going, Amazon is leading the market and continuously innovating, and you don't buy it because of 'switching costs.' Tell me the truth: do you have or are you studying for your MBA?

thats my point, its all based on an open stack. so while innovative and important (i no longer use my regular host) its reproducible. In fact there are already several players converging on this space (slicehost et al, appengine).

Long term this business is analogous to the hosting business: low margin and commoditized. is it a market? yes. Is it a big market? no. Why? because there is no lock-in.

If you can't understand this consider the scenario where 5 or 10 years from now Amazon is providing the infrastructure for 20% of the web, taking in 20% margins above the cost to operate the data centers and making a hefty profit. What stops Microsoft or Google or any other player for that matter from getting in on the action by undercutting them in price?

If you look at any (amazon scale) money maker online there is some sort of lock-in involved which is central to their business. Most often its a network effect of some kind. Go ahead, go down the list of the top 10 web companies and take a look yourself.

Just because something is technologically cool does not mean its going to make enough money to be significant to amazon's bottom line.

Also don't insult people if you can't grasp what they are saying.

In making this argument, you've missed the point of Amazon vs Google/Microsoft.

From http://www.wired.com/techbiz/it/magazine/16-05/mf_amazon?cur...:

Bezos doesn't disagree. "I'd be surprised if no one else does this," he says, pausing for effect. "It's a really good idea!" And there may be an ace up his sleeve. Any economist will tell you that a commodity business -- storing and processing data, for instance -- is a mug's game, with prices that plunge inevitably toward the cost of production (in the case of bits, pretty close to zero). That's music to Bezos' ears. "Commodity businesses don't scare us," he says. "We're experts at them. We've never had 35 or 40 percent margins like most tech companies."

What is Google's lock in? Anyone is free to write a better search engine and buy a bunch of hardware to throw at the problem. The switching cost is typing a different URL to search, or configuring your browser search box to use that engine.

The "write a better search engine" part is not easy, and Google's algorithms are proprietary, but their success is not really a network effect, is it?

Also, saying Amazon is providing "infrastructure" implies they are going to deliver their current product then stop. But they can continue to build higher levels on top of what they already have. Maybe next will be algorithms baked into Hadoop, so you can throw your data at it without writing any of your own code. They can keep building up like this indefinitely. If Microsoft had released the first version of Windows and stopped, it would have become a commodity eventually (even with network effects).

Cloudburst is the RMAP genetic search algorithm - and it is baked in already.

Sorry, I hate to be that guy and keep arguing, but I'll just say google does not fit here at all because: 1. there are huge up-front costs and a lot of IP to building a competing search engine (not the case here)

2. google is a consumer company where brand and user behavior are important (not the case here).

3. most importantly, they make their money with an ad network which has network effects and where they charge enormous margins

you'll notice in all the discussions between yhoo/msft the most important thing for both companies is their ownership share of the search ad network. this is key to the bottom line and long term relevance and its why you see proposals along the lines of we'll give you a bunch of money and a guarantee of a portion of revenues in exchange for ownership of your network

The point on rising above infrastructure only applies if they begin adding some sort of ip or secret sauce. Then they can get network effects if they have people building on a platform that is not easy to replicate, but we've seen nothing like that. Hadoop is not that.

You can install hadoop on your own machine today if you want: http://hadoop.apache.org/core/docs/current/quickstart.html

I understand thats harder and thats why consumers are using them, but its not hard in the sense of competing business.

In fact most large web companies _already_ do all of this stuff internally.

Obviously, failing to look ahead is a mistake. But it is also a mistake to project too far ahead. Remember Keynes' famous aphorism: "In the long run, we are all dead."

You're thinking in equilibrium terms. Yes, it's easy to imagine that one day this business will reach a low-margin equilibrium. Though that doesn't guarantee it will happen in our lifetimes. (An example of a factor that mitigates against it: Monopolies. Get a monopoly, and you can keep margins artificially high.) And just because something will eventually be commoditized doesn't mean it will never be a high margin business. Margins can be quite high at the beginning. And, if you stay out of the business because you're afraid of what will happen in ten years, you miss out on all that money that's on the table now. Just imagine if Microsoft had been unwilling to release Windows 95 because it was obvious that in ten years we'd all be trying to switch to Unix.

The electronic component industry knows all about this curve. Component manufacturers race to ramp up early production of a new part, because the part sells for higher margins when it is first to market. Within a year or two other manufacturers will clone your part, or invent alternatives, and your margins will drop to tiny levels. But by then you'll hopefully have ramped up something else. The business has worked that way for a long time.

In the case of this new Amazon service... I have never heard of Hadoop before. By releasing a turnkey service with big fanfare, Amazon will grow the market to include people like me. Because we have no clue what we're doing at first, they can charge us high margins for a standardized, widespread service with a relative wealth of documentation and buzz and (perhaps) some tech support. Eventually the market will stop growing, the customers will be educated and start hunting for bargains... and Amazon will move on to something else.

switching costs are zero

Amazon has my purchase history and preferences (about 10 years worth) as well as my trust and approval. Switching costs are not zero, in my case, at least.

Here's a really interesting part of the offering: Cloudburst RMAP DNA searches. A supercomputer by the hour for every biologist that wants one: that is revolutionary.

http://developer.amazonwebservices.com/connect/entry.jspa?ex...

The cloudblast project is here: http://apps.sourceforge.net/mediawiki/cloudburst-bio/index.p...

The paper on BlastReduce is here: http://www.cs.umd.edu/Grad/scholarlypapers/papers/MichaelSch...

Now someone needs to build better open visualization tools to handle the output of this system.

For a quick demonstration of why Amazon would do this, take a look at the old way: http://wiki.apache.org/hadoop/AmazonEC2

Also presumably they have the cluster well tuned, which is not simple, here is a relevant research paper about the scheduling algorithm which also goes into some of the tradeoffs and particularities of Hadoop on EC2: http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-99...

This can be pretty useful for data analysis and science. I don't see Amazon making money from this, but they must think the market is larger than I do. I may have to adjust my opinion.

Actually, the way it works they aren't dealing with a heterogeneous environment and a hadoop scheduler - it runs on an EC2 compute cluster just for you that they spawn on your behalf. Your job is the only job running on your compute cluster. One job per cluster. Sharing jobs would be better, and sharing a cluster with other users even better, but I don't see that it can do that yet.

I think that Amazon believes that making Hadoop easy to use and popular is the best way to sell more EC2 and S3 by giving people an easy way to burn cycles and fill gigabytes.

I think they're right.

Right. There are some EC2 specific issues though.

Most significant is the network speed. Small instances share a machine with other small instances so they can get more than their "fair share" of network capacity. So because of this the environment maybe effectively heterogeneous if some of the nodes are relatively starved for I/O because they are on a busy machine. Storing all the data on S3 makes this particularly relevant.

Since one of the defining features the the MapReduce model (as detailed by the 2004 Google paper: http://labs.google.com/papers/mapreduce.html) is that "moving computation is cheaper than moving data" the implementation of S3 becomes salient as well, where exactly is the data?

In a normal implementation of Hadoop one would run HDFS, which stores the data on the same machines that run the job. The scheduler attempts to minimize transfer AND transfer is still often the bottleneck when the data is shuffled for the reduce job. So interesting to see how this service compares.

EDIT: I haven't tried the new service though, but I will for sure after I get some sleep.

In other words...

MapReduce implementations have to take into account locality, i.e., the dataNodes have to be rack-aware for optimal I/O speed. Therefore, if their dataNodes are actually just cpu cores reading/writing blocks to S3 (other racks) then they are effectively getting kicked in the face.

How many people have dedicated hadoop clusters available to them in the first place? The point of the pattern, framework and product is to scale linearly and on demand, not to do the fastest computation per-machine possible.

As to IO - HDFS is going to divvy up the workload from S3 when the cluster bootstraps. After that the data will be local to each compute node, so they won't be getting kicked in the face. There is no reason to suppose that S3 replaces HDFS - its just used to load your data in.

Also, one would hope that their Hadoop setup IS rack aware - why shouldn't it be?

This is correct, in the job configuration only the initial data is loaded from s3, afterwards you can chain mapreduce steps that run off hdfs on the local machines.

Step 1: "-input", "s3n://elasticmapreduce/samples/similarity/lastfm/input/", "-output", "hdfs:///home/hadoop/output2/",

Step 2:

"-input", "hdfs:///home/hadoop/output2/", "-output", "hdfs:///home/hadoop/output3/",

etc..

There is no explicit support for mounting data from EBS yet, but in theory I think you can hack something together in the initial mapper to mount an EBS volume to be used in the second map task...

They still run HDFS, they just do it from S3. Presumably they are using EBS internally. This is exactly how you would do it if you set it up yourself... they're just doing it for you.

Small instances suck at Hadoop, you probably won't use them.

Moreover, Hadoop usage can be intense and it's sensible for Amazon to segregate it as a service to ensure quality and configure hypervisor's accordingly.

I don't see any evidence that Amazon has done this. It sounds like they're just using normal EC2 instances.

Does anyone see any obvious documentation on defining your own reduce function?

All the example use the built in "aggregate," which looks like it refers to the Hadoop aggregate sub-package. The lines output by the map functions start with the name of a Hadoop class within the aggregate package, which must tell it how to aggregate.

The PDF here seems to be a good start.

http://developer.amazonwebservices.com/connect/entry.jspa?ex...

The item similarity code example that uses the api has 4 python streaming mapreduce steps with custom reducers:

http://elasticmapreduce.s3.amazonaws.com/samples/similarity/...

AWS just put a pointer to the code bucket on s3, I'll add a link to a zip file with the full source including a script for converting the raw netflix data to something more friendly to Hadoop streaming.

The article is here:

http://developer.amazonwebservices.com/connect/entry.jspa?ex...

Check out the resources link at the end, the Cloudera Hadoop Hacks site has a few good intros to custom reduers in java, ruby, and python.

Here is a direct link to the full code example from the article:

http://elasticmapreduce.s3.amazonaws.com/samples/similarity/...

You will get much better performance out of Hadoop if you do this in Java, but I thought python would be more readable for a wider audience. Hadoop 18.3 doesn't have numerical sort in streaming (see https://issues.apache.org/jira/browse/HADOOP-2302 ), so the code is slightly more complicated than it needs to be. Dumbo will also probably be supported in the next release, then the code example would be even simpler.

Map/Reduce is really hard. As you can tell, this thing takes the operational pain out of Hadoop... not the thinkin part :) Beyond something like this: http://hadoop.apache.org/core/docs/current/mapred_tutorial.h... there isn't too much in the way of docs on how to think in map/reduce that I'm aware of, but I found the classes in the Pig tutorial to be quite helpful.

edit: oh, guy above me has great link!

Here is another great resource for how to "think" in terms of mapreduce (joins, keyswaps etc): http://www.cloudera.com/hadoop-training-basic

Great, except that it's no good for small jobs. Looks like if you want to run a 10-minute job you are charged for a full hour for each instance.

I remember one of their chief expenses a while back was under-utilization - they need enough capacity for the christmas rush etc, but it does unused during the slow times. It looks like they're letting other people pay for their under-utilized computing power and also deal with the utilization problem themselves beneath the 1 hour mark.

Every now and then I wonder if there's a market for someone to do something like app engine on EC2 in order to cater to the 10 minute job market.

With something like that you could just leave the EC2 instances up and just swap jobs in and out, track the time and charge some small markup for the convenience.

As long as the demand was there you'd just have to bring more instances up in peak times then let them die off one by one when things slow down. Put a nice client side API on it and use Amazons billing API so the client can handle all of their computing payments at the one point and you're ready to go.

That's a great idea.

It's alright as an idea, might not be too great as a business depending on the demand.

You'd need decent customer service when things went wrong or if people had questions about jobs / billing, so I imagine you'd need pretty high and reasonably consistent demand to cover that, especially since the markup would probably be tiny.

If anyone once to give it a try I'd like to see how it turns out for them.

If you have 1 10 minute job that runs on 1 machine, freaking run it on your laptop or desktop.

And if you spread your job across N machines so you get results in 10 min then be willing to pay for your impatience.

Deleted comment

Well, it does say "elastic"; perhaps you sometimes have huge batches to run, worthy of map/reducing, and sometimes have batches that could be run on a single machine instead, but since you're applying what's fundamentally the same transformation to both data sets, you only have one codebase, and so it would be nice if you could do it all with a single kind of infrastructure as well.

Actually, you can. And if you have small data, run the same job on one compute instance. On once instance, getting charged pennies per job from leftover EC2 time is not a big deal. If you do that kind of thing a lot instead of bigger jobs, map/reduce is the wrong way to solve your problem. Its for big jobs, not small jobs.

Bottom line of what this means: it took me hours to get scripts going to launch a cloudburst hadoop compute cluster on demand on Monday. And I've written such scripts before.

I just launched the SAME JOB in 15 seconds.

You know what else? I got distracted and left my own custom compute cluster on overnight and it cost me $50. Elastic Reduce shuts it down for me. (edited for clarity)

I was looking into building this. Amazon did it first. Glad I didn't put too much time into it.

Great point. Auto-shutoff makes up for the 15% pricing overhead. I've spent lots on forgetting to shutdown too. It should go in their marketing literature if they could figure out how to make it not sound like a negative for the normal service.

There's no way to automatically kill the cluster after the job is done? There definitely should be.

I believe rjurney ment his hand-made cluster from before didn't automatically shut down (as opposed to this new Elastic MapReduce feature which presumably would).

Right. The scripts mounted the data on EBS, configured a security group, etc. etc. but I didn't get around to automatic shutdown.

Elastic Reduce does that for you.

Although you can't do anything interesting like run Pig or Hive. 2.0 will be much better I'm sure.

There are ways, I just didn't get there yet since I just got it up and running and expected to wait until it was done.

http://groups.google.com/group/ec2ubuntu/browse_thread/threa...

My first thought was that http://www.cloudera.com/ was in big trouble but I actually think that this is the best thing that could have happened for them.

This is in reference to a previous thread, but interesting that 3 of the cloudera team have a degree in Mathematics.

http://www.cloudera.com/about

Maybe the question should be, does a Mathematician need to know programming, or can he just learn it on an as needed basis?

Yeah, pretty much like a Starbucks opening right next to your coffe shop. Suddenly every neighbor wants a cup of coffe, and you'll see increased attendance.

Cloudera offers are more like consulting. People need help setting up Hadoop jobs, understand MapReduce, or don't want to send data to S3.

Why is it >6x cheaper than the same EC2 instance?

Is it because they know that no one will just have it constantly run it as a true server like some do with EC2, so they'll get more fractional hours? Even though that's true to some extent, it doesn't seem like enough. Any insight?

EDIT: Twitter ftw! Apparently, the prices are additive, not replacitive. http://twitter.com/mndoci/statuses/1442516630

Does information beget information? If so, their model will probably prove quite lucrative.

Doung Cutting should be hired by Amazon!

Reply on news.ycombinator.com