Cloudera announces real-time query engine for Hadoop | Hacker News Reader

Cloudera announces real-time query engine for Hadoop

marketwatch.com

56 pointsTallGuyShort17 comments

"Cloudera Impala: Real-Time Queries in Apache Hadoop, For Real" http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-t...

Impala source code: https://github.com/cloudera/impala

Impala documentation can be found here: https://ccp.cloudera.com/display/IMPALA10BETADOC/Cloudera+Im...

Metamarkets also open sourced their data store today. High performance, distributed query database.

http://metamarkets.com/category/technology/druid/

This looks very interesting. I've got multiple PB of log files that I'd like to be able to do real-time and batch processing on. Log files are being generated at a small number of Gbps.

Anyone else got that much data and have a good solution?

I don't think anyone has managed to get interactive speeds (~seconds to max couple of minutes) at the PB scale.

What kind of analysis are you trying to do on that dataset? Log files can be verbose as well. Do you need all the data in the logs or can you cut down the data in each row to only what you need?

I noted that Marcel Kornacker, the architect of Impala, was the lead developer for the query engine of Google’s F1 project. In the end of the article:

http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-t...

So, what is the key innovation that changes hadoop from batch processing to real time. Is it similar to google's dremel?

This doesn't change Hadoop from batch processing to real time - it's a new query engine that uses the same data sources/formats as Hadoop and the same interfaces as Hive. So yeah, similar idea to Google Dremel / F1.

You're right that the data is still stored in HDFS, and you can access the data that was originally accessed in Hive. However, because of the way Impala works, it does in fact provide a real-time query interface for data that is sitting in Hadoop (HDFS). The back-end is written in C++ and accesses/processes data much differently than MapReduce which allows for some pretty awesome performance numbers :-)

It isn't like F1 at all... F1 is a multi-datacenter, transactional datastore and SQL, that google uses to replace mysql.

This aint nothing like that.

Correct, the technology is based on Google Dremel.

My reading was that columnar storage was a critical piece of Dremel. "Accessing the same data as Hive" doesn't sound like Impala is doing that. Is that optional then or is this just a query engine?

I can't say for sure, but I believe Doug Cutting's Trevni format is already in the beta but it's a slow version. Impala is initially focusing on architecture, I'm sure columnar format will have attention eventually.

https://github.com/cutting/trevni

But google's dremel actually doesnt use Map-Reduce at all. So is this using just the HDFS and a new layer along side map-r that is using the HDFS?

Impala does not seem to be using MR at all, see note: https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Freq...

Interesting, I wonder how this will be different than Apache Drill, championed by Ted Dunning (MapR)

They're both Dremel clones, but this one seems to be a lot more complete (as Drill is just getting started).

Reply on news.ycombinator.com