Big Data Analytics Systems: What Goes Around Comes Around | Hacker News Reader

Big Data Analytics Systems: What Goes Around Comes Around

slideshare.net

32 pointsmrry18 comments

Oh, it's coming around, alright.

https://www.usenix.org/conference/hotos15/workshop-program/p...

On slide 58, it's worth noting that Greenplum doesn't belong to EMC any more. It's been spun into Pivotal's Big Data Suite[0].

Disclaimer: I work for Pivotal Labs, another part of the same company.

[0] http://pivotal.io/big-data/pivotal-big-data-suite

What are people actually using these days for analytics? What is missing? (i.e. what would the dream startup do for you?)

There's a lot of hype around certain technologies, and I suspect it does not reflect reality. I've used hadoop+pig a few years ago and it was absolutely terrible.

I'm working on moving some analytics from pig + hadoop streaming over to Redshift.

The debug cycles are so much faster with sql queries.

You can use sql on hadoop...

I am assuming you tried Hive, Impala, Spark SQL ?

Redshift seems like a great fit for Adhoc analytics. Just a shame you can't run it in your own data centre (some of us can't use the cloud).

You can use citusdb (postgresql, columnar). Not the same, but close.

If you are using SQL... why didn't you try a columnar engine?

Redshift is a columnar engine

To all those who have asked, yes, hive is also in use.

The majority of big data analytics is using the standard Hadoop stack and testing Spark.

Dream startup would be to make the day to day life of running a cluster more manageable. If I had an app that could tell me how to optimize queries, tune the parameters and make the what/why/where/how questions easier I would buy it instantly.

Look no further. Netezza by IBM dies that all with SQL. Seriously, I'm impressed by that product. It also supports Hadoop.

I absolutely love Spark and really look forward to SparkR joining the main project this summer.

If the data is structured, columnar databases still rule.

But if you are stuck in SV's echo chamber it's all MapReduce/Hadoop/Spark/etc.

How about Spark + Parquet via SQLContext / SchemaRDD?

The overhead of that must be ridiculous.

What kind of overheads should we be aware of? Spark can run on a single machine, it could be an interesting benchmark to compare against Vertica or similar products. I've seen similar systems tuned for under 1s response times, even when running in distributed mode, though they were proprietary.

Wrong. Big Data is not some analytics system. It's a method to cure cancer, stop terrorism and end world hunger.

Honestly if you google "big data" + "cure cancer" or any of the things above you'll get an article about it. Big Data is over-hyped.

Reply on news.ycombinator.com