Hacker News Reader

Comment by Joeri | Hacker News Reader

So, that would mean the hadoop distro market would be down to just cloudera? A single vendor market is not a market, it's a legacy product.

View on HN

Everybody is moving to the cloud so EMR, Dataproc, Dataflow, etc. are still going to be around

Plus everyone is moving away from Hadoop and HDFS, so it's Spark, S3, GCS, Databricks, etc

AFAIK, AWS's EMR and Glue leverage Hadoop under the covers. Does that count towards that market? and/or at least keeping Hadoop development active?

Hadoop ≠ Hadoop distro. Hadoop is fine and healthy; what is in question is whether larger repackagings of it with other vendor-specific components is an interesting product any more.

Google's dataproc and Azure's HDInsight as well. The cloud players are swallowing whatever is left of the hadoop market.

EMR is their managed distro of Hadoop.

DataPipeline is their managed Hadoop for ETL.

Glue ETL is their managed Spark.

And it all absolutely counts towards the market.

Is Microsoft Office or Microsoft Windows a legacy product?

Azure's HDInsight all-in-one suite uses it.

HDInsite is built by hortonworks which was acquired by cloudera.

In addition to HDInsight being cloudera, it's also basically a legacy product. Azure technical solutions people are now all pushing databricks. And personally, when designing a new project, I don't see why you would use HDInsight instead of a bespoke distributed vm architecture with one (azure storage) or two-tier storage (one local to the vm, one in Azure storage) unless you really need to use some hadoop jar you found lying around.

Actually, there is another vendor. A new one - www.logicalclocks.com - we are selling a next-generation version of Hadoop, with consistent distributed metadata for HDFS, it's built on TLS/SSL (not kerberos), and it's laser focussed on Data Science. We have customers with racks with 10s of GPUs training models with 10s of TBs of feature data. And we also built the industry's first enterprise feature store for ML on this platform.

References:

https://www.logicalclocks.com/feature-store/

https://www.logicalclocks.com/introducing-hops-hadoop/

Reply on news.ycombinator.com