MapR may shut down as investor pulls out after ‘extremely poor results’ | Hacker News Reader

MapR may shut down as investor pulls out after ‘extremely poor results’

siliconangle.com

108 pointsheyyyouu82 comments

As many others pointed out, these product vendors are getting killed by the cloud (I am not saying whether it is good or bad). On-demand compute and storage scalability of cloud makes perfect sense for Big data infrastructure.

These companies are failing because they can't establish a use case of running their product on cloud. It is ironic because when AWS started EMR, MapR was one of the distributions they offered via EMR[1]. Over period of time, they weened off MapR and stopped offering them as deployment option. So it is another case of AWS cannibalizing its own third party ecosystem.

Once this was gone, MapR got reduced just another marketplace partner [2] and their additional licensing cost didn't make sense when native EMR was sufficient for most use case.

[1] https://aws.amazon.com/emr/mapr/pricing/ [2] https://mapr.com/partners/partner/amazon-elastic-mapreduce-a...

Which has to be a gut punch because EMR feels like a bag of bolts rather than any kind of polished system.

And MapR is so much more efficient than vanilla Hadoop, AWS definitely makes more money off the less polished yet mainstream version.

IMO it is not a fair comparison between EMR and MapR flavor of hadoop as they are designed for different usecases. MapR and other flavors of hadoop are engineered to make the task of administering a long running cluster easy. In comparison, EMR has a rudimentary set of tools for peaking into a running cluster.

Whereas AWS offering is ideally suited for transient clusters. It relies on other solutions like Athena, Redshift Spectrum to cater for ad-hoc use cases such as querying and reporting. In this regard, EMR has much better support for programmability and elastic resource provisioning which is really important for transient clusters.

I think we are talking past each other. MapR is a highly tuned system compiled to native binaries. It has nothing to do with the emphermality of the clusters, it has to do with job runtimes.

To me, and a few people that I know, the most interesting part of MapR was their NFS server over multiple nodes. It solved a real problem that real customers (i.e. enterprises with money for whom tech made stuff work and not was work in itself) were willing to pay a lot of money for ( see Isilon, EMC, NetApp, etc ) but that was a non-sexy part of the MapR business -- Big Data part was much sexier.

Over the last 5 years the real money making enterprises solved their "it should look like a big file system" problem so "the rest of our stuff works" issue stopped being an issue ( in a process Isilon got bought by the EMC, EMC got bought by Dell ) either by buying those specialized solutions, moving to object store or building home grown systems that worked with their specific applications and big data people went with newer, more shiny big data solutions, leaving MapR with no market.

Which of the shinier, newer solutions do you think people have found more attractive in particular? Genuinely curious, thanks!

I'm not someone very much in the "big data" [0] but from what I see streaming analytics especially Spark-based solutions are eating the world that used to be dominated by Hadoop.

I also think cloud based data lakes and tooling around them seriously decreased the appeal of Hadoop.

[0] If I can fit your big data into my memory, it is not "big data".

i mean spark is often deployed on top of yarn where it's still in the "hadoop" sphere.

But I'm not loading up on my SuperMicro, Dell or HP servers. So there's no need to buy a decent Hadoop implementation from MapR.

Spark is very much part of Hadoop. It uses Hadoop libraries throughout it including for the core part of reading/writing data

And that adage of big data can't fit in memory is nonsense these days. We run clusters with hundreds of terabytes of RAM which is very much big data. It's pretty easy and affordable with the cloud.

When was the last time you bought a MapR license to use it in a cloud?

When MapR was selling them the M5 and M7 were deployed in a data center on servers.

The increasing adoption of object stores and other not-quite-filesystems, even when a real filesystem would be the more appropriate choice, is definitely part of this. For good or ill, people would rather work on or with one of the simplified alternatives. I'm in that camp myself now, working on a system with lame HDFS-like semantics after two decades of proving the "POSIX can't scale" folks wrong. It's just not a battle worth fighting any more, at either the technical or business level.

That's why I think MapR missed the mark. They had a software only solution one could deploy on a top of generic hardware in a data center to solve hundreds of TB to small number of PB storage problem that was real for enterprises and since those nodes would also be able to do compute jobs that was the way to get enterprises to adapt that technology. Going to object store was possible but it required dev time and in the game of build or buy the buy of the non-core tech is nearly always preferable. And, in case of MapR, it could be done immediately not after a massive refactoring to move to object store.

But the MapR sales people were impossible to deal with. They wanted to talk about Big Data. And all the wonderful things that we could do with it.

Me: I have lots of files. Think billions. Lets talk about what and how your stuff works to solve this problem because I'm hearing some people successfully used it on a million files scale.

Them: That's great. Let me tell you about big data that you can do using it.

Me: That's ok. I really want to solve my files problem.

Them: Big data is the future! You can change your application to do big data.

Needless to say it went nowhere fast.

Sounds familiar and not surprising. s/Big Data/Java/g. Their Chief Marketing Officer is the guy who was the CMO back then at those dusk years of Sun and presided over everything getting named "Java", even the things completely unrelated to Java ... even Sun's stock ticker got renamed into JAVA :)

Interesting post. Out of curiosity: aren't there others who have developed cluster storage exposed as NFS / FUSE?

Absolutely. In fact, that was my last project, which I worked on for almost a decade. Unfortunately, it's a tough market. At the low end, everyone thinks they can do it themselves. At the high end, you're up against either the "POSIX is dead let's use an object store" crowd or the enterprise EMC/NetApp/Isilon crowd. On the other side of the ledger, development costs are high. Developers are harder to find, more equipment is necessary, and the testing bar is higher, compared to other kinds of software. Low revenue + high cost = life sucks. That's why I work on a bespoke not-quite-filesystem now.

Thank you very much for taking the time!

Nutanix and VMWare have been doing that pretty successfully since ~2012.

They all sucked. MapR sucked the least. Considering a comparable EMC solution was ~20x more expensive and less flexible, MapR happened to be in the right place at the right time with the right software but blew it because they decided they knew better what the customers needed to buy vs what the customers wanted to buy.

Fundamentally, file system clusters are a very difficult business to get into and make enough money to justify being in that business. On a low end there's open source stuff that kind of sort of works. Selling consulting on a top of it is at best a ramen-profitable business. No one is buying a Ceph-consulting for a million dollars if they run a business that needs that size of storage -- the data is too valuable to do a semi-custom solution and be a test case for Ceph, Gluster, etc.

On the top end there are EMCs of the world with $350K/node pricing + yearly 10% support contract. Finally, there's a "rewrite the app" alternative ( call it 1 year, million $ price tag to move to object store ) that a customer company is always considering.

So the sweet spot is enterprise solution with a license cost of about $12k-$24k per year per node which has EMC/Isilon/NetApp-like functionality that unbundles software from hardware. The tricky part is that it needs to be a proven solution that works on the comparable EMC/Isilon/NetApp sized deployments. To do that one needs to have a lot of excellent engineers that cost a lot of money, a lot of sales engineers that intrinsically understand the commonality between the customers' requirements and can explain it to those engineers and a lot of very expensive test beds.

Thanks for taking the time! Its fascinating learning about this in an area (storage) I know nothing about.

I remember being in a pitch meeting with MapR a few years ago. It was for a large academic institution’s HPC group. We were all very interested in their NFS/HDFS tools, which would have met a very critical need. They just kept talking about the Hadoop workflows, which wouldn’t have flown at all for this group. Most HPC workflows/tools don’t map very well to Hadoop or at the minimum would have required tools to have been rewritten.

We really wanted to spend the money, but as far as I remember, we didn’t end up doing anything more than a test install.

I’d still love a middle option that could handle petabyte scale storage without the Isilon price tag.

CERN?

No, this was at Stanford a few years ago, so not close to CERN level HPC.

Our lab has modest compute requirements, but need a ton of storage, so we were very interested in a mid-range storage option.

The sweet spot you describe sounds like Quobyte to me.

...except for the "proven solution" part, Quobyte marketing VP.

Honest question - what constitutes "proven"?

Someone else that I know of a similar scale has used it successfully and is happy with it.

This was exactly my experience with MapR. They refused to sell you the products you actually wanted.

My first response to this news was "good riddance", my second thought was "I hope they open source the storage stuff."

We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void.

The question is going to be: will anyone provide an intelligent way to maintain compatibility with applications that expect to interface with HDFS rather than POSIX? It's a bit of a gap right now from what we see.

Note that apparently the layoffs have already happened. Engineers are already interviewing elsewhere.

Hadoop's whole appeal was it was cheap and scalable. Did it actually work without serious engineering teams maintaining each distribution? absolutely not. the hype was purely VC funded.

fast forward 10 years and hadoop has basically been killed off by hosted storage services that are more expensive but 10000x easier to manage.

I'm not really familiar with this industry. Are these Hadoop clusters hosted locally by these businesses? And MapR et all were providing the software/consulting to help manage them?

Then I'm guessing better cloud hosted options came out offering similar capabilities.

If that's the case was it really a big surprise that "cloud" hosting would eat any self-hosted platform's lunch?

yes mostly locally hosted back then. the idea was you'd have this massive distributed file system to hold all your data, and VC backed comopanies like Cloudera, MapR, Hortonworks promised everyone that they'd build the SQL layer, the data warehousing, BI, and all the other enterprise features you'd need to basically replace your expensive Teradata, Oracle Exadata, and other data warehousing systems.

But of course it didn't play out that way.

Actually it did play out that way.

Most enterprises these days will have a massive distributed file system (S3 or equivalent) that has most of their data. And most will be running a decent percentage of data transformation jobs using Spark or some SQL layer e.g. Presto and then running BI tools like Tableau using Athena/Redshift Spectrum (or equivalent) as their SQL layer.

It's just that this is all playing out on the cloud instead of on-premise with some vendor. But you have definitely been seeing the decline of the core enterprise data warehouse.

So in the very recent past (i.e. a few years) businesses wanting to do Data Science ran a distro of Hadoop e.g. MapR that they bought from the vendor. They charged an exorbitant charge per node (e.g. $10k) because they figured they would get people to switch from Teradata or Oracle.

Now what the cloud offered was so much more compelling. You had per hour pricing on the order of $20 for a minimal cluster. You had unlimited autoscaling so you didn't have to do capacity planning and go through procurement processes to pre-order hardware/licenses. And of course you had unlimited, ultra-cheap storage courtesy of S3. And it also allowed each team to have their own mini-cluster instead of everyone relying on some giant one.

I don't think the recent explosion in Data Science would've happened without the cloud.

Which is hilarious, given how expensive it is to run analytics on AWS. My current company looked into recreating our internal analytics environment on the cloud, and realised that it wouldn't be cost effective.

Yes, MapR was local hosting.

Do you actually work in this industry ?

Because Hadoop has not been killed off. In fact it is far bigger than it has ever been and growing each year. It's just that it is all transitioning to the cloud with parts like HDFS being replaced with more scalable solutions like S3/EMRFS.

But you can't say it's been killed off when on AWS you have managed Hadoop (EMR), managed Hadoop pipelines (DataPipeline) managed Spark (Glue ETL), managed Hive Metastore (Glue Catalog) etc. And similar on Azure or GCP.

> Because Hadoop has not been killed off.

> HDFS being replaced with more scalable solutions like S3/EMRFS.

Thank you for re-affirming my point.

I'm not going to go into the typical HN "I'm going to nitpick your point down to the bone and argue you over the pointless details", but 2012 Hadoop is not the same thing as the tools you're describing.

Except you’re not being just a bit inaccurate, you’re flat wrong.

Simply put, Hadoop is not HDFS, it is a much larger Apache ecosystem that includes Spark, Hive, MR, NiFi, etc. All of which are doing fine.

I’m not sure how common that usage is . ‘Hadoop Ecosystem’ surely encompasses Spark and the rest, but I’d argue ‘Hadoop’ most commonly refers to MR, HDFS, and maybe YARN. Which is only to say I’d be surprised to hear an application described as ‘using Hadoop’ and find it using Spark on data in S3

Spark is NOT hadoop. You don't even need hadoop for spark anymore. Spark is more or less completely independent from the hadoop ecosystem.

Just because they're compatible doesn't mean they're synonymous. You can use spark with mesos or kubernetes instead of YARN.

So I deal with hundreds of Data Scientists and none of this is true for us.

Hadoop vendors are simply getting killed by the cloud. AWS EMR, Azure ML, Google Dataproc. And many people are simply swapping HDFS for S3 or similar object stores. Never seen anyone switching to very expensive, more HPC style NVME setups. That seems strange given the data sizes we are working with.

And 95% of what Data Scientists do is ETL, Data Curation, Feature Engineering etc and which Spark is still the overwhelming dominant player. And Spark uses Hadoop filesystem API underneath which is trivial to replace hdfs:// with s3://

It really depends on what the workload is and if performance or scale are priorities. Our team and the products we build come from the HPC side of the fence and we've been engaged with all sorts of large businesses that know their performance requirements (think 100s of GB/s read bandwidth), and it leads them down the path of either HPC filesystems or newer NVMe-oriented filesystems. Does 'AI' or 'data science' always mean 'high performance'? No, but it definitely can.

Everyone's data volume is different as well...working a proposal for several petabytes of NVMe storage right now. It's not everyone, but use cases are out there for large volumes of high performance storage.

Fair point on the ability to swap the hdfs interface trivially, I hope it's that simple everywhere. Another issue we run into is that these companies have invested heavily in these Hadoop clusters, and would prefer to continue to get some use out of the mountains of HDDs that are captive in these environments. So tiering/HSM functionality is another facet of the issue that these environments will anchor for quite some time.

The problem you have with Weko and those newer filesystems is that the tooling hasn't caught up. They are trying to get their APIs upstreamed into TensorFlow. But what about Spark, Pandas, Arrow, etc? It's not enough to have just the training part of the pipeline - you need to be the filesyste for the whole pipeline. That's what we provide with HopsFS.

Wow... how do you achieve petabytes of NVMe? like, what is the biggest drive that you can get today?

And then you just plug a ton of them into what kind of "motherboard" to interconnect them? I guess Infiniband based bus? or I am totally off-base here?

Any more details are appreciated, if only to satisfy my curiosity :)

Dell sells poweredges with 3.84 TB nvme drives, some of the 2U servers can pack 24 IIRC. If you do RAID10, 1PB is “only” 100U roughly.

I literally can't remember how many years it has been since the last time I heard something positive about Hadoop.

It's at least 4.

We build a next-gen version of HDFS (HopsFS) that has distributed, consistent, transactional metadata where small files (<1MB) can be stored in NVMe disks in the metadata layer. And it's open-source - cf WekaIO, etc. And it's performance is backed by peer-reviewed publications at tier-1 conferences (Usenix fast, ACM middleware, etc).

https://www.logicalclocks.com/millions-and-millions-of-files...

Our business model is to build a data science platform, Hopsworks, around our distributed metadata layer. And yes we use YARN (training models) and also Kubernetes (serving models). The choice of resource mgr is really just an implementation detail, as the platform is backed by a REST API.

Very interesting! I did not know that this existed. I know some people who will be interested.

Some background reading/viewing: https://www.logicalclocks.com/eventscustom/

And it works in the cloud - HA and high throughput. On Spotify's Hadoop workload, we got 1.6 million file system ops/secs, highly available over 3 availability zones. On GCE.

Lots of interesting discussion here about billions of files and what not. I don't really work with big data -- the largest amounts of data I work with are in the range of tens of millions at most, and my modest "ETL" is about reading from APIs and cleaning up incoming data into consistent schemas, refining data (e.g. geocoding addresses, or correlating against public demographic records/statistics), a process that is super fast and doesn't even require multiple nodes for the most part -- so I never encounter anything I can't solve with some Postgres or Redis. What I'd love to know is what everyone is actually doing with their huge clusters. I can understand the need for big data in hard sciences (CERN, or genetics, or similar), and of course machine learning (e.g. image feature extraction), and then there are real time auction use cases like ad servers that probably have some big data component. What else are people doing out there?

This is a great question -- I'd love to know this too. I watch this space from the enterprise perspective and I see a lot of big data talk/prep but not so much execution, if that makes sense.

MapR has been reduced.

Hadoop was the floppy disk of big data. Ubiquitous, but always beat by other solutions.

This is simply not true. Hadoop is growing massively.

All that is happening is that the vendors are being killed by the cloud.

Any impacted engineers looking for a new home in the Bay Area or Seattle, feel free to reach out moonjo@amazon.com to learn more about teams with Redshift.

So, that would mean the hadoop distro market would be down to just cloudera? A single vendor market is not a market, it's a legacy product.

AFAIK, AWS's EMR and Glue leverage Hadoop under the covers. Does that count towards that market? and/or at least keeping Hadoop development active?

Hadoop ≠ Hadoop distro. Hadoop is fine and healthy; what is in question is whether larger repackagings of it with other vendor-specific components is an interesting product any more.

Google's dataproc and Azure's HDInsight as well. The cloud players are swallowing whatever is left of the hadoop market.

EMR is their managed distro of Hadoop.

DataPipeline is their managed Hadoop for ETL.

Glue ETL is their managed Spark.

And it all absolutely counts towards the market.

Everybody is moving to the cloud so EMR, Dataproc, Dataflow, etc. are still going to be around

Plus everyone is moving away from Hadoop and HDFS, so it's Spark, S3, GCS, Databricks, etc

Is Microsoft Office or Microsoft Windows a legacy product?

Azure's HDInsight all-in-one suite uses it.

HDInsite is built by hortonworks which was acquired by cloudera.

In addition to HDInsight being cloudera, it's also basically a legacy product. Azure technical solutions people are now all pushing databricks. And personally, when designing a new project, I don't see why you would use HDInsight instead of a bespoke distributed vm architecture with one (azure storage) or two-tier storage (one local to the vm, one in Azure storage) unless you really need to use some hadoop jar you found lying around.

Actually, there is another vendor. A new one - www.logicalclocks.com - we are selling a next-generation version of Hadoop, with consistent distributed metadata for HDFS, it's built on TLS/SSL (not kerberos), and it's laser focussed on Data Science. We have customers with racks with 10s of GPUs training models with 10s of TBs of feature data. And we also built the industry's first enterprise feature store for ML on this platform.

References:

https://www.logicalclocks.com/feature-store/

https://www.logicalclocks.com/introducing-hops-hadoop/

Weren't they killed the moment Spark appeared, having heavily invested in accelerating Hadoop? I am surprised that Cloudera is still around...

And spark was sort of killed by Storm / Heron as even Spark Streaming does microbatches and isn't truly realtime.

Wasn't it the other way around? Wasn't Storm killed by Spark? At least once Spark was out and gained traction, nobody in my bubble talked about Storm anymore. But I guess it depends on the bubble you're in.

yeah their timeline seems backwards

Wouldn't Flink be more likely to replace Spark? I don't hear about a lot of people using Storm these days, but I guess a lot of this depends on whatever bubble you happen to be in.

in what world was spark killed by storm/heron? last i checked the spark conference had over 10k attendees this year. storm is barely supported and heron never picked up steam.

What ?

Spark Streaming is just one part of Spark and the far less used part as well.

Spark hasn't been killed by anything and is still heavily used for ETL and ML.

No one uses Storm anymore. It was Spark that killed Storm, not the other way around.

No the cloud wiped them out.

Why spend $10k per node in an upfront licensing deal when you can buy it from AWS for a few dollars an hour.

This is sad. I remember the MapR demo at the NVidia GTC a couple years ago, and their demo was amazing. I hope they pull a rabbit out of the hat and are able to continue, but if they don't then I hope they Open Source as much as they can.

I think they'll be fine. At my last company we had a large MapR cluster and it was hands down more reliable and user-friendly than anything else. Maybe Cloudera will pick them up.

We've used them as a more reliable/resilient/replicated (with a lower operational effort) provider of HBase tables and Kafka streams, and a bit of file storage.

Hope they're able to open source some of their tech if this does come to pass.

Reply on news.ycombinator.com