Hacker News Reader

Comment by notacoward | Hacker News Reader

The increasing adoption of object stores and other not-quite-filesystems, even when a real filesystem would be the more appropriate choice, is definitely part of this. For good or ill, people would rather work on or with one of the simplified alternatives. I'm in that camp myself now, working on a system with lame HDFS-like semantics after two decades of proving the "POSIX can't scale" folks wrong. It's just not a battle worth fighting any more, at either the technical or business level.

View on HN

That's why I think MapR missed the mark. They had a software only solution one could deploy on a top of generic hardware in a data center to solve hundreds of TB to small number of PB storage problem that was real for enterprises and since those nodes would also be able to do compute jobs that was the way to get enterprises to adapt that technology. Going to object store was possible but it required dev time and in the game of build or buy the buy of the non-core tech is nearly always preferable. And, in case of MapR, it could be done immediately not after a massive refactoring to move to object store.

But the MapR sales people were impossible to deal with. They wanted to talk about Big Data. And all the wonderful things that we could do with it.

Me: I have lots of files. Think billions. Lets talk about what and how your stuff works to solve this problem because I'm hearing some people successfully used it on a million files scale.

Them: That's great. Let me tell you about big data that you can do using it.

Me: That's ok. I really want to solve my files problem.

Them: Big data is the future! You can change your application to do big data.

Needless to say it went nowhere fast.

Sounds familiar and not surprising. s/Big Data/Java/g. Their Chief Marketing Officer is the guy who was the CMO back then at those dusk years of Sun and presided over everything getting named "Java", even the things completely unrelated to Java ... even Sun's stock ticker got renamed into JAVA :)

This was exactly my experience with MapR. They refused to sell you the products you actually wanted.

My first response to this news was "good riddance", my second thought was "I hope they open source the storage stuff."

Interesting post. Out of curiosity: aren't there others who have developed cluster storage exposed as NFS / FUSE?

Absolutely. In fact, that was my last project, which I worked on for almost a decade. Unfortunately, it's a tough market. At the low end, everyone thinks they can do it themselves. At the high end, you're up against either the "POSIX is dead let's use an object store" crowd or the enterprise EMC/NetApp/Isilon crowd. On the other side of the ledger, development costs are high. Developers are harder to find, more equipment is necessary, and the testing bar is higher, compared to other kinds of software. Low revenue + high cost = life sucks. That's why I work on a bespoke not-quite-filesystem now.

Thank you very much for taking the time!

They all sucked. MapR sucked the least. Considering a comparable EMC solution was ~20x more expensive and less flexible, MapR happened to be in the right place at the right time with the right software but blew it because they decided they knew better what the customers needed to buy vs what the customers wanted to buy.

Fundamentally, file system clusters are a very difficult business to get into and make enough money to justify being in that business. On a low end there's open source stuff that kind of sort of works. Selling consulting on a top of it is at best a ramen-profitable business. No one is buying a Ceph-consulting for a million dollars if they run a business that needs that size of storage -- the data is too valuable to do a semi-custom solution and be a test case for Ceph, Gluster, etc.

On the top end there are EMCs of the world with $350K/node pricing + yearly 10% support contract. Finally, there's a "rewrite the app" alternative ( call it 1 year, million $ price tag to move to object store ) that a customer company is always considering.

So the sweet spot is enterprise solution with a license cost of about $12k-$24k per year per node which has EMC/Isilon/NetApp-like functionality that unbundles software from hardware. The tricky part is that it needs to be a proven solution that works on the comparable EMC/Isilon/NetApp sized deployments. To do that one needs to have a lot of excellent engineers that cost a lot of money, a lot of sales engineers that intrinsically understand the commonality between the customers' requirements and can explain it to those engineers and a lot of very expensive test beds.

I remember being in a pitch meeting with MapR a few years ago. It was for a large academic institution’s HPC group. We were all very interested in their NFS/HDFS tools, which would have met a very critical need. They just kept talking about the Hadoop workflows, which wouldn’t have flown at all for this group. Most HPC workflows/tools don’t map very well to Hadoop or at the minimum would have required tools to have been rewritten.

We really wanted to spend the money, but as far as I remember, we didn’t end up doing anything more than a test install.

I’d still love a middle option that could handle petabyte scale storage without the Isilon price tag.

CERN?

No, this was at Stanford a few years ago, so not close to CERN level HPC.

Our lab has modest compute requirements, but need a ton of storage, so we were very interested in a mid-range storage option.

Thanks for taking the time! Its fascinating learning about this in an area (storage) I know nothing about.

The sweet spot you describe sounds like Quobyte to me.

...except for the "proven solution" part, Quobyte marketing VP.

Honest question - what constitutes "proven"?

Someone else that I know of a similar scale has used it successfully and is happy with it.

Nutanix and VMWare have been doing that pretty successfully since ~2012.

Reply on news.ycombinator.com