Hacker News Reader

Comment by pinewurst | Hacker News Reader

At this point, this is really an apples-to-oranges comparison.

Lustre, as truly awful as it is, is a POSIX filesystem (or close enough for (literally) government work).

Redhat/Ceph only at the end of April, announced that POSIX functionality was ready for production. Personally, that's not when I'd choose to deploy production storage. Ceph object and nominally block have much more time in production.

If you need POSIX, trusting Ceph at this point is an issue unless, as you say, MTBF isn't a concern. You might want to try BeeGFS, a similar logical model but much simpler to implement, performance up to a very high level, and a record of reliable HPC deployments (as oxymoronic as that sounds).

If you can do with object then certainly exorcise Lustre from your environment in favor of Ceph (or try Scality if non-OS isn't an issue). Lustre's only useful as a jobs program anyway - keeping people occupied who'd otherwise be bodging up real software.

View on HN

A proprietary (and solid) alternative to Lustre would be GPFS, which also has a long track record in HPC (and other markets in which IBM thrives).

As someone who completely shares your Lustre sentiment, I can't fathom why Intel keeps pouring resources into it.

GPFS has an amazing number of features, offers high performance and, given a certain fiddliness of configuration and administration, is reliable and performant. It can even sit on top of block storage that itself manages with advanced software RAID and volume management.

The problem (surprise!) is IBM. It's mature software, which means 21st Century Desperate IBM sees it as a cash cow - aggressively squeezing customers - and as something they can let their senior, expensive developers move on from - or lay them off in favor of "rightsourcing". You can certainly trust your data to it (unlike Lustre), but it'll be very expensive, especially on an ongoing basis, and the support team isn't going to know more than you by then. Also expect surprise visits from IBM licensing ninja squads looking for violations of the complex terms, which they will find.

As for Lustre, it brings to mind Oliver Wendell Holmes, Jr's, "Three generations of imbeciles are enough". I've been at least peripherally involved with it since 1999, with LLNL trying to strong-arm storage vendors into support. Someone should write a book following 16 years of the tangled Lustre trail from LLNL/CMU/CFS -> Sun -> Oracle -> WhamCloud -> OpenSFS -> ClusterStor -> Xyratex -> Seagate -> Intel (and probably ISIS too).

The answer to your question IHMO, is that Intel just isn't that smart. They're basically a PR firm with a good fab in the basement. What do they know about storage or so many other things? People don't remember when they tried to corner the web serving market back during the 1st Internet boom. They fail a lot, but until now had enough of a cash torrent coming that it didn't matter. They still do, of course, but there are inklings of an ebb.

Yeah, yeah, GPFS was one of them that inspired my HPC and cloud comparison. It, combined with management software, got one to about 80-90% of what they needed for cloud filesystems. It was badass back when I read about it being deployed in ASC Purple. I didn't know it turned into some stagnating, fascist crap with IBM. Sad outcome for such great technology.

Typical IBM, though. (shakes head)

Sounds like good recommendations based on what research I've done in these things. I forgot BeeGFS but it was in my bookmarks. Must be good in some way. ;)

Cheers!

Reply on news.ycombinator.com