Lustre is a monstrosity - badly designed, poorly implemented, very hard to configure and keep running or even get adequate performance under other than a single limited use case.
Good riddance to bad rubbish.
Hmm?
From the article: " In the core HPC market, Lustre has 75 of the Top 100 systems on the Top 500 rankings, and is used on nine of the top ten systems."
That is a strong argument for it being performant. That 'single limited use case' you mention is... high performance computing.
it definitely served a need that nothing else did, but from an armchair position we could collectively do better
And no, "single limited use case" I mean, is a relatively small number of very large files that need to be sequentially streamed out/in as fast as possible. That's a small subset of HPC.
GPFS is the gold standard but it's expensive.
Lustre is more a byproduct of the HPC vendors trying to get more margin by using open source. They've drunk from the poisoned chalice in that the amount of time, money and effort required to get Lustre to acceptability (limited as that may be) has been far more than GPFS license/support cost. Even from the brain-eating zombie that is today's IBM.
Confirming! And going on a small tangent (ha!).
Our previous configuration was Lustre and XFS/NFS; the former was the scratch file system for HPC applications and the latter for home directories and what not.
Lustre was definitely a beast (in a good sense), but we'd occasionally get bit by a work load high in metadata operations - which would bring Lustre to its knees due to latency.
XFS/NFS was great for its purpose (no HPC workloads), but we'd also get bit by a user or users reading seemingly tiny, innocuous files (inadvertently in parallel) which would cause load averages to spike; surprisingly the latency wasn't as bad as the Lustre workload mentioned above.
Not too drink the GPFS kool aid here, but it's definitely solved both problems above. It has its issues, but definitely handles the common I/O patterns seen on our cluster.
At least the Lustre developers (pre-Intel) had the foresight to enable extremely good debugging - you could simply enable a few procfs settings and easily find the offenders.
It's still amusing to me however, that the biggest offenders of Lustre slow-downs were single core processes. I'd check the RPC counts per node, find the violator, and then check per PID statistics; it was always (95%+) a single core application performing thousands of calls.
We never did experiment with the 2.x branch of the software. I recall one of our co-workers stating that even the developers did not believe the dual MDS set-up was production ready at that time.
MooseFS seems to do its job well, the only problem is that by default it lacks HA. The master server is very important since it keeps track of where all data is located, so if it is gone your data is gone. You can set up one or more metalogger nodes, which sole purpose is to backup the metadata information. You can also manually promote metalogger to master.
They have commercial offering that includes HA, but I never used so hard to comment about it.
You could also set up HA yourself using corosync and peacemaker, but it's a bit challenging task.
Ceph and Gluster can't achieve the kind of performance required for HPC scratch. Hadoop can, but if you are managing a general purpose HPC cluster, nearly all software out there is expecting to write to a posix filesystem. Much of this software is built out of old C and FORTRAN that is not going to be rewritten. Researchers are not software engineers. You need a "lowest common denominator" of storage that you can read/write at high speed in parallel. There are few technologies for this particular use case.
The others that I know of (but haven't tried) are:
- OrangeFS (http://www.orangefs.org/) - BeeGFS (http://www.beegfs.com/) - OpenAFS/Arla (ancient, and weird, it is one of the big reasons Scientific Linux exists as a distro: http://www.openafs.org/, http://www.stacken.kth.se/project/arla/) - Scality (http://www.scality.com/) - GPFS (Spectrum now: https://www-03.ibm.com/systems/storage/spectrum/scale/) - Qumulo (they say they can hit the numbers, but I have doubts: http://qumulo.com/
There are probably many more, but these are the ones I know about.
I run two Gluster clusters for more boring enterprisey reliability.
It is still kind of a mess - there are rough edges all over. Log messages make zero sense by themselves - you either need to learn what they're indicative of or read the code. Frequently, typoing configuration commands (configuration is mostly imperative command line driven, not via configuration files) so that it fails, actually does make changes, such that you'll have to unwind what it did before you can try again, etc.
All that said, Gluster is pretty solid for our sue - distributed, fault-tolerant POSIX storage. But I wouldn't use it in an HPC environment.
Unfortunately, it's prior licensing scheme prevented larger initial adoption. A good chunk of large scale setups I've seen have adopted Ceph, but I cannot comment on performance.
Any knucklehead can install it and it works transparently with excellent performance. It's currently not enterprise-grade fault tolerant - that's the worst thing I can say about it.
Its getting there quickly. BeeGFS v6 has metadata mirroring via buddy groups, and data mirroring the same way. A few things are still not there, but it is rapidly getting closer.
Being open source helps as well.
Definitely recommend it, and have deployed it and supported it many places in my previous life: http://scalableinformatics.com .