Comment by sanguy | Hacker News Reader

Comment by sanguy | Hacker News Reader

Lustre is a pure-bread racehorse. It is not something the general masses should ever take a ride on; or even get up close and personal with. It will bite you; kick you; piss on you; crap on you; stomp on you; and generally try to kill you. It is cranky. It is not user friendly. It doesn't play nice with others.

But if you do know how to ride it; you have track to ride it on; and it decides to let you ride; you are in for a real treat. It is a rush that can't be beat. Even though after that ride your body will ache from the beating it's been given.

In the right workloads Lustre can't be beat. I've seen it completely transform large HPC clusters from dogs to kentucky derby winners. But as mentioned by many it is a complex beast that needs the right people; right hardware; and right workloads to make it run.

We used it for many years for large geospatial big data workloads. Huge images to many 1000's of HPC nodes. Worked great. But it was highly sensitive to the technical staff running it; and ultimately it became too "cranky" to use in production mode without Gandalf and his wizard army to keep it from eating itself.

So we now run on NFS over IB and happily give up some performance for 24*365 uptime.

> pure-bread

OT: I think you meant pure-bred. :) Reminds me of this section from Orwell's «Politics and the English Language»:

> Some metaphors now current have been twisted out of their original meaning without those who use them even being aware of the fact. For example, toe the line is sometimes written as tow the line. Another example is the hammer and the anvil, now always used with the implication that the anvil gets the worst of it. In real life it is always the anvil that breaks the hammer, never the other way about: a writer who stopped to think what he was saying would avoid perverting the original phrase.

It's not uncommon to see anvils with their horns broken off from too much hammering.

Is it possible multiple hammers were involved?

Such a wonderfully insightful and well written comment and you choose to belabor a typo? sigh

>>> But it was highly sensitive to the technical staff running it;

Care to be more specific?

I'd be very interested in how much "some performance" is :) I'm assuming good old nfsv3? No encrypted transport etc?

In my experience, Lustre on a typical medium-size (several hundred node) cluster has been more reliable than NFS (at least once the cluster vendor was out of the way), and you can actually use it for MPI-IO, which is crucial. It's been quite straightforward to operate, at least with a simple LNET setup. I don't understand this stuff about racehorses. NFS has been easily flattened in various circumstances, too.

That said, I'd have chosen PVFS^WOrangeFS at my previous site if I'd had the choice, but I've not been able to make or find a useful performance comparison.

Actually Lustre gives good "out of the box" performance... I've used a few HPC clusters and have almost never felt the need to issue commands to configure striping, etc.

if your alternative to Lustre is NFS over IB, you didnt need Lustre in the first place

Reply on news.ycombinator.com