Google and Red Hat announce cloud-based scalable file servers

The basic answer to your question is that networks are slow compared to local storage. In order to get decent performance, you must either avoid network round trips or amortize their cost over many operations. We - not just GlusterFS but distributed filesystems in general - can do this pretty well for some operations. We can batch, buffer, cache, etc. This works great for plain old reads and writes to large files (for example). It doesn't work so well for operations that have to touch many small files. For those, in order to ensure the required level of consistency/currency, we have two choices.

(1) Send a request per file to get current metadata.

(2) Cache metadata, and participate in some sort of consistency protocol to make sure we don't serve stale cached information.

Both approaches have workloads where they perform better and workloads where they perform worse. In addition, the second approach adds a lot of complexity, especially in a system where failures are common and a loosely coordinated set of servers must respond (systems with a single master server have an easier time here but are less resilient). The inherent difficulty of this approach is why e.g. CephFS has taken so long to mature.

GlusterFS took the first route instead. It does mean that "ZOT" (Zillions Of Tiny files) workloads will perform poorly. I won't deny that. On the other hand, it's easier to test or prove correct, and the time not spent on solving the hard version of that problem - often for little eventual benefit - can instead be spent on other kinds of improvements. Some people are happy with that. Some are not. Some spread FUD. Some try to implement the practical equivalent of a distributed filesystem on top of some alternative (e.g. object stores) with their own even more serious limitations, and experience even more pain as a result. Some are initially unhappy with these tradeoffs, but work with us and learn to work more effectively with these limitations to enjoy other benefits. That's life in the big city.

Unfortunately, I don't remember most of the details. It's fallen out of my headspace in the months since I've left. ;)

One has to wonder why the software even lets you run it with an unsafe number of nodes though... (Lots of distributed software is guilty of this...)

This is not a given, the cluster can and should refuse to operate if it can not get a majority vote for the master, if that is required to prevent data corruption. With two nodes that means both nodes must be active and reachable, with three nodes one node can fail.

Do you mind sharing which GlusterFS version you're on, which kernel, and roughly what your load profile is? (eg. lots of small files, read-heavy, write-heavy, etc.) Also, are you using Red Hat Gluster Storage or the open source version?

I don't. I am using stock vsftp with a PAM module that allows authentication against our web application.

The issue with Lustre is that it usually requires kernel patches if you are not running the aforementioned distributions that are designed for HPC. Specifically the Lustre OSD used a modified version of ext2 among other issues.

That could be getting better now with the new ZFS based OSD though I wouldn't put money on it being "easy" to install.

Lustre also doesn't provide any means of replication. If you want to achieve HA with Lustre you need to make each OSD individually HA. This can be done with multi-pathed SAS arrays and a ton of scripting but it's still not exactly a walk in the park.

Hopefully one day we will see a real high performance distributed filesystem that also bundles replication, tiering and some semblance of POSIX compatiblity. I doubt Ceph is going to be it so we are probably still 5-10 years from a solution to the problem.