We've been told for years that file system primitives are bad, unscalable and nasty. part of this is because NFS can be a dick, but a lot is because large scale file systems generally are fast, scalable, coherent, secure and reliable (pick two).
A lot of things can be managed with S3, but it has a lot of flaws.
Unless you've worked at a place where you've had access to something like GPFS(or whatever IBM calls it now), where you can put hooks into everything, and programme how you want to mirror/cache/copy namespaces, it all seems impossible.
Take FB for instance. They've gone balls deep with an S3 clone, which is both slow and difficult to use. Every operation takes about 3 seconds. (pulling a 10k file? pulling a 100meg file? list operation? mkdir? all 3 seconds.) Its an interface on top of the underlying blob store, so its looks like it stores it's metadata in a SQL Db (ie like lustre with metadata node and object storage nodes)
That sucks for machine learning or any kind of "normal" file operations this means there is lots of heavy engineering to get something like performance that would be solved by an ephemeral clustered file system.
The other big thing to note is that clustered filesystems are a massive massive pain in the arse to manage, especially if you choose the wrong one. (looks knowingly in the direction of ceph and gluster) you need to know what tradeoffs you are choosing before you install. Lustre is/was effectively a raid-0 over the network, Ceph is just plain inefficient, gluster is just silly, GPFS is held back by IBM being IBM.
for most people clustered filesystems are wrong choice.