The most ridiculous part, they didn't have a way to run distributed commands over ssh. When a filesystem mounted in read-only mode, somebody had to ssh into each of the 175 servers manually and run commands to fix it.