We've had slowness on https://Clara.io because of what we believe was excessive file system fragmentation as we were running on zfs and then later brtfs. We were on zfs and brtfs because it allowed for easy and quick backups via snapshots. But eventually writes were slowed leading to long lock times and making everything basically unworkable. A reinstall would always fix the issue for a while. We believe that the same reasons that backups were fast, the copy on write feature, is the same reason we got excessive fragmentation after a while.

I believe it was because we were not using xfs as our file system, which we have recently moved over to. So far the issue hasn't re-occured.

Other than this issue, MongoDB has been pretty good. Indices are important and formulating queries so that they are index based is also important (the same result can be obtained in multiple ways, some of which likely do not use the indices, while other formulations will use them.)

We use Redis as a cache in front to reduce the unnecessary repetitive load on MongoDB for common operations.

We have MongoDB cluster setup and it has saved us a number of times. Also it is really just a live backup system -- we've had a few hardware failures and we still have never had to restore for an archive backup, we just resync from one of the failover servers. But using that is likely a no-brainer, and you are likely already doing it.