> Bonus question) Why are we thinking about all of this. You should have just used s3?

NFS and Ceph both support the standard POSIX file access model, notably including random-access reads and writes on a file without replacing the entire file and consistency models that let you reason about directories and their contents usefully. If your goal is to run the reference implementation of git, or something that is compatible with it, on the filesystem, you absolutely need something that supports this.

You could write something new that speaks the git protocol but is designed to be backed by an eventually-consistent object store instead of a POSIX-compliant file store ... but that seems like an equally big challenge, honestly.