Thank you very much, it's appreciated.
Relevant part from our initial look into this https://gitlab.com/gitlab-com/infrastructure/issues/59#note_...
"My gut feeling with all this is that the NFS server is the main bottleneck. It is maxing out all the time in CPU time dedicated to IO (check graph) and our last finding is also pointing in the direction that the way the LVM is setup (using linear) is not efficient at all, and it is generating a lot of overhead by fragmenting the actual filesystem for reads (on rewrites) and by concentrating all the writes into one single drive at time. The first test I want to make is to create the new NFS server, and add monitoring on that one to rule out completely the fact that the NFS server is the main bottleneck, hopefully this should happen today."