Thanks for the suggestions.
We're already planning a separate router for the management network ("Apart from those routers we'll have a separate router for a 1Gbps management network.").
All Ceph journals will be on SSD too. I've added a question about combining this with bcache in https://gitlab.com/gitlab-com/www-gitlab-com/commit/a9cc9aad...