A bit confused why you are trying to roll your own storage solution. Ceph is great for a lot of applications, but I am not sure it really fits the bill for what you describe. Especially when you indicate you are going to use spinning disk behind it. Have you looked at any of the storage arrays on the market? Your TCO is likely to be much lower and your performance/resiliency much higher if you buy something with $100's of millions of R&D behind it rather than all the hours and costs of rolling your own. Not saying its impossible to make it work, but it just sounds like something that will be a PITA going forward.