- Dropbox did not want to be running a single version of the software / a single region, because they consider the risk of a single software bug / human error resulting in data loss too high. However, the alternative they choose introduces a completely new code base which will have to be battle tested. This increases the risk of a data loss bug, which would affect a smaller fraction of the data, but any significant data loss issue would be game-over for a company like Dropbox. Did they consider partitioning the system into smaller subsets (some single region, other multi-region), using staged roll-outs of new software versions? Or is there really some fundamental incompatibility between Magic Pocket and multi-region?
- The "New Replication Model" story sounds a bit too simplified. It seems to re-introduce some issues that the single region Magic Pocket solution had already solved: the size of the IO operations becomes quite small again (fractions of the 4M block size), placement of data on the disks becomes less predictable, which could cause increasing rebuilt times when a disk fails. Also, the number of IOs to read or write an object increases significantly (2-3x in the example), which means that the observed advantages in latency go hand in hand with a 2-3x lower maximum supported load than in the Magic Pocket case, before the latency explodes due to running out of IOPS on the HDD's. The whole design seems to ask for far more IOPS than the Magic Pocket solution, which sounds like an odd match to SMR HDD's.
These issues are maybe alleviated by the fact that moving data to the cold tier happens asynchronously, and the cold data is accessed very infrequently, resulting in far less IOPS being required for the cold storage region. However, it also makes the option of combining hot and cold data on a single disk much more difficult (which for HDDs is the way to make optimal use of the limited IOPS vs. their huge capacity - I suspect Amazon / Google use this for their near-line storage solution). Moving from the 2+1 example to e.g. 4+1, to reduce cross region storage costs even more, becomes now a though call as it now goes hand in hand with an even larger increase in IOPS cost.
- The claimed "simplicity" of deleting data in the proposed scenario is rather relative. If they are using SMR drives, deleting data and reclaiming space are complex and expensive operations. They might reduce it to a non-distributed problem (which is still a significant gain, of course), but it is far from trivial.
Probably a lot of the finer, left-out details of their cross-region system address these issues, and if not, maybe the cross region system and the single region Magic Pocket solution will converge again in a later phase.