Multi-region is pretty essential IMO. You can’t get the same cost effectiveness by building your own multi-region S3, stapling multiple buckets together. The basic premise of S3 is that you get a keystore interface, and S3 handles error correction and distributing the chunks among multiple physical machines on the cheap. Multi-region is the same product, at a larger scale. The complicated part is that somebody has to pay for the network bandwidth (whereas in a single region, it’s so cheap it’s unmetered).

The CAS thing is also pretty essential. Everybody wants to build some simple storage system on top of S3 and without CAS it’s pretty damn hard to have any kind of consistency guarantees, even for very simple systems… you end up having to build something outside S3 to manage consistency. An easy to understand use case is backups. Suppose you are using S3 as a backend for a backup system which deduplicates backups. You want to make backups from multiple locations, you want to deduplicate because there’s a lot of duplicate data floating around (maybe you have terabytes of video files getting copied, ML model weights, something else big that you copy around), and you want to expire old backups. You can almost build this on top of plain S3, and the only reason you can’t is because it’s unsafe to expire old data in such a system if any backup is writing (because the other backup may add a new reference to data, racing against the expiration / garbage collection process). A simple CAS gives you a lot of tools to solve this. The alternative to CAS is doing something kinda silly, like running a DynamoDB table as a layer of indirection.

Neither of these things add much complexity to S3.

(I think append is less useful and potentially a lot more complicated, both in terms of its API implications and in terms of the underlying complexity. If I want “append”, then I can use multipart uploads, or just upload multiple objects and reassemble them on the client side.)