We're using LVM cache with Ceph on spinning disks for object storage workloads in a large, on-prem private cloud. In other words, this cache is behind the S3 backend, but in front of spinning disks.

When we built this, we had a choice of using all-SSD, spinning disk only, or a combination of both using LVM cache.

LVM cache gave us a lot better performance per $ than using all-SSD.

As for what type of actual application workloads? What do you use AWS S3 for? You name it, we have it:

Backups, container registry, data warehouse, pictures, PDF docs, config data...