"HDFS and S3 are designed around large blocks (optimized to overcome slow random I/O on disks), while new NVMe hardware supports fast random disk I/O (and potentially small blocks sizes). "
Interesting solution presented here - keep the block size constant, but put the small files on NVMe disks.
Disclaimer, one of the authors here.
You can see a video with discussion of the NVMe support in HopsFS in this Spark Summit talk: