Hacker News Reader

> The data has to be on the filesystem. You can’t load data from a blob store like AWS S3.

You sure about this? You can mount S3 locally, and the filesystem should be transparent to the kernel.

There are two ways of doing this that I know of:

1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compression to get better performance.

2. Syncing to filesystem with AWS DataSync. This would work, yes, but it's a S3 specific feature and not all blob storage systems have it.

HDF5 has some S3-native features emerging (https://www.hdfgroup.org/wp-content/uploads/2019/11/SC19-HDF...), and there's also this relevant comparison reading HDF5 files as efficiently as native Zarr: https://medium.com/pangeo/cloud-performant-reading-of-netcdf...

> 1. FUSE filesystems. This means sending really slow queries to S3, and so you're much better off using compression to get better performance.

As author of https://github.com/kahing/goofys/ I respectfully disagree :-)

I don't mean the filesystem implementation is inherently slow, necessarily: I'm talking about bandwidth. If you get 3× compression, read times from S3 will be 3× as fast—but that means no mmap().

Reply on news.ycombinator.com