Hacker News Reader

> As Hugging Face hosts nearly 11PB of datasets with Parquet files alone accounting for over 2.2PB of that storage

11PB on S3 would cost ~$250k per month / $3m per year.

HuggingFace has raised almost $400M.

Not saying it's nothing, but probably not a big deal to them (e.g. ~10 of their 400+ staff cost more).

HuggingFace really is such an amazing resource to the ML community. Not just for storing datasets, but being able to stand up a demo of my models using spaces for anyone to use? It's hard to overstate how useful that is.

We are here to help lower that :-) . As we can push dedupe to the edge we can save on bandwidth as well. And hopefully make everyone upload and download faster.