Quantcast File System on S3

quantcast.com

20 pointslumens4 comments

For some context:

Quantcast moved off Hadoop in favor of a home-rolled solution (QFS) because of Hadoop wasn't scaling well to our needs (40PB+ data processed/day).

This release of QFS (https://github.com/quantcast/qfs) includes support for S3, among other improvements.

I wonder how it compares/relates to https://minio.io

Can it run on minio, for example?

One difference I see is that QFS is block-based and each file is composed of multiple S3 objects. Hence, there is no file size limit.

Hi, we recently announced Quantcast File System version 1.2. Besides many bug fixes and improvements, it comes with the ability to use Amazon S3 as a backend store. You can check it out on QFS Github. I personally contributed to this release, and I'd be happy to answer any questions. Thanks.

Reply on news.ycombinator.com