Hacker News Reader

Comment by meritt | Hacker News Reader

While S3 storage costs are fairly reasonable, request costs are unrelated to file size, which makes it an absolutely terrible choice for filesystem-like activities. I say this from experience: S3 PUT requests make up a very significant portion of our AWS bill as we write a massive number of very small files to S3.

Does anyone know of a middle-layer solution that automatically concatenates files into larger chunks (similar to HDFS) prior to persisting to S3 and for retrieval utilizes HTTP Range requests to fetch the individual file? I'm going to build it if it doesn't exist. 100MB chunks would reduce our S3 PUT costs by ~1000x.

View on HN

Couldn’t find anything from a google search, but seems you could buffer files for writes, tar them, PUT the tar, and then use the tar index to perform range reads against the tar file.

Edit: “dar”, “zip”, or “dump/restore” might be able to do this: https://serverfault.com/questions/59795/is-there-a-smarter-t... (will create an archive with an index file for random access, vs tar streaming requiring reading the entire file)

If you come up with another solution, I’d love to hear it!

Do yourself a favor and use a Hetzner Storage Box: https://www.hetzner.com/storage-box

Even if you don't include any extra costs, just keeping 10 TB on Hetzner will cost you 40€ max, whereas with S3 it will cost you at least $230.

A single Hetzner box is not really comparable to S3 in terms of durability and availability.

My experience is that they go down every 2-3 years, usually the problem is with the hard rive, occasionally with some other component. Under normal circumstances it takes them up to one hour to replace the hard drive, and then the array gets rebuilt for a few hours with the system being online. That means something like 2 hours downtime every 2-3 years, that's quite acceptable for my usage scenarios. In terms of money spent during that time, the difference is enormous.

Like most businesses on a cloud provider, we're fairly locked-in to AWS without major changes. Going offsite would replace S3 PUT costs with significant bandwidth costs.

My friend also got a real big bill for S3 PUT requests, they are just too expensive.

When we built JuiceFS [1], a POSIX filesystem on top of S3, there is feature in the roadmap to combine multiple small files together before uploading into S3 (each file is a slice of chunks in S3). We will let you know when it's ready.

[1] https://juicefs.io/

heh, when s3 first came out they didnt have per request charges. then someone built a fuse fs on top of s3, and adding those charges was largely to convince people not to use s3 as a filesystem

I POCd dynamodb(+api gateway+lambda for reads) for a project that was tons of tiny files dominated by puts. was a pretty trivial switch(writes was one extra line of code, reads was like 10 total lines), estimated cost savings was large(though we didnt fully calculate it). it didnt exist at the time, but i think lambda@edge might work really well for reads.

cperciva’s tarsnap server implements a log structured filesystem on S3. [1] This enables bundling of multiple writes into large objects, with the explicit goal of reducing per-request costs. It’s not open source, but maybe some of the high level ideas can be cribbed?

[1] http://www.daemonology.net/blog/2008-12-14-how-tarsnap-uses-...

Entirely untested and just spitballing... use Firehose to concat a bunch of small files and store based on size or time passage as a larger file to s3.

Then use Lambda transform in firehose to index the byte ranges for each record (file)? (Biggest q in the process would be here... you might have to index after it hits s3)

S3 also allows the Range request header out of the box as well.

Interesting problem, and I can totally related with s3 put costs be astronomical.

Dunno if it would work in your case but ... I've had fun mounting a ZFS file system using an entire EFS volume as the "physical" back end. You have to give up on writes actually being on disk when you think they are, but the payback is that's it's really kinda fast.

Depends on how you’re writing. You could send everything to a kinesis or firehouse stream and use that to batch inputs, or you could write to an Elastic File System and batch there with a scheduled function

HDFS does no such thing. If you store a million small files it'll store a million small files on disk as well.

I think the parent might mean HDF5.

Reply on news.ycombinator.com