Comment by toomuchtodo | Hacker News Reader

Comment by toomuchtodo | Hacker News Reader

Parent Full thread

Any object storage system (S3, Backblaze B2, Azure Blob, GCP) should be able to handle those file sizes with proper chunking into smaller parts (limit details below per object store).

S3: https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.... (Max object size: 5TB, Max single multipart size: 5GB)

Backblaze B2: https://www.backblaze.com/b2/docs/large_files.html (Max file size: 10TB, Max single object size: 5GB)

Azure: https://docs.microsoft.com/en-us/rest/api/storageservices/un... (Max file size: 4.75 TB, Max single block size: 100MB)

Google: https://cloud.google.com/storage/quotas (Max file size: 5TB, doesn't appear there is a lower limit for objects to be composed into a single object, docs could be better in this regard)

@khc: Terminology updated to be more clear for S3

I'd drop Microsoft from that list because the request for for something "reliable."

It's reliable enough, if you can get it to Microsoft's cloud. But for the last six months I've struggled putting very large files into Azure, using five different connections from five different providers in three locations. Small files are no problem. But large ones take two, three, or four tries.

I don't disagree Azure is hot garbage, just listing for completeness. S3 and Backblaze B2 are my go-to object stores.

Hi, GCS engineer here. The lower limit on composing objects is one source object, in which case you are not so much composing as you are copying with style. Zero source objects is an error. I will file a note about the docs, thanks.

Thank you so much! Is there a limit on size of objects besides the 5TB max described in the docs, similar to other object stores where multiparts have lower limits than the total composed object?

There is also a lower bound of 0 bytes :)

You can compose a 4TB object with a 1 byte object, or you can compose 32 150GB objects, just so long as the destination object doesn't go over 5 terabytes.

Brilliant! Thank you again!

S3 max single object size is 5TB. The 5GB limit is for single part upload API. You can use the multi-part API to upload single object in multiple parts.

Reply on news.ycombinator.com