Enhancing storage efficiency and reduce storage costs with S3-batch-object-store

TL;DR: s3-batch-object-store is a Go module that allows for batch uploading of objects to a single S3 file and retrieving each object separately using the AWS S3 API, fetching only the bytes for that specific object.

The method basically consists of appending multiple objects to a single file, keep the information of where each object is placed in the file, and then upload one single file to s3 with many objects in it, reducing drastically the number of PUT operations needed to store a large number of objets.

The developer company educed storage costs by 70% with our new approach. You can head to their s3-batch-object-store GitHub repository (https://github.com/embrace-io/s3-batch-object-store) to check out the module and try it out.