Secor: A service persisting Kafka logs to S3, GCS and Azure Blob Storage

I love this. This is a property already baked into another messaging queuing system called Pulsar and I've thought about writing my own for internal use (maybe to be open-sourced). The only quirk that I see is this link in the project description readme:

> it is guaranteed that each message will be saved in exactly one S3 file

This can get expensive. Specifically a 1-to-1 write per message with S3 (or any cloud storage provider) gets expensive very, very quickly. Further, when reading from cloud storage, you've got to read one message (object) at a time which adds up as well. I'm not talking about egress bandwidth either, I'm talking about S3 PUT/GET operations. I'd love to see some kind of batching operation that takes groups of messages (perhaps configurable time/size limits?) and writes them as a blog to cloud storage.

I've wondered for a while now why there isn't something like a cross between Kafka (durable queue broker) and Datomic (a "database" partitioned into storage chunks, where each chunk canonically lives in object storage and is only cached by the instances doing the queries.)

I'd love to see a queuing infrastructure that consists of two parallel clusters, running on the same machines:

• a set of write nodes, which each "own" partitions on a topic (in the sense of being that partition's sole writer); where a write-node receives writes to its partition until it fills up a chunk, at which point it shoves the chunk into object storage;

• a set of read nodes, which serve reads and/or streaming subscription requests by 1. streaming archival data by retrieving chunks from object storage into a read-node-local MRU-on-disk-pressure cache; and then 2. once the read stream "catches up" to the end of the archival data, it would switch to proxying to the relevant write-node, which would serve the query/stream from its uncommitted chunk.

-----

I feel like this queue architecture would be useful for all sorts of things, at least for my own use-cases. Especially because it'd scale perfectly-well down to running one read-node and one write-node on one machine, while still being able to make durability guarantees about the committed data. (The durability of uncommitted data, I don't much need to worry much about, personally.)

Is there anything like this on the market?

I've been following this project for a long time, but I have to ask what are the advantages over using Kafka Connect Sinks?

https://docs.confluent.io/current/connect/kafka-connect-s3/i...

At prior company we used Secor to backup Kafka topics to S3 and would use to replay/reingest data with success. If I were to reimplement Kafka somewhere new it would definitely be one of my "Go to Production" requirements.

We've used this at Patreon to get data from our various sources in the application to Redshift. We've since replaced this with Kinesis Firehose

I also used FluentD for this. Basically we alredy had fluentd, and fluentd can output to anything, not just S3.