spetz

Sort of, but anyone familiar with Kafka should feel like at home, although there a few subtle differences when it comes to the transport protocol, hierarchy etc. And VSR clustering is hopefully coming this week!

> I am a passenger and I ride and I ride

There's an open PR to implement sink + source for MQTT https://github.com/apache/iggy/issues/3385

Also, since we've been asked about this a lot, there's a high chance, that we might implement MQTT Gateway, just like we currently do for Kafka (proxy on top, as a separate runtime).

Awesome! You can find mine too in one of the older blog posts, hence the name :)

It's more like Kafka or Pulsar in terms of being the message streaming infrastructure (so an append-only log, not the message broker like, e.g., raw RabbitMQ). I think that the main docs page should give you a good understanding of how the data is stored/organized https://iggy.apache.org/docs/ - for example, on top of the topics, we also have "stream" which is just an extra hierarchy that can be used for something like multi-tenancy isolation or anything else depending on the use case. And yes, there's a built-in RBAC (read/manage particular streams, topics, servers, etc.). As for protocols, we have native support for TCP, QUIC, WebSocket (these 3 are stateful binary protocols), and HTTP as well.

There are a bunch of things that sum up to the overall performance gains - using Rust, thread-per-core architecture + shared-nothing (think of Seastar), custom zero-copy serialization, io_uring for disk * network I/O, and VSR-based consensus (inspired by TigerBeetle) - we simply build all this stuff from the ground up to make the most out of modern hardware and Kernel features, with a custom protocol on top (but the Kafka proxy/gateway is also WiP). And there are also connectors (sink, source), benchmarking runtime, CLI and so on.

Apache Iggy™ has officially graduated from the Apache Incubator and is now an Apache Software Foundation Top-Level Project (TLP)

It’s been quite a journey, from a small Rust message-streaming experiment in 2023 to an independent Apache project with a growing community. And its biggest feature yet, being the Viewstamped Replication Revisited (VSR) clustering, is coming soon.

Thank you for the mention! BTW, we're currently working on VSR (Viewstamped Replication) to provide the proper clustering :)

Thank you, and yes, 100% agreed about Krishna's work! :)

Thanks! Actually, the mentioned PR was just an internal merge; the one to the master branch is this one https://github.com/apache/iggy/pull/2299 which happened last week :)

Sans-IO is for clients only, and we'd like to introduce it one day to SDK to support the different runtimes as well as the blocking, sync client.

There's a core difference between the message queue and the stream, and one of them is that the append-only log acts as a simple database from which multiple independent consumers can read records at the same time; hence, you don't truncate the append-only log unless specified in its settings via a particular cleanup policy (e.g. based on message expiry or its allowed size).

Iggy is a message streaming infrastructure (think of Kafka for example), not a messaging platform like Discord or so :)

That's a good name! :)

Right, I meant TCP without TLS, and when it comes to QUIC I only did benchmark using the "dangerous" mode (disabled local cert validation). I don't have the exact numbers, but AFAIR on MacOS raw TCP was a few times faster than QUIC, while on Linux (e.g. PopOS distro) TCP was maybe around 20-30% faster?

Still, there are plenty of options to be configured in Quinn (buffer size, send/receive window etc.), so it might be that with some tweaking the performance would be similar or QUIC would be even faster than TCP - this is also one of the things that I'd like to spend more time with in the future. We expose most of these options in the server configuration file, so it's easy to adjust.

Thank you for contributing to such a great library, really easy to work with :)

Nice one, looks quite similar indeed! May I ask why did you discontinue the further work?

I did some dotnet OSS in the past, but there's something special about Rust community and I guess that if I'd picked another language instead, I wouldn't gather such an amazing team :)

As already mentioned, we've decived to use monoio, just to have something to begin with, it will take at least a few months to rewrite the core parts to make use of io_uring and thread-per-core approach (if feasible), so staying on the bleeding edge is more like a sandbox for us - we might decide to go with another runtime, like glommio, mfio or something else.

Of course, they hide the details behind io_uring, it's just that monoio seemed even easier to work with. It's not set in stone, though, actually there are new runtimes being developed as we speak, for example mfio - we'll see what will be our final choice, but we've decided to start with monoio.

Speaking of the other platforms, having the fallback to epoll or kqueue (they've even announced some Windows integration) is nice to have, however, at some point we might just purely focus on Linux development anyway (io_uring only), if there'd be any issues e.g. when it comes to the code design to provide the compatibility across multiple OS.

Thanks, I do agree, they're both lovely! :D

Can't say for sure, we share some of the same concepts that can be found in Kafka or RabbitMQ Streams, but at the same time, try to build something from the ground up and focus on low-level I/O improvements :)

Thank you :)

I had 0 Rust experience and close to 0 of system programming experience (except playing with some lower level communication protocols etc. but nothing fancy), but I'd say that something like message streaming isn't the system programming - it's close to building the database, but nowhere close to making direct usage of low level system APIs.

Thank you, feel free to ping us anytime and/or join our open Discord!

JetStream, Kafka, Redpanda, RabbitMQ Streams, Fluvio - quite a few message streaming solutions out there. Thanks!

I've started with QUIC, as I wanted to try out something new. Still, TCP protocol we have in place is a bit faster than QUIC, but it might be due to the lack of some additional tuning or so. On a side note, QUIC on MacOS is slow, when compared to Linux.

Thank you, I'd like to mention that the team is really awesome, as they've decided to join me in my efforts, just to have some fun as well :)

This is a message streaming infrastructure, similar to Kafka, Redpadna etc. You can think of it as a kind of a simple database (so-called Write Ahead Log), being responsible for appending the messages in a highly performant manner.

Monoio seems to be the most performant runtime, and actually easy to use - we have decided to go with "bleeding edge" approach, as it will still take at least a few months to implement io_uring and other optimizations, as we'll have to rewrite some of the core parts and most likely shift towards thread-per-core architecture.

Exactly, we have multiple SDKs, and the blog uses Rust Zola engine :)

It's a message stream, so more like Kafka, Redpanda or RabbitMQ Streams (plugin). Fluvio is more mature (it's an actual product & company behind it), but we have our own ideas how to make Iggy a competitive message streaming solution :)

Thank you, that's exactly how it all started! :) At some point, we will try to incorporate some benchmarks and comparisons with the other tools.