Why was Apache Kafka created?
bigdata.2minutestreaming.com
bigdata.2minutestreaming.com
[1]: https://www.linkedin.com/blog/engineering/infrastructure/int...
Confluent is 11 years old and IPOed several years ago. It was founded by 3 ex-LinkedIn people who originally designed Kafka. 2 of the 3 founders are still at the company. LinkedIn never used Confluent, Confluent was a company founded to sell an enterprise version of the open source project (and later a cloud version).
These days though, unless I was at Fortune 100 scale, I'd absolutely turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run.
Also I like Kafka because I met two pretty Russian girls in San Francisco a decade back and the group we were in played a game where we described what the company we worked for did in the abstract, and then tried to guess the startup. They said "we write distributed streaming software", I guessed "confluent" immediately. At the time confluent was quite new and small. Fun night. Fun era.
The main two things that makes Postgres less suitable for Kafka-type logs is that tables aren't very efficient for sequentially ordered data, and that deletion incurs bloat until vacuumed. You could solve both by providing a new table engine (table access method), although I'm not sure you can control heap storage placement to the degree desired for a physically ordered table. But you could also do a lot of tricks to make it delete faster (append only means no updates; just prune from the head without MVCC when provably safe against concurrent reads?) and make filtering faster.
Kafka is of course more than that, but I bet you can get quite far with this.
I don't have any experience with Redis Cluster Streams. Could you please tell us how it is simpler to manage? IMO, installing and managing a Kafka cluster in a non Fortune 100 scale is simple enough: run 1 java command for zookeeper, run another java command for a broker (with recent version of Kafka, zookeeper is not needed anymore). The configuration files are not very simple but not very complicated either. When we have another machine, we can run another broker on it.
Redis Cluster Streams is cheaper to run because it's written in C, doesn't need a VN to run? Or because its messages are stored in RAM not SSD?
> Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale.
Could you expand on this - when does it drop C? Are you referring to cases where you write to Kafka without waiting for all replicas to acknowledge the write? (acks=1)
And why was it a big step - what other systems didn't embrace dropping the C?
Native support for dropping correctness in SQL is tricky, see Transaction Isolation Levels, but I mostly mean in overall web architecture, rather than specifically in one database or the other.
Now I’m wondering if we’re all overthinking this when we could just use rendezvous hashing and a bunch of database servers with a heap table called “eventlog” and be done with it…
I sue Kafka precisely for that. Redis Cluster Streams have AOF persistence logs as I see from the doc. How stable it is?
At least this was the reason we decided to use Kafka instead of simple queues.
It was useful when we built new consumer types for the same data we already processed or we knew we gonna have later but cant build now due to prorities.
Core NATS is an ephemeral message broker. Clients tell the server what subjects they want messages about, producers publish. NATS handles the routing. If nobody is listening, messages go nowhere. It's very nice for situations where lots of clients come and go. It's not reliable; it sheds messages when consumers get slow. No durability, so when a consumer disconnects, it will miss messages sent in its absence. But this means it's very lightweight. Subjects are just wildcard paths, so you can have billions of them, which means RPC is trivial: Send out a message and tell the receiver to post a reply to a randomly generated subject, then listen to that subject for the answer.
NATS organizes brokers into clusters, and clusters can form hub/spoke topologies where messages are routed between clusters by interest, so it's very scalable; if your cluster doesn't scale to the number of consumers, you can add another cluster that consumes the first cluster, and now you have two hubs/spokes. In short, NATS is a great "message router". You can build all sorts of semantics on top of it: RPC, cache invalidation channels, "actor" style processes, traditional pub/sub, logging, the sky is the limit.
Jetstream is a different technology that is built on NATS. With Jetstream, you can create streams, which are ordered sequences of messages. A stream is durable and can have settings like maximum retention by age and size. Streams are replicated, with each stream being a Raft group. Consumers follow from a position. In many ways it's like Kafka and Redpanda, but "on steroids", superficially similar but just a lot richer.
For example, Kafka is very strict about the topic being a sequence of messages that must be consumed exactly sequentially. If the client wants to subscribe to a subset of events, it must either filter client-side, or you have some intermediary that filters and writes to a topic that the consumer then consumes. With NATS, you can ask the server to filter.
Unlike Kafka, you can also nack messages; the server keeps track of what consumers have seen. Nacking means you lose ordering, as the nacked messages come back later. Jetstream also supports a Kafka-like strictly ordered mode. Unlike Kafka, clients can choose the routing behaviour, including worker style routing and deterministic partitioning.
Unlike Kafka's rigid networking model (consumers are assigned partitions and they consume the topic and that's it), as with NATS, you can set up complex topologies where streams get gatewayed and replicated. For example, you can streams in multiple regions, with replication, so that consumers only need to connect to the local region's hub.
While NATS/Jetstream has a lot of flexibility, I feel like they've compromised a bit on performance and scalability. Jetstream clusters don't scale to many servers (they recommend max 3, I think) and large numbers of consumers can make the server run really hot. I would also say that they made a mistake adopting nacking into the consuming model. The big simplification Kafka makes is that topics are strictly sequential, both for producing and consuming. This keeps the server simpler and forces the client to deal with unprocessable messages. Jetstream doesn't allow durable consumers to be strictly ordered; what the SDK calls an "ordered consumer" is just an ephemeral consumer. Furthermore, ephemeral consumers don't really exist. Every consumer will create server-side state. In our testing, we found that having more than a few thousand consumers is a really bad idea. (The newest SDK now offers a "direct fetch" API where you can consume a stream by position without registering a server-side consumer, but I've not yet tried it.)
Lastly, the mechanics of the server replication and connectivity is rather mysterious, and it's hard to understand when something goes wrong. And with all the different concepts — leaf nodes, leaf clusters, replicas, mirrors, clusters, gateways, accounts, domains, and so on — it's not easy to understand the best way to design a topology. The Kafka network model, by comparison, is very simple and straightforward, even if it's a lot less flexible. With Kafka, you can still build hub/spoke topologies yourself by reading from topics and writing to other topics, and while it's something you need to set up yourself, it's less magical, and easier to control and understand.
Where I work, we have used NATS extensively with great success. We also adopted Jetstream for some applications, but we've soured on it a bit, for the above reasons, and now use Redpanda (which is Kafka-compatible) instead. I still think JS is a great fit for certain types of apps, but I would definitely evaluate the requirements carefully first. Jetstream is different enough that it's definitely not just a "better Kafka".
Jetstream is even more limited than most Kafkas on number of streams https://github.com/nats-io/nats-server/discussions/5128#disc...
What are your impressions of Red panda?
We're particularly interested in NATS' feature of working with individual messages and have been bitten by Kafka's "either process the entire batch or put it back for later processing", which doesn't work for our needs.
Interested if Redpanda is doing better than either.
NATS/Jetstream is amazing if it fits your use case and you don't need extreme scalability. As I said before, it offers a lot more flexibility. You can process a stream sequentially but also nack messages, so you get the best of both worlds. It has deduping (new messages for the same subject will mark older ones as deleted) and lots of other convenience goodies.
We've looked at Redis streams but me and a few others are skeptical as Redis is not known for good durability practices (talking about the past; I've no idea if they pivoted well in the last years) and sadly none of us has any experience with MQTT -- though we heard tons of praise on that one.
But our story is: some tens of terabytes of data, no more than a few tens of millions of events / messages a day, aggressive folding of data in multiple relational DBs, and a very dynamic and DB-heavy UI (I will soon finish my Elixir<=>Rust SQLite3 wrapper so we're likely going to start sharding the DB-intensive customer data to separate SQLite3 databases and I'm looking forward to spearheading this effort; off-topic). For our needs NATS Jetstream sounds like the exactly perfect fit, though time will tell.
I still have the nagging feeling of missing out on still having not tried MQTT though...
If its tens of TB in a stream, then I've not personally stored that much data in a stream, but I don't see why it wouldn't handle it. Note that Jetstream has a maximum message size of 1MB (this is because Jetstream uses NATS for its client/server protocol, which has that limit), which was a real problem for one use case I had. Redpanda has essentially no upper limit.
Note that number of NATS servers isn't the same as the replication factor. You can have 3 servers and a replication factor of 2 if you want, which allows more flexibility. Both consumers and streams have their own replication factors.
The other option I have considered in the past is EMQX, which is a clustered MQTT system written in Erlang. It looks nice, but I've never used it in production, and it's one of those projects that nobody seems to be talking about, at least not in my part of the industry.
Do you have any other recommendations? The time is right for us and I'll soon start evaluating. I only have NATS Jestream and MQTT on my radar so far.
Kafka I already used and rejected for the reasons above ("entire batch or nothing / later").
As for data, I meant tens of terabytes of traffic on busy days, sorry. Most of the time it's a few hundred gigs. (Our area is prone to spikes and the business hours matter a lot.) And again, that's total traffic. I don't think we'd have more than 10-30GB stored in our queue system, ever. Our background workers aggressively work through the backlog and chew data into manageable (and much smaller chunks) 24/7.
And as one of the seniors I am extremely vigilant of payload sizes. I had to settle on JSON for now but I push back, hard, on any and all extra data; anything and everything that can be loaded from DB or even caches is delegated as such with various IDs -- this also helps us with e.g. background jobs that are no longer relevant as certain entity's state moved too far forward due to user interaction and the enriching job no longer needs to run; when you have only references in your message payload, this enables and even forces the background job to load data exactly at the time of its run and not assume a potentially outdated state.
Anyhow, I got chatty. :)
Thank you. If you have other recommendations, I am willing to sacrifice a little weekend time to give them a cursory research. Again, utmost priority is 100% durability (as much as that is even possible of course) and mega ultra speed is not of the essence. We'll never have even 100 consumers per stream; I haven't ever seen more than 30 in our OTel tool dashboard.
EDIT: I should also say that our app does not have huge internal traffic; it's a lot (Python wrappers around AI / OCR / others is one group of examples) but not huge. As such, our priorities for a message queue are just "be super reliable and be able to handle an okay beating and never lose stuff" really. It's not like in e.g. finance where you might have dozens of Kafka clusters and workers that hand off data from one Kafka queue to another with a ton of processing in the meantime. We are very far from that.
Jetstream is written in Go and the Go SDK is very mature, and has all the support one needs to create streams and consumers; never used it from Elixir, though. EMQX's Go support looks less good (though since it's MQTT you can use any MQTT client).
Regarding data reliability, I've never lost production data with Jetstream. But I've had some odd behaviour locally where everything has just disappeared suddenly. I would be seriously anxious if I had TBs of stream data I couldn't afford to lose, and no way to regenerate it easily. It's possible to set up a consumer that backs up everything to (say) cloud storage, just in case. You can use Benthos to set up such a pipeline. I think I'd be less anxious with Kafka or Redpanda because of their reputation in being very solid.
Going back to the "whole batch or nothing", I do see this as a good thing myself. It means you are always processing in exact order. If you have to reject something, the "right" approach is an explicit dead-letter topic — you can still consume that one from the same consumer. But it makes the handling very explicit. With Jetstream, you do have an ordered stream, but the broker also tracks acid/nacks, which adds complexity. You get nacks even if you never do it manually; all messages have a configurable ack deadline, and if your consumer is too slow, the message will be automatically bounced. (The ack delay also means if a client crashes, the message will sit in the broker for up to the ack delay before it gets delivered to another consumer.)
But of course, this is super convenient, too. You can write simpler clients, and the complicated stuff is handled by the broker. But having written a lot of these pipelines, my philosophy these days is that — at least for "this must not be allowed to fail" processing, I prefer something that is explicit and simpler and less magical, even if it's a bit less convenient to write code for it. Just my 2 cents!
This is getting a bit long. Please do reach out (my email is in my profile) if you want to chat more!
> Jetstream clusters don't scale to many servers (they recommend max 3, I think)
You can have clusters with many servers in them, 3 is actually the minimum required if you want fault-tolerance, that's how you scale JetStream horizontally: you let the stream (which can be replicated 1, 3 or 5 times) spread themselves over the servers in the cluster.
JetStream consumers create state on the server (or servers if they are replicated) and are either durable with a well known name or 'auto-cleanup after idle time' (ephemeral), and indeed allow you to ack/nack each message individually rather than just having an offset.
However, that is with the exception of 'ordered consumers', which are really the closest equivalent to Kafka consumers in that the state is kept in the client library rather than on the servers. They deliver the message in order to the client code, they take care of re-deliveries and recovering from things like getting disconnected from a server, and only ever deliver messages in order, no need to explicitly ack the messages, and if you want to persist your offset (which is the sequence number of the last message the ordered consumer delivered) just like the consumer group clients in Kafka persist their offset in a stream, you would persist your offset in a NATS KV bucket.
And indeed you can now even go further and use batched direct gets to get very good read speed from the stream and no extra server state in the server besides an entry in the offset KV, performance of the batched direct gets is very high and can match the ordered consumer's speed. Besides no incurring no server state, another advantage of stateless consuming is that all the servers replicating the stream will be used to process direct get requests not just the currently elected leader (don't forget to enable direct gets for the stream, it's not on by default). So you can scale the read throughput horizontally by increasing the number of replicas.
The mechanics of replication: streams and stateful consumers can be replicated using 1, 3 or 5 servers. Servers connect directly together to form a cluster and jetstream assets (streams/consumers) are spread out over the servers in the cluster. Clusters can be connected together for form super-clusters. Super-cluster means that access to JetStream assets is transparent: streams/consumers located in one cluster can be accessed from any other cluster. You can have streams that mirror or source from other streams, those mirrors could be located in other clusters to offer faster local access. You can easily move on the fly JS assets from one cluster to another. Leaf nodes are independent servers (which can be clustered) that connect to a cluster like a client would. Being independent means they have their own security for their own clients to connect to them, they can have their own JS domain and you can source to/from streams between the leaf node's domain and the hub (super-cluster). Leaf nodes can be daisy chained.
Sorry, what I meant that each stream (which forms a Raft group) doesn't scale to more. I thought it was 3, but thanks for the correction.
Everything else you wrote confirms what I wrote, no? As for batch direct gets, that's great, but I'm not sure why you didn't go all the way and offered a Kafka-type consumer API that is strictly ordered and persists the offset natively. I've indeed written an application that uses ordered consumers and persists the offset, but it is cumbersome.
Every time I've used Jetstream, what I've actually wanted was the Kafka model: Fetch a batch, process the batch, commit the batch. Having to ack individual messages and worry about AckWait timeouts is contrary to that model. It's a great programming model for core NATS, but for streams I think you guys made a design mistake there. A stream shouldn't act like pub/sub. I also suspect (but can't prove) that this leads to worse performance and higher load on the cluster, because every message has to go through the ack/nack roundtrips.
I'd also like to point out that Jetstream's maximum message size of 1MB is a showstopper. Yes, you can write big messages somewhere else and reference them. But that's more work and more complexity. With Kafka/Redpanda, huge messages just work, and are not particularly a technical liability.
Streams can have more than 3 replicas. Technically they can have any number of replicas but you only get extra HA when it's an odd number (e.g. 6 replicas doesn't offer more HA than 5, but 7 does). Typically the way people scale to more than one stream when a single stream becomes a bottleneck is by using subject transformations to insert a partition number in the subject and then creating a stream per partition.
Point taken about wanting to have the 'ordered consumer + persist the offset in a KV' built-in, though it should really not be cumbersome to write. Maybe that could be added to orbit.go (and we definitely welcome well written contributions BTW :)).
> Having to ack individual messages and worry about AckWait timeouts is contrary to that model
Acking/nacking individual messages is the price to pay for being able to have proper queuing functionality on top of streams (without forcing people to have create partitions), including automated re-delivery of messages and one-to-many message consumption flow control.
However it is not mandatory: you can set any ack policy you want on a consumer: ackAll is somewhat like committing an offset in Kafka (it acks the sequence number and all prior sequence numbers), or you can simply use ackNone meaning you forgo completely message acknowledgements (but it will still remember the last sequence number delivered (i.e. the offset) automatically).
For example using a pull consumer with ack policy=none and doing 'fetch' to get batches of messages is exactly what you describe what you want to do (and functionally not different from using an ordered consumer and persisting the offset).
And yes, having acks turned on or off on a consumer does have a significant performance impact: nothing comes for free and explicit individual message acking is a very high quality of service.
As for the max message size you can easily increase that in a server setting. Technically you can set it up all the way to 32 MB if you want to use JetStream and up to 64MB if you want to just use Core NATS. However many would advise you to not increase it over 8 or 16 MB because the large the message are the more the potential for things like latency spikes (think 'head of the line blocking') increased memory management, increased slow consumers, etc...
Essentially their base product (NATs) has a lot of performance but trades it off for reliability. So they add Jetstream to NATs to get reliability, but use the performance numbers of pure NATs.
I got burned by MongoDB for doing this to me, I won’t work with any technology that is marketed in such a disingenuous way again.
Can you point to where they are using core NATS numbers to describe Jetstream?
The issue in the docs was that there are no available Jetstream numbers, so I talked over a video call to the field CTO, who cited the base NATs numbers to me, and when I pressed him on if it was with Jetstream he said that it was without: so I asked for them with Jetstream enabled and he cited the same numbers back to me. Even when I pressed him again that “you just said those numbers are without Jetstream” he said that it was not an issue.
So, I got a bit miffed after the call ended, we spent about 45 minutes on the call and this was the main reason to have the call in the first place so I am a bit bent about it. Maybe its better now, this was a year ago.
I’m telling you why I am skeptical of any tech that intentionally obfuscates trade-offs, I’m not making a comparison on which of these is worse; and I don’t really care if people take my anecdote seriously either: because they should make their own conclusions.
However it might help people go in to a topic about performance and reliability from a more informed position.
The other day i was listening to a podcast with their ceo from maybe 6 months ago, and he talked quite openly about how jetstream and consumers add considerable drag compared to normal pubsub. And, more generally, how users unexpectedly use and abuse nats, and how they've been able to improve things as a result.
It looks like you got frustrated by my refusing to give figures of performance for JetStream: I always say in meetings that because there are too many factors that affect greatly JetStream performance (especially compared to Core NATS which mostly just depends on the network I/O) I can not just give any number as that would likely not accurately reflect (better or worse!) the number that you would actually see in your own usage. And that rather you should use the built-in `nats bench` tool to measure the performance for yourself for your kind of traffic requirements and usage patterns, in your target deployment environment and HA requirements.
On top of that, the performance of the software itself is still evolving as we release new versions that improve things and introduce new features (e.g. JetStream publication batches, batched direct gets) that greatly improve some performance numbers.
I assure you that I just don't want to give anyone some number and then you try it for yourself and you can't match those numbers, nothing more! We literally want you to measure the performance for youself rather than to give you some large number. And that's also why the docs don't have any JetStream performance numbers. There is no attempt at any kind of disingenuity, marketing, or pulling wool over anyone's eyes.
And I would never ever claim that JetStream yields the same performance numbers as Core NATS, that's impossible! JetStream does a lot more and involves a lot more I/O than Core NATS.
However, if I get pressed for numbers in a meeting: I do know the orders of magnitude that NATS and JS operate at, and I will even be willing to say with some confidence that Core NATS performance numbers are pretty much always going to be up to the 'millions of messages per second'. But I will remain very resistant to making any claim any specific JS performance numbers because in the end the answer are 'it depends' and 'how long is a piece of string' and you can scale JetStream throughput horizontally using more streams just like you can scale Kafka's throughput by using more partitions.
Now in some meetings some people don't like that non-answer and really want to hear some kind of performance number so I normally turn the question and ask them what their target message rates and sizes are going to be. If their answer is in the 'few thousands of messages per second' (like it is in your case if I'm not mistaken about the call in question) then, as I do know that JetStream typically comfortably provides performance well in excess of that, I can say with confidence that _at those kinds message rates_ it doesn't matter whether you use Core NATS or JetStream: JetStream is plenty fast enough. That's all I mean!
The delivery guarantees section alone doesn’t make me trust it. You can do at least once or at most once with kafka. Exactly once is mostly a lie, it depends on the downstream system: unless going back to the same system, the best you can do is at least once with idempotancy
If you need all those things, there just are not a lot of options.
In this case it's something different - this was an honest question, and received two useful replies, so why downvote?! The mental model of people using Kafka is useful to know - in this case the published data being more log-like than stream-like since it's retained per a TTL policy, with each "subscriber" having their own controllable read index.
The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better.
Using S3 was better, but we still hit data loss problems (on our end, not S3 's, to clear) which isn't great when you need to bill on some of that data.
Then I heard about Kafka, bought a copy of I <3 Logs, and decided that maybe Kafka was worth the complexity, and boom, it was. No more data loss, and a happier business management.
I was headhunted for my current company for my Kafka experience. First thing I realised when I looked at the product was - "Ah, we don't need Kafka for this."
But the VP responsible was insistent. So now I spend a lot of time doing education on how to use Kafka properly.
And the very first thing I start with is "Kafka is not a queue. It's a big dumb pipe that does very smart things to move data efficiently and with minimal risk of data loss - and the smartest thing it does, it choosing to be very dumb.
Want to synchronously know if your message was consumed? Kafka don't care. You need a queue."
Good luck!
Also, SQS isn't pub/sub. Kafka and SQS really have very different use cases.
however, my latest wrapper for jedis does seem to be holding up and haven't had too many issues, but I have a very robust checking for dropped connections.
I used RabbitMQ a few years back on a C++ project, and at the time (has anything changed?) the best supported C++ client library seemed to be AMQP-CPP which isn't multi-thread safe, therefore requiring an application interface layer to need to be written to address this in a performant way.
In our case we wanted to migrate a large legacy system from CORBA (point to point) to a more flexible bus-based architecture, so we also had to implement a CORBA-like RPC layer on top of Rabbit, including support for synchronous delivery failure detection, which required more infrastructure to be built on top of AMQP-CPP. In the end the migration was successful, but it felt like we were fighting AMQP-CPP a lot of the way.
"To follow the hype train, Bro" is often the real answer.
> If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka.
Or just freaking MQTT.
MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messaging system (which you should not do anyway).
Companies get standard tech stacks people are happy to work with, because working with them gets people experience with tech stacks that are standard at many companies. It's a virtuous cycle.
And sure even if you need just a specific thing, it's often better to go slightly overkill for something that's got millions of stack overflow solutions for common issues figured out. Vs picking some niche thing that you are now 1 of like six total people in the entire world using in prod.
Obviously the dose makes the poison and don't use kafka for your small internal app thing and don't use k8s where docker will do, but also, probably use k8s if you need more than docker instead of using some weird other thing nobody will know about.
And 5 years later the responsible of the decision left the company with a giant pile of mess behind him/her.
But let's see things positively: he can now add "Kafka at scale" on the CV.
Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work.
We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many apps across our entire stack have the same basic requirement, publish a lot of data which people want to analyse somewhere later. Kafka gives us a single mechanism to do that.
So now my question. I've never used MQTT before. What are the benefits of using MQTT in our setup vs using kafka?
MQTT is a publish/subscribe protocol for large scale distributed messaging, often used in small embedded devices or factories. It is made for efficient transfer of small, often byte sized payloads of IoT device data. It does not replace Kafka or RabbitMQ - messages should be read off of the MQTT broker as quickly as possible. ( I know this from experience - MQTT brokers get bogged down rapidly if there are too many messages "in flight")
A very common pattern is to use MQTT for communications, and then Kafka or RabbitMq for large scale queuing of those messages for downstream applications.
That is currently the problem.
A lot of usage of Kafka I have seen in the wild are not for log stream or queing but deployed as a simple pub/sub messaging service because "why not".
Kafka isn’t a queue. It’s overkill to use it as one.
Kafka is a great place to persist data for minutes, hours or days before it’s processed. It fully decouples producers and consumers. It’s also stupidly complex and very hard to operate reliably in an HA configuration.
MQTT is good for when data needs to leave or enter your cloud, but persistence is bolted on (at least it is in mosquitto), so a crash means lost data even though you got a PUBACK.
[0] https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...
Sure, it can do it. But it's not efficient or what it's good at.
You're running a distributed system. They aren't simple.
Especially on AWS. AWS is really a double-bladed sword. Yeah, you'll get tutorials to set up whatever distributed system pretty quickly, but your nodes aren't nearly as reliable. Your networking isn't nearly as reliable. Your costs aren't nearly as reliable and administration. Headaches go up in the long run
But if you don't understand distributed systems, it almost makes it worse because its tempting to segment the system across dozens of microservices which all have to talk with each other and synchronize, and the whole thing becomes a buggy, slow clusterfuck.
That’s not coded “you’re reinventing the wheel”; WarpStream had some significant drawbacks, so I’m truly curious about different approaches in the message-log-backed-by-blob-store space.
The dimensions we focus on are number of streams (unlimited, so you can do granular streams like per user or session), internet accessibility (you can generate finely-scoped access tokens that can be safely used from clients like CLIs or browsers), and soon also massive read fanout for feed-like use cases.
The arguments in the article are very compelling. But as soon as you choose Kafka you realize the things you hate.
Many of the reasons are stupid things - like it uncovers otherwise unimportant bugs in your client code. Or that it just makes experimenting a hassle because it enforces poking around in lots of different places to do something. Or that writing and maintaining the compulsory integration test takes weeks of your time.
Sure - you can replay your data - but not until you have fixed all the issues for that special case in your receiving service.
I think maybe my main gripe (for us) was that it was a difficult to get an understanding what is actually inside your pipe. Much easier to have that in a solid state in s3?
At the end of they day you get annoyed because it slows you down. In particular when you are a small localized team.
I don’t remember where I read this, but someone made the observation that writing a stream processing system is about 3x harder than writing a batch system, exactly for all the reasons you mentioned. I’m looking at replacing some of our Kafka usage with a clickhouse table that’s ordered and partitioned by insertion time, because if I want to do stuff with that data stream, at least I can do a damn SQL query.
At least compared to building a batched pipeline with SQL. I think you should really think hard whether you really need a streaming pipeline. And even if you find that you do, it may be worthwhile to make a batched pipeline as your first implementation.
I did exactly what you describe in my previous job. In the beginning with reluctance from our architects who wanted to keep banging the dead horse and did not understand the power of SQL "SQL is not real programming, engineers write java" (ok maybe I deserve a straw-man yellow card here, they don't deserve all of that). But I think they understood after a while.
With AWS Athena and Airflow. Good luck, consider me your distant moral support.
Distributed ledgers like Bitcoin do store transitions as events, and that's because nodes need the transitions to valid the next state. So you might say that Bitcoin is a widely run piece of software, using an event driven architecture.
Not every system needs to have all of it's state transitions available for for efficient reading. And often times you can derive the state transition from the previous and next state if you really need them (Git does that). Even though Git can compute all of the state transitions for the system, it doesn't store events, it stores snapshots.
Like the article says, fan-out is a key design characteristic. There are "redis streams" now but they didn't exist back then. The durability story and cluster stories aren't as good either, I believe, so they can probably take you so far but won't be as generally suitable depending on where your system goes in the future. There are also things like RedPanda that speak Kafka w/o the Java.
However, if you CAN run on a single node w/o worrying about partitioning, you should do that as long as you can get away with it. Once you add multiple partitions ordering becomes hard to reason about and while there are things like message keys to address that, they have limitations and can lead to hotspotting and scaling bottlenecks.
But the push/pop based systems also aren't going to give you at-least-once guarantees (looks like Redis at least has a "pop+push" thing to move to a DIFFERENT list that a single consumer would manage but that seems like it gets hairy for scaling out even a little bit...).
I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?
Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly slow ones, when I in fact very much could, for 25ish years.
Java programs have a feel and it’s “stuttery resource hog”. Whatever may be possible with the platform, that’s the real-world experience.
With over 15 years of professional experience since then, my perspective has shifted: Java demonstrates its strength when stability, performance, and scalability are required (e.g. bloody enterprise)
A common misconception comes from superficial benchmarking. Many focus solely on memory consumption, which often provides a distorted picture of actual system efficiency.
I can point to EU-scale platforms that have reliably served over 100 million users for more than a decade without significant issues. The bottleneck is rarely the language itself, it is the depth of the team’s experience.
When other languages can do the same thing with an order of magnitude less RAM, any other efficencies in the system tend to be overshadowed by that and be the sticking point in peoples memories.
You may argue that holding on to this extra memory makes subsequent calls and reads quicker etc, but in my experience generally people are willing to sacrifice milliseconds to gain gigabytes of memory.
I haven't worked with too much Java, but I suspect that the distaste many have for it is due to its wide adoption by large organizations and the obfuscating "dressed up" tendency of the coding idioms used in large organizations.
The runtime isn't inherently slow, but maybe it's easier to write slow programs in Java.
jokes aside, we got a shift in the industry where many java programs were replaced by electron-like programs which now take 20x memory
While for typical backend situations, reference counting has a crazy high throughput overhead, doing atomic inc/decs left and right, that instantly trashes any kind of cache, and does it in the mutator thread that would do the actual work, for the negligible benefit of using less memory. Meanwhile a tracing GC can do (almost) all its work in another thread, not slowing down the actually important business task, and with generational GCs cleaning up is basically a no-op (of just saying that this region can now be reused).
It's a tradeoff as everything in IT.
Also, iPhone CPUs are always a generation ahead, than any android CPU, if not more. So it's not really Apples to oranges.
* languages like c++ and Rust simply don’t allocate as much as Java, instead using value types. Even C# is better here with value types being better integrated.
* languages like c++ and Rust do not force atomic reference counting. Rust even offers non atomic ref counting in the standard library. You also only need to atomic increment / decrement when ownership is being transferred to a thread - that isn’t quite as common depending on the structure of your code. Even swift doesn’t do too badly here because of the combination of compiler being able to prove the permission of eliding the need for reference counting altogether and offering escape hatches of data types that don’t need it.
* c++, Rust, and Swift can access lower level capabilities (eg SIMD and atomics) that let them get significantly higher throughput.
* Java’s memory model implies and requires the JVM to insert atomic accesses all over the place you wouldn’t expect (eg reading an integer field of a class is an atomic read and writing it is an atomic write). This is going to absolutely swamp any advantage of the GC. Additionally, a lot of Java code declares methods synchronized which requires taking a “global” lock on the object which is expensive and pessimistic for performance as compared with the fine-grained access other languages offer.
* there’s lots of research into ways of offering atomic reference counts more cheaply (called biased RC) which can safely avoid needing to do an atomic operation in places completely transparently and safely provided the conditions are met .
I’ve yet to see a Java program that actually gets higher throughput than Rust so the theoretical performance advantage you claim doesn’t appear to manifest in practice.
Of course with manual memory management you may be able to write more efficient programs, though it is not a given, and comes at the price of a more complicated and less flexible programming model. At least with Rust, it is actually memory safe, unlike c++.
- ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to use atomic instructions. This may or may not matter, I'm not claiming it's worse, especially when used very rarely as is the case with typical c++/rust programs.
> You also only need to atomic increment / decrement when ownership is being transferred to a thread
Java can also do on-stack replacement.. sometimes.
- regarding lower level capabilities, java does have an experimental Vector API for simd. Atomics are readily available in the language.
- Java's memory model only requires 32-bit writes to be "atomic" (though in actuality the only requirement is to not tear - there is no happens before relation in the general case, and that's what is expensive), though in practice 64-bit is also atomic, both of which are free on modern hardware. Field acces is not different from what rust or c++ does, AFAIK in the general case. And `synchronized` is only used when needed - it's just syntactic convenience. This depends on the algorithm at hand, there is no difference between the same algorithm written in rust/c++ vs java from this perspective. If it's lockless, it will be lockless in Java as well. If it's not, than all of them will have to add a lock.
The point is not that manual memory can't be faster/more efficient. It's that it is not free, and comes at a non-trivial extra effort on developers side, which is not even a one-time thing, but applies for the lifetime of the program.
That’s a bold claim to make that doesn’t seem to actually be true from my experience. Your 5ghz CPU can probably do ~20 billion non atomic reference adjustments whereas your GC system has to have atomics all over the place or it won’t work and atomics have parasitic performance on unrelated code due to bus locks and whatnot.
> Java can also do on-stack replacement.. sometimes
That’s not what this is. It’s called hybrid RC and it applies always provided you follow the rules.
> The point is not that manual memory can't be faster/more efficient. It's that it is not free, and comes at a non-trivial extra effort on developers side, which is not even a one-time thing, but applies for the lifetime of the program.
The argument here is not about developer productivity - the specific claim is that the Java GC lets you write higher throughput code than you would get with Rust or C++. That just isn’t true so you end up sacrificing throughput AND latency AND peak memory usage. You may not care and are fine with that tradeoff, but claiming you’re not making that tradeoff is not based on the facts.
No, that has never been the specific claim - you can always write more efficient code with manual memory management, given enough time, effort and skill. I wasn't even the one who brought up c++ and rust. Like literally I write this twice in my comment.
What I'm talking about is reference counting as a GC technique vs tracing as a GC technique, all else being equal - it would be idiotic to compare these two if no other "variable" is fixed. (Oh and I didn't even mention the circular references problem, which means you basically have to add a tracing step either-way unless you restrict your language so that it can't express circular stuff).
As for the atomic part, sure, if all it would do is non-atomic increments then CPUs would be plenty happy. And you are right that depending on how the tracing GC is implemented, it will have a few atomic instructions. What you may miss is how often each run. On almost every access, vs every once in a while on a human timescale. Your OS scheduler will also occasionally trash the performance of your thread. But this is the actually apples to oranges comparison, and both techniques can do plenty of tweaks to hide certain tradeoffs, at the price of something else.
And I also mention that the above triad of time, skill and effort is not a given and is definitely not free.
As for “skill” this is one thing that’s super hard to optimize for. All I can do is point to existence proofs that there’s no mainstream operating system, browser or other piece of high performance code written in Java and it’s all primarily C/C++ with some assembly with Rust starting to take over the C/C++ bits. And at the point where you’re relegating Java to being “business” logic, there’s plenty of languages that are better suited for that in terms of ergonomics.
Java is used in HFT (well, there are two types of "high frequency", one where general purpose CPUs are already too slow, where it obviously doesn't apply (neither do rust or c++)) - but sure, I wouldn't write a runtime or other piece of code in Java where absolute control over the hardware is required. But that's a small niche only. What about large distributed systems/algorithms? Why is Java over-represented in this niche (e.g. Kafka, Elasticsearch, etc)?
> And at the point where you’re relegating Java to being “business” logic, there’s plenty of languages that are better suited for that in terms of ergonomics.
That's subjective.
AFAIK that doesn’t really happen. They won’t insert atomic accesses anywhere on real hardware because the cpu is capable of doing that atomically anyway.
> Additionally, a lot of Java code declares methods synchronized which requires taking a “global” lock on the object which is expensive and pessimistic for performance as compared with the fine-grained access other languages offer.
What does this have to do with anything? Concurrency requires locks. Arc<T> is a global lock on references. “A lot” of Java objects don’t use synchronized. I’d even bet that 95-99% of them don’t.
Concurrency does not require locks. There’s entire classes of lock free and wait free algorithms. Arc<T> is also not a lock - it uses atomics to manage the reference counts and no operation on an Arc needs to wait on a lock (it is a lock-free container).
> “A lot” of Java objects don’t use synchronized. I’d even bet that 95-99% of them don’t.
Almost all objects that are used in a concurrent context will likely feature synchronized, at least historically. That’s why Hashtable was split into HashMap (unsynchronized) and ConcurrentHashMap (no longer using synchronized). Thats why you have StringBuffer which was redone into StringBuilder.
> and to implement a ConcurrentHashMap in Rust you would still need to lock
There’s many ways to implement concurrency safe hashmaps (if you explicitly needs such a data structure as the synchronization mechanism) without locks. Notably RCU is such a mechanism (really neat mechanism developed for the kernel although not super user friendly yet or common in userspace) and there are also generational garbage techniques available (kind of similar to tracing GC conceptually but implemented just for a single data structure). A common and popular crate in Rust for this is DashMap which doesn’t use locks and is a concurrency safe hashmap.
Still not in the standard library. The only way in Rust is to use a global lock around map. Seems to be worse than the situation in Java. You could implement the same thing and use a third party library in Java too. So your original point of "everything uses a global lock" is "overly strong"
The strengths and weaknesses of the standard library aren’t relevant. But if we’re going there, the reason they’re not in the Rust standard library is likely in practice concurrent data structures are an anti pattern - putting a lock around a data structure doesn’t suddenly solve higher order race conditions which is something a lot of Java programmers seem to believe because the standard library encourages this kind of thinking.
As for “my comment” about “global lock” (your words not mine), it’s that the implicit lock that’s available on every object is a bad idea for highly concurrent code (not to mention the implicit overhead that implies for every part of the object graph regardless of it being needed anywhere). Don’t get me wrong - Java took a valiant effort to define a solid memory model for concurrency when the field was still young. Many of the ideas didn’t pan out and are antipatterns these days for high performing code. Of course none of that pertains to the original point of the conversation - tracing GCs have significantly more overhead in practice because they’re very difficult to be opt in, carry quite a penalty if not, Rc/Arc is much better as it’s possible to do opt-in when you need shared ownership (which isn’t always), and in practice loops don’t come up often enough to matter and when they do there’s still solutions. In other words tracing GCs drop huge amounts of performance on the floor and you can read all the comments to see how the claims are “it’s more efficient than Rc”, or “performance is free” or even “it doesn’t matter because the programmer is more efficient”. I’d buy the efficiency argument when the only alternative was C/C++ and came with serious memory safety baggage, but not any of the others and memory safety without sacrificing performance of C++ in my view is a solved problem with Rust.
The JVM tends to hold onto memory in order to make things faster when it does wind up needing that memory for actual stuff. However, how much it holds on to, how the GC is setup, etc are all tunable parameters. Further, if it's holding onto memory that's not being used, these are prime candidates to be stored in virtual memory which is effectively free.
Nonetheless, tracing GCs do have some memory overhead in exchange for better throughput. This is basically the same concept as using a buffer.
-----
And can you tell which of these websites use Java from "the feel"? AWS cloud infra, a significant chunk of Google, Apple's backends, Alibaba, Netflix?
Is C++ bloated because of the memory Chrome uses?
I’ve never seen another basic tech used to develop other programs that’s so consistently obvious from its high resource use and slowness, aside from the modern web platform (Chrome, as you put it). It was even more obvious back when we had slower machines, of course, but Java still stands out. It may be able to calculate digits of Pi in a tight loop about as fast as C, but real programs are bloated and slow.
Especially that like half of the web runs on Java, you just have absolutely no idea when it silently does its job perfectly.
Source?
W3 seems to think its more like ~5%
From the website.
And 76% of these websites is PHP, which seems to mean.. they can determine PHP more easily for a website (nonetheless, there are indeed a lot of WordPress sites, but not this amount).
They hog RAM, are slow, and are a bitch to configure.
The way most Java code is written is terrible Enterprise factory factory factory.
Maybe I have been lucky, or that the practice is more common in certain countries or eco systems? Java has been a very productive language for me, and the code has been far from the forced pattern usage that I have read horror stories about.
Depending on how much memory is used by the Thread stack (presumably 1M-512K by default, allegedly 128K with Alpine base images) that's your 1G-500M heap space usage improvement right off the bat.
The migration from JDK17 to JDK21 was uneventful in production. The only issue is limited monitoring as a thread dump will not show most virtual threads and the micrometer metrics will not even collect the total number of active virtual threads. It's supposed to work better in JDK24.
The Spring Framework directly supports virtual threads with "spring.threads.virtual.enabled=true" but I haven't tried it to comment.
Go's GC is tuned more for latency at the expense of throughput (not sure if it still applies, but Go was quite literally stopping the "business" mutator threads when utilisation got higher to be able to keep up with the load - Java's default GC is tuned for a more balanced approach, but it can deliver it at very high congestion rates as well. Plus it has a low-latency focused GC which has much better latency guarantees, and it trades off some throughput in a consistent manner, so you can choose what fits best). The reason it might sometimes be more efficient than Java is simply value types - it doesn't create as much garbage, so doesn't need as good a GC in certain settings.
Rust code can indeed be better at both metrics for a particular application, but it is not automatically true, e.g. if the requirements have funny lifetimes and you put a bunch of ARC's, then you might actually end up worse than a modern tracing GC could do. Also, future changes to the lifetimes may be more expensive (even though the compiler will guide you, you still have to make a lot of recursive changes all across the codebase, even if it might be a local change only in, say, Java), so for often changing requirements like most business software, it may not be the best choice (even though I absolutely love Rust).
Worse latency every ten minutes tends to be fine.
Such as? The only area where you have to "drop" features is high-frequency trading, where they often want to reach a steady-state for the trading interval with absolutely no allocations. But for HFT you would have to do serious tradeoffs for every language.
In my experience, vanilla java is more than fine for almost every application - you might just benchmark your code and maybe add some int arrays over an Integer list, but Java's GC is an absolute beast, you don't have to baby it at all.
The reason is quite well known. Supporting multiple languages is a cost. If you only have to support one language, everything is simpler and cheaper.
With Java, you can write elegant code these days, rely on ZGC, not really worry too much about GC and get excellent performance with quick development cycles for most of your use cases. Then with the same language and often in the same repo (monorepo is great) you can write smarter code for your hot path in a GC free manner and get phenomenal performance.
And you get that with only having one build system, one CI pipeline, one deployment system, some amazing profiling and monitoring tooling, a bunch of shared utility code that you don't have to duplicate, and a lot more benefits.
That's the reason to choose Java.
Of course, if you're truly into HFT space, then they'll be writing in C, C++ or on FPGAs.
https://docs.spring.io/spring-framework/docs/2.5.x/javadoc-a...
Everything.
Why do you think Kubernetes is NOT written in Java?
Golang has little to distinguish itself technically. It has a more modern std lib (for now) and isn't Oracle.
Which aren't trivial, but they aren't Trump cards.
Nope.
None of what you said are any of the reasons given that it WAS written in Java already [0] but rewrote it all in Go explicitly because of its performance, concurrency and single binary distribution characteristics.
Those were enough technical advantages to abandon any thought of a production-grade version of k8s in Java.
[0] https://archive.fosdem.org/2019/schedule/event/kubernetesclu...
Because someone wanted a new, shiny toy.
It seems to me that perhaps it wasn’t the languages fault but the authors.
Kafka is used because the Java folks don't want to learn something new due to job security, even though there are faster and compatible alternatives that exist today.
Rather use Redpanda, than continue to use Kafka and then complain about how resource intensive it is alongside zookeeper and all the circus the comes with it and make AWS smile as you're losing hundreds of thousands a month.
1: https://cwiki.apache.org/confluence/display/kafka/kip-500:+r...
https://www.confluent.io/blog/zookeeper-to-kraft-with-conflu...
It was later open-sourced in 2011 and became a top-level project under the Apache Software Foundation in 2012.
The creators went on to co-found Confluent, a company that provides commercial support and enterprise features around Kafka.
The look of the page -- that's Substack's default UI, you can't control it too much. The other images are created by me.
I'm simply curious what parts give that "cheap" look so I can improve. On Reddit I've had massive amounts of downvotes because they guess the content is AI, when in fact no AI is used in the creation process at all.
One guess I have is the bullet points + bolding combo. Most AIs use a ton of that, and rightly so, because it aids in readability.
Are they really generating data at this scale?
I cant even imagine a system which creates and stores this much data this fast.
Looking at the 2012 paper, it implies a 1.35TB per day (they store 9.5TB across all topics at 7d retention)
https://www.linkedin.com/blog/engineering/infrastructure/int...
I believe these companies save literally every ounce of data they can find. Once you have the infra and teams for it, it seems easy to make a case for storing something.
Similarly, Uber has shared they push 89 GB/s through Kafka - 7.7 PB/s. People always ask me - what is a taxi/food-delivery app storing so much
17PB = about 567 of those drives... being totally filled... per day.
I was hoping somebody would come and say this is a simple spelling error or something.
The cost of the drives alone seems astronomical, let alone the logistics of the data center keeping up with storing that much data.
EDIT: I have just realised that they are probably only processing at this speed, rather than storing it, can anyone confirm if they store all the logs they process?
That comes out to 87 disks a day. Assuming a 7 day retention period (this is on the high side), it’s not unthinkable to have a 600-1800 disk deployment (accounting for replication copies)
Yep. Whole week can be easily stored in 1-2 racks.
From Wikipedia.
I know, Claude was also enthusiastic about it.
You don’t need push, it’s just a performance optimization that almost never justifies using a whole new tool.
Go back to the article, it wasn't about event-sourcing or replacing a DB for application code.
Dismissing this as «just a social network» understates the real constraints: enormous scale, global privacy rules, graph queries, near-real-time feeds and abuse controls. Periodic DB queries can work at small scale, but at high volume they either arrive late or create bursts that starve the primary. Capturing changes once and pushing them through a distributed transaction log such as Kafka evens out load, improves data timeliness and lets multiple consumers process events safely and independently. It does add operational duties – schema contracts, idempotency and retention – yet those are well-understood trade-offs. The question is not push versus pull in the abstract, but which approach meets the timeliness, fan-out and reliability required.
> You don’t need push, it’s just a performance optimization that almost never justifies using a whole new tool.
It is not about drama but about fit for purpose at scale.
Pull can work well for modest workloads or narrow deltas, especially with DB features such as incremental materialised views or change tables. At large scale, periodic querying becomes costly and late: you either poll frequently and hammer the primary, or poll infrequently and accept stale data. Even with cursoring and jitter, polls create bursty load and poor tail latencies.
Push via change data capture into a distributed log such as Kafka addresses such pain points. The log decouples producers from consumers, smooths load, improves timeliness and lets multiple processors scale independently and replay for backfills. It also keeps the OLTP database focused on transactions rather than fan-out reads.
This is not free: push introduces operational work and design care – schema contracts, ordering being per-partition, duplicate delivery and idempotency, back-pressure and retention governance including data-protection deletes. The usual mitigations are the outbox pattern, idempotent consumers, DLQ's and documented data contracts. The data processing complexity now belongs in each consumer, not the data processing engine (e.g. a DB).
Compute–storage separation in modern databases raises single-cluster ceilings for storage and read scale, yet it does not solve single-writer limits or multi-region active-active writes. For heavy write fan-out and near-real-time propagation, a CDC-to-log pipeline remains the safer bet.
To sum it up, both pull and push are valid – engineering is all about each specific use case assessment and the trade-off analysis. For small or bounded scopes, a well-designed pull loop is simpler. As scale, fan-out and timeliness requirements grow, push delivers better timeliness, correctness and operability.
> LinkedIn used site activity data (e.g. someone liked this, someone posted this)1 for many things - tracking fraud/abuse, matching jobs to users, training ML models, basic features of the website (e.g who viewed your profile, the newsfeed), warehouse ingestion for offline analysis/reporting and etc.
Who controls the database? Is it the fraud/abuse team responsible for the migrations? Does the ML team tell the Newsfeed team to stop doing so many writes because it's slowing things down?
https://www.infoq.com/news/2025/06/linkedin-northguard-xinfr...
> According to LinkedIn's engineers, Kafka had become increasingly difficult to manage at LinkedIn's scale (32T records/day, 17 PB/day, 400K topics, 150 clusters).
Wtf is LinkedIn doing that they create 17 PB/day?????
Based on my work managing large-scale Kafka environments across critical sectors, I have identified that their stability and efficiency are upheld by a set of essential practices and tools. These are the non-negotiable pillars for success:
Health Checks & Observability: Proactive cluster health monitoring and complete visibility into the data flow are paramount.
Failure Management: Implementing dedicated portals and processes for handling Dead-Letter Queues (DLQs) ensures that no critical information is lost during failures.
Automation & DevOps: I leverage Strimzi for Kubernetes-native cluster management, orchestrating it through ArgoCD and GitOps practices. This ensures consistent, secure, and repeatable deployments.
The correct application of these engineering principles allows for remarkable results. For instance, at a large fashion retail group, I successfully scaled an environment to handle a peak traffic of 480,000 TPS. This high-availability system is efficiently maintained by a lean operational team of just two junior-to-mid-level professionals.
From my perspective, success in adopting Kafka is determined by the business context and the maturity of the applied software engineering. The investment in a well-planned architecture and a robust support ecosystem has a clear return, paying for itself through a significant reduction in operational costs (OPEX) within an estimated two-year period.
Taming Kafka isn't about new, complex secrets. It's about applying the same robust software engineering and architecture fundamentals we've relied on for +50 years (Software Engineering). The platform is new (2011), the principles are not.
Perhaps the fact that I'm not a native English speaker may have caused this confusion. I just made sure my text was written correctly in English with a translator.
Generated comments are what people here feel strongly about rejecting. Not just LLM-written ones, but also comments that have been filtered through translators/editors/checkers and so on.
Readers are becoming hypersensitive to these, so it is better to just write in your own voice and post that. The benefit of authenticity outweighs the cost of a few grammar or spelling errors.