Hacker News Reader

With pipes between threads, the OS is able to suspend the reading thread when the buffer is empty, or suspend the writing thread when the buffer is full, because the OS "owns" the buffer and the threads. With distributed systems connected via Kafka for example, there is no "process" overseeing both ends that can suspend or resume hollistically, so its difficult for producers to know and respond to back pressure.

View on HN

Isn't the buffer then the TCP one, which has its own backpressure system?

TCP will only slow down the writer if the network or the Kafka broker can't keep up. It the Kafka consumer can't keep up, their backlog will grow unhindered until Kafka runs out of disk space.

I wish there was an easier built in way to do it, but a producer could read the backlog length of it's consumers and try to throttle itself.

Reply on news.ycombinator.com