Hacker News Reader

Comment by dre85 | Hacker News Reader

Very interesting article! I was left wondering though in what ways the queue abstraction solution failed?

for this specific example, I think the shared library is not the correct approach. Queues work for simple fifo behavior, but in this case they also need fairness (and perhaps other logic, like rate limiting per client, different priority for certain clients etc)

For example, "Customer-A is clogging up the work queue and starving other customers out". The solution to this could look something like linux's completely fair scheduler, where every client is allocated some amount of messages per interval of time. This means messages need to be processed in a different order then they are enqueued, and queues are not good at reordering messages.

I would suggest implementing the queue abstraction as a rest or grpc service, backed by a database like postgres, which holds message payloads and everything related to message state (in progress, retry after times, etc). Now we can implement all the necessary scheduling logic within our queue service.

They hinted at the fact that kafka is not actually a queue, and it especially has problems with fairness if you try to use it as a queue for unequally sized "jobs" and/or unequally sized consumers. Kafka is for the exceptional case; actual message queues are for the default/general case.

That makes sense, but it also doesn't really support the article's overall argument about the perils of "standardization without understanding". If what you're saying is true then it could very well be that the approach to standardize the queuing would have worked perfectly well had the senior dev chosen a different technology to underlay his library (ie not Kafka), right?

Yes technically the Kafka matter is a secondary to the author's main point, which is understanding each problem first. So yeah I would not waltz in and say, "Oh, no need for details, obviously you need to standardize on RabbitMQ not Kafka," without a second thought, even though it's likely that's roughly where we're gonna end up.

But even then, I am not going to write a custom library when the drivers for either product are perfectly adequate. Standardization works best when standards are flexible, and trying to force my homemade library on every team is inflexible (usually it's just arrogance).

Reply on news.ycombinator.com