Hacker News Reader

Comment by saisrirampur | Hacker News Reader

Thanks for posting this question. As explained in the article we started off with pull/push model (with configurable batching) and it worked well for streaming to data warehouses, where 30s+ was acceptable latency.

We added Queues as a supported target, where one of our users wanted single digit second latency. This is when we introduced channels. We agree that it is a small change. But the latency improvements were significant and wanted to share it with broader programming/Go community of Go Channels and the objective impact they can have.

View on HN

Pull/Push itself is not the problem unless you have stats to back such implied claims. It is far more likely that your highly inefficient Pull/Push implementation caused the problem. Without any real insight on why the pull/push based approach is slower, there is nothing the "Go community" can get.

All valuable details were omitted, e.g. 1) how much data is being moved? a few megabytes? what is the state of art latency? maybe 0.001 second for that? when latency is still seconds huge, what can be further improved, in the go runtime or in the apps? etc

A highly efficient pull/push model without using go channel can easily beat your 1-5 seconds latency. A crappy implementation using channel can get worse.

Delay of several seconds is when you pass on information mouth to mouth, in 2023, that kind of delay doesn't justify a blog article on such fancy implementation. I'd suggest you to delete the implementation to rebuild from ground zero.

Thanks for the comment. We will aim to write a more deeper blog on this in the future.

We were observing 60<>40 ratio for Pull/Push, as specified in the blog and majority of the Push was target data-store (network) bound. So there was less room of optimization on the Go side.

The test was done with ~10-15K TPS on Postgres and the target was Azure Event Hubs, which has a limit of 1MB write batch size.

Yes, thank you.

Reply on news.ycombinator.com