Building a Streaming Platform in Go for Postgres | Hacker News Reader

Building a Streaming Platform in Go for Postgres

blog.peerdb.io

167 pointssaisrirampur49 comments

Moving to channels seems like a weird choice. Channels should basically always be an implementation detail not exposed in public APIs.

Author of the blog here, curious what a better alternative would be in this context. The channel has to be passed around for the producer and consumer to interface with each other. Are there better patterns for this?

Not the parent, but I personally dislike it when Go libraries use channels in their public APIs, as it forces a specific concurrency model on the consumer; in particular, channels are quite slow, being protected by an internal mutex, so you're always paying for the overhead no matter if you need it or not.

You also have to be very careful about managing the channel lifecycle. If you're not pulling (selecting from) the channel, the library will be permanently stuck. So you must now have a way to tell the library to stop sending, and it must cancel any in-flight send operations if you call producer.Stop() or whatever. In my experience libraries often have bugs in their channel code. It's far too easy to get deadlocks with channels that have interdependencies, and you have to be very careful about buffered versus unbuffered channels, as they behave differently.

A better API, in my opinion, is to offer a callback or single-method interface. Then the implementer of that callback or interface can choose to use channels internally if they desire, or they can use something else. You get the same backpressure support since you can treat it as synchronous.

After all, a channel's send interface is essentially just:

    type Channel[T any] interface {
        Send(T)
    }

But a "chan T" doesn't offer this flexibility.

My rule of thumb for channels is that they're goroutine glue, not an API primitive. Build APIs out of interfaces, not channels. The only thing that uses channels should be the one that's controlling the goroutines, because it's the thing that orchestrates them.

That said, it's not a hard rule. There are places where channels may have their place in a public API, though I'm not sure I can think of any examples off-hand.

this breaks select to send and is a terrible reduction in capability.

you can always wrap channels to make them worse and less capable, but your API should expose the more capable option.

I think it is a matter of preference. For me personally I use raw channels and goroutines all day every day and I really like using them. Channels are a core primitive in golang so I think it is worth getting familiar with them.

As you say being able to select is really nice too.

> as it forces a specific concurrency model on the consumer

I found your excuse above is really nonsense. when your program is in Golang, you've already picked side, the concerned concurrency model has already been chosen by the user.

we are not talking about one of random concurrency models, we are talking about channel based sychnronization and communication in golang, if you don't want that and consider it as an issue, you shouldn't be using golang in the first place.

Looks like the channel field is private in CDCRecordStream, but exposed by GetRecords. The callers mostly loop over Record objects. [1]

If I wanted to encapsulate iterating over a channel of Records, maybe it would be something like Go's io.Pipe function [2], which returns a PipeReader and PipeWriter? Except that it would work on Records rather than byte streams.

I don't have enough context to know if the extra encapsulation is a good idea in this case, though.

[1] https://github.com/search?q=repo%3APeerDB-io%2Fpeerdb%20GetR... [2] https://pkg.go.dev/io#Pipe

Please see this great talk by Bryan C. Mills touching on the subject: https://youtu.be/5zXAHh5tJqQ?t=421

Why have consumers and producers vs doing it all in one goroutine, utilizing some kind of connection pool?

Because then you are consuming, or producing, you can’t do both at the same time. You are either reading from a stream of data, or you are writing it. Using goroutines to separate these allows you to do both at the same time, as soon as data is available on the channel or you receive the signal to stop.

To get higher throughput we would need one goroutine to pull from the replication slot while the other is pushing to the target. The idea is to keep the Postgres connection useful and reading the slot while also pushing to the target asynchronously.

Use an iterator object that can use channels behind the scenes.

Using channels to talk between goroutines is the perfect choice. There’s plenty of go API’s that require a chan of a type.

Go's channels also have a lot of footguns and should be abstracted in a handle type in a public API.

No. Being the first class element in the go language, channel is nothing different from a []byte or a regular struct.

The whole purposes of having such built-in channel as a part of the core language is to encourage everyone to use it whenever possible, the reason is dead simple - it is much less likely to shoot yourself in the foot.

Except it itself is exposed via a public API. Context interface also exposes a chan

   Done() <- chan struct{}

What are you trying to say? That using a chan as a function argument can shoot yourself in the foot? The whole idea behind channels is to not shoot yourself in the foot with mutex's, deadlocks, etc. What are you trying to say? Wrapping a chan in a type handle is safer than passing a chan of a type? Nope.

That's not at all what I was referring to. It's not merely having channels as function arguments. Channels broadly have footguns around how closed and nil channels are treated and you can run into work contention/starvation issues accidentally, since they do have their own throughput limits. And wrapping them in your own handle types that's specific to the library's use case allows you to deal with those issues and swap out the implementation of the cross-goroutine bits without breaking the downstream API.

Channels broadly have footguns around how closed and nil channels are treated

I appreciate your concerns but one could say similar things about nil pointers or accessing slices out of range. Anything you use should be handled with care.

exactly.

you can't abstract over channels easily unless you're missing the point of channels.

I think the rule is: don't use a channel unless at some point you need a select with 2+ branches. The context.Done is a case in point. You're not just waiting around for a context to finish. You do two things: try some network request and wait to see if the context cancels before it finishes. Because it's made to work in a select block like that, there wasn't another, better thing to use than a channel for context.Done. But for just iterating through some stream, you take a context in and return an iterator object that will do all the channel stuff internally.

Well, I think read-only channels in public APIs are fine. Writable or write-only channels are dangerous and should be avoided given that the "close ownership" -- to coin a phrase -- is ambiguous. For the write end, I agree that a handle type is appropriate.

I think there are exceptions, for example Gos contexts expose a channel.

Seems fine to me. What's the problem?

channels seemed to fit very nicely in a gRPC streaming API call. But gRPC is not for everyone.

I don't think choosing channel itself is an issue, if it works for the said platform, that is great.

What I don't understand is why there should be such a blog article telling people that you decided to use channel in your go based program? This is like writing an article telling people that you decided to use the MMU when building an OS.

Thanks for posting this question. As explained in the article we started off with pull/push model (with configurable batching) and it worked well for streaming to data warehouses, where 30s+ was acceptable latency.

We added Queues as a supported target, where one of our users wanted single digit second latency. This is when we introduced channels. We agree that it is a small change. But the latency improvements were significant and wanted to share it with broader programming/Go community of Go Channels and the objective impact they can have.

Pull/Push itself is not the problem unless you have stats to back such implied claims. It is far more likely that your highly inefficient Pull/Push implementation caused the problem. Without any real insight on why the pull/push based approach is slower, there is nothing the "Go community" can get.

All valuable details were omitted, e.g. 1) how much data is being moved? a few megabytes? what is the state of art latency? maybe 0.001 second for that? when latency is still seconds huge, what can be further improved, in the go runtime or in the apps? etc

A highly efficient pull/push model without using go channel can easily beat your 1-5 seconds latency. A crappy implementation using channel can get worse.

Delay of several seconds is when you pass on information mouth to mouth, in 2023, that kind of delay doesn't justify a blog article on such fancy implementation. I'd suggest you to delete the implementation to rebuild from ground zero.

Thanks for the comment. We will aim to write a more deeper blog on this in the future.

We were observing 60<>40 ratio for Pull/Push, as specified in the blog and majority of the Push was target data-store (network) bound. So there was less room of optimization on the Go side.

The test was done with ~10-15K TPS on Postgres and the target was Azure Event Hubs, which has a limit of 1MB write batch size.

Yes, thank you.

I wonder if they have a setup without Docker. Not a fan, wonder why they even bother with Docker if they're using GO. Thats one of the great things about GO. Easy distributable that works on all platforms.

Not all of our code is in Go. PeerDB has multiple components: the workers, the UI and the query layer. Some of which is in Rust, Go and Typescript.

While it would certainly be possible to package them into individual binaries, I found it significantly easier to define the stack in a docker compose file with the requisite environment setup.

Reply on news.ycombinator.com