Hacker News Reader

Comment by earthboundkid | Hacker News Reader

Moving to channels seems like a weird choice. Channels should basically always be an implementation detail not exposed in public APIs.

View on HN

Author of the blog here, curious what a better alternative would be in this context. The channel has to be passed around for the producer and consumer to interface with each other. Are there better patterns for this?

Why have consumers and producers vs doing it all in one goroutine, utilizing some kind of connection pool?

To get higher throughput we would need one goroutine to pull from the replication slot while the other is pushing to the target. The idea is to keep the Postgres connection useful and reading the slot while also pushing to the target asynchronously.

Because then you are consuming, or producing, you can’t do both at the same time. You are either reading from a stream of data, or you are writing it. Using goroutines to separate these allows you to do both at the same time, as soon as data is available on the channel or you receive the signal to stop.

Not the parent, but I personally dislike it when Go libraries use channels in their public APIs, as it forces a specific concurrency model on the consumer; in particular, channels are quite slow, being protected by an internal mutex, so you're always paying for the overhead no matter if you need it or not.

You also have to be very careful about managing the channel lifecycle. If you're not pulling (selecting from) the channel, the library will be permanently stuck. So you must now have a way to tell the library to stop sending, and it must cancel any in-flight send operations if you call producer.Stop() or whatever. In my experience libraries often have bugs in their channel code. It's far too easy to get deadlocks with channels that have interdependencies, and you have to be very careful about buffered versus unbuffered channels, as they behave differently.

A better API, in my opinion, is to offer a callback or single-method interface. Then the implementer of that callback or interface can choose to use channels internally if they desire, or they can use something else. You get the same backpressure support since you can treat it as synchronous.

After all, a channel's send interface is essentially just:

    type Channel[T any] interface {
        Send(T)
    }

But a "chan T" doesn't offer this flexibility.

My rule of thumb for channels is that they're goroutine glue, not an API primitive. Build APIs out of interfaces, not channels. The only thing that uses channels should be the one that's controlling the goroutines, because it's the thing that orchestrates them.

That said, it's not a hard rule. There are places where channels may have their place in a public API, though I'm not sure I can think of any examples off-hand.

this breaks select to send and is a terrible reduction in capability.

you can always wrap channels to make them worse and less capable, but your API should expose the more capable option.

I think it is a matter of preference. For me personally I use raw channels and goroutines all day every day and I really like using them. Channels are a core primitive in golang so I think it is worth getting familiar with them.

As you say being able to select is really nice too.

> as it forces a specific concurrency model on the consumer

I found your excuse above is really nonsense. when your program is in Golang, you've already picked side, the concerned concurrency model has already been chosen by the user.

we are not talking about one of random concurrency models, we are talking about channel based sychnronization and communication in golang, if you don't want that and consider it as an issue, you shouldn't be using golang in the first place.

Looks like the channel field is private in CDCRecordStream, but exposed by GetRecords. The callers mostly loop over Record objects. [1]

If I wanted to encapsulate iterating over a channel of Records, maybe it would be something like Go's io.Pipe function [2], which returns a PipeReader and PipeWriter? Except that it would work on Records rather than byte streams.

I don't have enough context to know if the extra encapsulation is a good idea in this case, though.

[1] https://github.com/search?q=repo%3APeerDB-io%2Fpeerdb%20GetR... [2] https://pkg.go.dev/io#Pipe

Please see this great talk by Bryan C. Mills touching on the subject: https://youtu.be/5zXAHh5tJqQ?t=421

Use an iterator object that can use channels behind the scenes.

Using channels to talk between goroutines is the perfect choice. There’s plenty of go API’s that require a chan of a type.

Go's channels also have a lot of footguns and should be abstracted in a handle type in a public API.

Except it itself is exposed via a public API. Context interface also exposes a chan

   Done() <- chan struct{}

What are you trying to say? That using a chan as a function argument can shoot yourself in the foot? The whole idea behind channels is to not shoot yourself in the foot with mutex's, deadlocks, etc. What are you trying to say? Wrapping a chan in a type handle is safer than passing a chan of a type? Nope.

exactly.

you can't abstract over channels easily unless you're missing the point of channels.

I think the rule is: don't use a channel unless at some point you need a select with 2+ branches. The context.Done is a case in point. You're not just waiting around for a context to finish. You do two things: try some network request and wait to see if the context cancels before it finishes. Because it's made to work in a select block like that, there wasn't another, better thing to use than a channel for context.Done. But for just iterating through some stream, you take a context in and return an iterator object that will do all the channel stuff internally.

That's not at all what I was referring to. It's not merely having channels as function arguments. Channels broadly have footguns around how closed and nil channels are treated and you can run into work contention/starvation issues accidentally, since they do have their own throughput limits. And wrapping them in your own handle types that's specific to the library's use case allows you to deal with those issues and swap out the implementation of the cross-goroutine bits without breaking the downstream API.

Channels broadly have footguns around how closed and nil channels are treated

I appreciate your concerns but one could say similar things about nil pointers or accessing slices out of range. Anything you use should be handled with care.

No. Being the first class element in the go language, channel is nothing different from a []byte or a regular struct.

The whole purposes of having such built-in channel as a part of the core language is to encourage everyone to use it whenever possible, the reason is dead simple - it is much less likely to shoot yourself in the foot.

Well, I think read-only channels in public APIs are fine. Writable or write-only channels are dangerous and should be avoided given that the "close ownership" -- to coin a phrase -- is ambiguous. For the write end, I agree that a handle type is appropriate.

I think there are exceptions, for example Gos contexts expose a channel.

Seems fine to me. What's the problem?

channels seemed to fit very nicely in a gRPC streaming API call. But gRPC is not for everyone.

Reply on news.ycombinator.com