Show HN: Rill – Composable concurrency toolkit for Go
github.com
github.com
Example of what it looks like:
// Convert a slice into a channel
ids := rill.FromSlice([]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil)
// Read users from API with concurrency=3
users := rill.Map(ids, 3, func(id int) (*User, error) {
return api.GetUser(ctx, id)
})
// Process users with concurrency=2
err := rill.ForEach(users, 2, func(u *User) error {
if !u.IsActive {
u.IsActive = true
return api.SaveUser(ctx, u)
}
return nil
})
// Handle errors
fmt.Println("Error:", err)
- Makes concurrent code composable and clean
- Works for both simple cases and complex pipelines
- Built-in batching support
- Order preservation when needed
- Centralized error handling
- Zero dependencies
Surprisingly, part of my inspiration came from scala (which I haven't touched since 2014). Back then Scala had transformable streams and the "Try" typethen.
To summarize, I believe Rill's performance would be fine for any problem solvable with Go channels. I've used it for a wide variety of use cases - while I'm not sure I can call them high-throughput, I've had pipelines transferring hundreds of GBs of data with no performance issues.
I found your library a few weeks ago when I was annoyed by nothing like this being built into the standard library. It’s been a breeze to use so far.
A neat trick I found to gauge bottlenecks in pipelines is using Buffers between steps and running a goroutine that periodically prints `len(buffered)/cap(buffered)`
I've seen that kind of pattern lead to TONS of surprise race conditions in user-code, because everyone always thinks that "it returned" means "it's done". Which is reasonable, nothing else is measurable on the caller side - changing that won't be noticeable by callers and may violate their expectations and cause crashes.
The current solution is to make pipeline stages context-aware (which is often happens automatically) and cancel the context before returning. This is the responsibility of the user and can lead to problems you described.
I haven't yet found a better solution to this. On the other hand, exact same thing happens if your function spawns a goroutine and returns. That goroutine would run until done, unless it's context aware.
Regarding the "Delay" and "infiniteBuffer" functions - these are part of the work on adding support for feedback loops to Rill. I haven't yet found a reliable and user friendly way to do it, so this work is on hold for now.
As the author of https://github.com/ahmetb/go-linq, it's hard to find adoption for libraries offering "syntactic sugar" in Go, as the language culture discourages those kind of abstractions and keeping the code straightforward.
I was wondering if this could be it... Any thoughts in that direction, please let me know!
(1) One for running the function graph, and (2) one for running the resource graph.
(1) https://github.com/purpleidea/mgmt/tree/master/lang/funcs/da... (2) https://github.com/purpleidea/mgmt/tree/master/engine/graph
TBQH, I think I'm decent at concurrency, but I never considered it my passion or expertise and while both of those _work_ I do know that I have some concurrency bugs hanging out, and I'd love real professional help here.
In the odd chance you'd be willing to hack on things with me, I'd be particularly elated. There might be a possible symbiosis!
I'm @purpleidea on Mastodon, Matrix (where we have an #mgmtconfig room) and a few other places in case you'd like to chat more!
I would call this parallelism rather than concurrency.
The main issue I have with this library's implementation is how errors are handled. Errors are retrieved rather than assigned- but assignment is preferable because it gets verified by tools. In my library I used a channel for errors- that gives ultimate flexiblitiy- it can be converted to wait and collect them to a slice or to perform a cancellation on first error.
TIL: HN doesn't like writing tests. The downvotes on this are hilarious. "Job security" ¯\_(ツ)_/¯.
You stay away from something you don't understand after 10 years of working with it? What kind of logic is that? Channels aren't magic.
Subtle bugs in what? Have you considered that maybe you have bugs because you aren't writing tests?
If you aren't unit testing that stuff, then how are you able to fix/change things and know it is resolved?
My experience is that I built a binary that had to run perfectly on 30,000+ servers across 7 data centers. It was full of concurrency. Without a litany of automated tests, there is no way that I would have trusted this to work... and it worked perfectly. The entire deployment cycle was fully automated. If it passed CI, I knew that it would work.
It wasn't easy, it took a lot of effort to build that level of testing. But it was also totally bug free in production, even over years of use and development.
Perhaps a less defensive posture might invite more discussion.
I can't think of a single production problem that can't be replicated with a unit test. If you're seeing a problem in production, you need to fix it. How do you fix it? You write a test that replicates the problem and then fix the code, which fixes the test.
First you claimed before that unit tests will catch your subtle concurrency bugs before they happen, and that's just not often the case. They are subtle, might involve many systems and weird edge cases and often don't get caught BEFORE they happen. Of course anyone can write a test to replicate the problem after seeing it in production and spending hours/days debugging it.
More importantly, "Write comprehensive tests" is technically a strategy to avoid any bug ever. You can tell C programmers not to segfault by writing comprehensive tests but that doesn't negate the point that the language makes it easy to write segfaults. "Write more tests" is not a rebuttal to saying C makes some classes of errors easy to write. Writing comprehensive tests often takes a lot of time, is often not prioritized by companies, and is especially hard with distributed systems, concurrency, mocks and complex applications. If we just said "git gud noob" in the face of error prone and difficult abstractions, we might as well all be using assembly.
I guarantee you I won't be using your product. Just something to consider.
Regarding my company, I respect your decision, but I hope that if our paths cross again, I might have the opportunity to change your mind through actions that demonstrate the value we provide to our customers.
Perhaps bringing it down a notch could help you connect with others’ perspectives as well.
"very easy to have bugs when working with channels and the way it handles errors etc"
If you've done some programming you'll find this to be true. You have to think a LOT harder if doing concurrency, and you generally have to do a lot more tests.
Go - WITHOUT that much testing, is often surprisingly error free compared to more dynamic languages just out of the box both language side and how a lot of development happens. Python by contrast, you can have errors in dependencies, in deployment environment (even if code is fine), based on platform differences (tz data on windows), and plenty of runtime messes.
Channels are not as default safe / simple after compile as a lot of other go.
Try programming without channels in go and this may become clearer.
I don't blame Go. It's an inherently difficult problem space. As a result, testing isn't a trivial job either. I wish it was.
Again with the HN downvotes, hilarious. People really hate the truth.
Think about it: If you're writing code and don't realize that it can deadlock under certain conditions, how are you going to realize that you need to test for whether it deadlocks under those conditions? If you're writing code that interpolates one kind of string into another and don't realize that you may have created an XSS vulnerability, are you suddenly going to gain that insight when you're writing the tests?
I'm not arguing that you magically predict everything that's going to happen. But, without those tests and the culture of writing tests and writing your code to be testable, you're screwed when you do have bugs in production.
That's nice, but it's far from "knowing your code works". When code works, it doesn't burn you in production.
How do you know your code is going to work in production before you got there? You wrote tests.
You know your code is going to work in production before you got there not because you wrote tests but because you thought about what it means for your code to work in production and then came up with a plan to generate the required confidence that it actually will work in production. And, for any nontrivial system, tests can only satisfy part of that plan.
The goal isn't to have well-tested code. The goal is to have code that you can easily be confident will work as intended. And testing is only good at establishing some kinds of that confidence. For the other kinds, such as confidence you're not going to launch a bunch of security disasters or concurrency landmines, you need to do something else: types, proofs, correctness by construction, model checking, and so on.
I wrote about this idea almost twenty years ago: https://blog.moertel.com/posts/2006-10-10-unit-testing-is-a-...
I elaborated in a later post: https://blog.moertel.com/posts/2012-04-15-test-like-youre-be...
That you seem to believe otherwise is probably why a lot of people are having trouble with your claim that that world doesn't need better concurrency abstractions, just more tests because "if you write comprehensive unit tests, it is not easy to have bugs in golang."
All the negativity (downvotes) has come from people who are trying to argue that writing tests doesn't solve the problem of bugs. The same people who don't write a lot of tests AND have a lot of bugs. ¯\_(ツ)_/¯
I write a lot of tests and I don't have bugs. I have decades of experience and millions of lines of code, with this simple fact. I know it is true, at least for me.
I don't know what else to bike shed here other than the constant downvoting by people who somehow don't believe my claim. The loss in karma doesn't bother me, I know I'm right on this and it appears as though the only people who disagree with me are the same people who don't write tests (and have a lot of bugs).
golang is a relatively simple language. It is why I like it so much. Occasionally there are somewhat difficult things to reason about, but if you write golang code that is easily testable (and this requires thought and planning), then my experience is that even the "harder" channel/goroutine code can always be tested in one way or another.
For the record, I write lots of tests, and don't have bugs. I even wrote a testing framework. Nobody is arguing that writing tests is dumb. The pushback is on your insistence that writing tests is all you need:
> All the negativity (downvotes) has come from people who are trying to argue that writing tests doesn't solve the problem of bugs.
Writing tests doesn't solve the problem of security bugs. Writing tests doesn't solve the problem of concurrency bugs. Writing tests to prove your code is bug free in those cases is expensive and error prone.
People who care about these things know to go beyond testing when testing isn't enough. That's why things like model checkers exist.
Nobody is arguing that tests are dumb. The argument is that if writing tests is all you're doing to get the bugs out of your code, you probably aren't very effective at preventing certain classes of problems.
For instance: Show me the tests you'd write to prove your software doesn't have XSS vulnerabilities.
I'd have tests around the code that renders 3rd party user input and integration tests for the display of the data.
I've built some of the most heavily trafficked websites on the planet (porn), with user input (comments) and have never had an XSS issue.
If you can't show me the tests you'd use to prove you don't have XSS problems, it's hard to believe that your tests are effective at preventing XSS problems.
> I've built some of the most heavily trafficked websites on the planet (porn), with user input (comments) and have never had an XSS issue.
Right, because the gold standard for proof in the security field is "we never had [read: noticed] an issue."
You're also being absurd. We started this talking about golang testing and it has somehow gone off the rails to me having to prove things to you about XSS? Come on, what is with the hostility? Is this how you treated people while working at Google?
I'm not sure how to meaningfully distinguish the two here. I'm saying it takes extra effort, not denying that it's my job. It's non-trivial, that's all I'm trying to say.
> How do you fix a production bug without breaking something else? You write tests so that you know your code works.
Of course, you're right. Sometimes writing the tests can be harder than writing the code, though.
which misses the point of the OP.
Technically you may be right, but it’s not a helpful viewpoint. What the world needs are abstractions that are easier to understand and program correctly, not assertions that everyone else is doing it wrong and just needs to be smarter/work harder.
If you want an analogy, I'm arguing that a condom helps prevent STD's and unwanted pregnancies. It isn't perfect, but at least it is better than not wearing a condom at all. Nobody loves wearing a condom, but we do it cause it is the right thing to do.
This is where other forms of software testing come into play. Such as integration tests.
You write tests to cover edge cases. If you miss one, you write more tests when you come across them. This isn't magic.
…before such point you have hard to find bugs in your software.
And that’s the crux of everyone’s argument against your “just write better code” fallacy ;)
> This isn't magic.
No it’s not. That’s why people disagree with your assessment that unit tests can catch every type of bug.
If unit tests really were that magical then we wouldn’t need for other methods of tests.
I mean, do you even know what a unit test is? It’s meant to be self-contained but the problem with multithreaded code is that you can start to introduce side effects that happen outside that functions scope.
I’ve got exactly that issue right now calling OS APIs from cgo. Some of those APIs (particularly with SDL) are very thread sensitive and you cannot unit test for that because the problem lies outside that functions scope. So the only way to test for that is with e2e or integration tests instead.
I also never said unit tests were the ONLY tests.
Nobody disagrees with your point about the importance of unit tests. But the way you present that point is more than a little naive (and that’s the polite way of putting it).
Trying to handwave and say that your code is too complex to be tested is very strange to me. Maybe take a step back and ask yourself why your code is too complicated to test. Maybe refactor it a bit to be less complicated or more easily testable.
> Here's a practical example: finding the first occurrence of a specific string among 1000 large files hosted online. Downloading all files at once would consume too much memory, processing them sequentially would be too slow, and traditional concurrency patterns do not preserve the order of files, making it challenging to find the first match.
But this example will process ALL items, it won't break when a batch of 5 finds something?
For proper pipeline termination the context has to be cancelled. So it should have been be Like:
func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel()
urls := rill.Generate(func(send func(string), sendErr func(error)) {
for i := 0; i < 1000 && ctx.Err() == nil; i++ {
send(fmt.Sprintf("https://example.com/file-%d.txt", i))
}
})
...
func Map[A, B any](in iter.Seq[Try[A]], n int, f func(A) (B, error)) iter.Seq[Try[B]] func Map[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan Try[B]
They may not be "necessary" for concurrency but given the various primitives available they're the clear choice in this situation. They do everything an iterator does, plus some stuff. The only disadvantage is that their operations are relatively expensive, but as, again, we are already in a concurrency context, that is already a consideration in play.
How would I implement timeout? In case a HTTP call takes too long?
[1] https://pkg.go.dev/git.sr.ht/~mariusor/ssm#example-Retry
[2] https://git.sr.ht/~mariusor/frankenjack/tree/master/item/sou...
I need a feedback on this. What kind of builtin context support would work for you? Do you need something like errgroup's ability to automatically cancel the context on first error?
Another part is to provide solutions for some real problems I had. Most notably batching, error handling (in multi stage pipelines) and order preservation. I thought that they are generic enough to be the part of general purpose library.
The problem in golang is the channel cleanup, particularly. is a mess. In occam they come and go simply by existing as variables in seq or par blocks. Occam is very static though, so the equivalent to goroutines are all allocated at build. (Occam pi attempted to resolve this iirc).
https://en.m.wikipedia.org/wiki/Occam_(programming_language)
Some of the patterns in this library are reminiscent of constructs occam has as basic constructs, such as the for each block, although the occam one must have the number of blocks known at build time.
The fact so many people in golang reach for mutexes constantly is a sign things are not all well there.
The usage of mutexes doesn't make channels "bad" for the same reason that usage of atomics doesn't make mutexes bad.
Even optimizing to where if an app instance new it ran the inert on update for a specific unique index by storing that in a hash map and only running updates from there on out to increase the count of occurrences of that event was enough to find significant performance gains as well.
It's generally good for languages to provide relatively low-level functionality and let libraries be able to build on top of it, because as the programming language development world has now learned many times over, the hardest code to change is the code in the language and its standard library. It isn't the job of the language itself to provide every possible useful iteration on the base primitives it provides.
After quick scanning, I'd highlight these 2 differences: - Rill is type safe and uses generics - Rill does not try to hide channels, it just provides primitives to transform them
Uber CFF does code generation, and has more of a focus on readability and complex dependency chains: https://github.com/uber-go/cff
Judging by the praise, I'm probably in the minority, but as a code reviewer, I’d much rather see straightforward loops, channels, and Go's native constructs over something like Rill.
I've never paid my bills with Go, but `Map` and `ForEach` don't seem all that different than `for _, u := range Users` to me. Yes, the former is "functional" but only mildly.
But your point about the lack of contexts is definitely a deal breaker for me personally too.
sync.Map is a concurrency-safe hash map. Same name, totally different thing.
Pascal: https://www.freepascal.org/docs-html/ref/refse104.html#x224-... Python: https://docs.python.org/3/whatsnew/3.8.html#assignment-expre...
https://en.m.wikipedia.org/wiki/Assignment_(computer_science...
bugs originating from the similarity of == vs = has probably cost the industry millions over the last 3 decades.