destel

Just be careful with model personalization. Gemini once learned that I'm an engineer and now "translates" all conversations into engineering language, even if we talk about some philosophical stuff

I had similar bad results with gpt 4o/5 when they got these image generation capabilities.

Don’t know what’s the reason: my bad prompting or these models being tuned to work with photos/illustrations only

Some examples are mind blowing. It’s interesting if it can generate web/app designs

You're not late. Just yesterday I've configured new reply notifications via RSS.

Typically function "f" does two things. 1. Performs calculations (this can be parallelized). 2. Writes results somewhere (this must happen sequentially and in the correct order).

Here's the typical OrderedLoop usage example from the article:

  OrderedLoop(in, out, n, func(a A, canWrite <-chan struct{}) {
   // [Do processing here]
   
   // Everything above this line is executed concurrently,
   // everything below it is executed sequentially and in order
   <-canWrite
   
   // [Write results somewhere]
  })

Without the "canWrite" the entire body of the "f" will be executed either concurrently or sequentially. With it - we can have this dual behavior, similar to critical sections protected by mutexes.

It's worth mentioning that OrderedLoop is a low-level primitive. User-level functions like OrderedMap, OrderedFilter and others do not need the "canWrite" channel to be exposed.

Thanks. This replyTo pattern is very similar to promises in other languages.

UPD.

I've just made a small but important clarification to the article. While in many cases it's easier and even preferred to calculate all results, accumulate them somewhere, then sort; this article focuses on memory bound algorithms that support infinite streams and backpressure.

I've just made a small but important clarification to the article. While in many cases it's easier and even preferred to calculate all results, accumulate them somewhere, then sort; this article focuses on memory bound algorithms that support infinite streams and backpressure.

Wow, that’s some seriously sophisticated stuff - it’s not that often you see a heap used in typical production code (outside of libraries)!

Your first example definitely gives me merge-sort vibes - a really clean way to keep things ordered across multiple sources. The second and third scenarios are a bit beyond what I’ve tackled so far, but super interesting to read about.

This also reminded me of a WIP PR I drafted for rill (probably too niche, so I’m not sure I’ll ever merge it). It implements a channel buffer that behaves like a heap - basically a fixed-size priority queue where re-prioritization only happens for items that pile up due to backpressure. Maybe some of that code could be useful for your future use cases: https://github.com/destel/rill/pull/50

I haven’t used Java for about a decade, so I’m not very familiar with streams api.

Your snippet looks good and concise.

One thing I haven’t emphasized enough in the article is that all algorithms there are designed to work with potentially infinite streams

Hi everyone, I’m the author of the article. Happy to answer any questions or discuss concurrency patterns in Go. Curious how others tackle such problems.

Hi everyone. I built a tiny web game where the goal is to pick the lowest number that nobody else picked. It runs once per day — if someone picks the same number as you, you’re both eliminated.

Since the mechanics are simple, I focused on feel: vintage-poster styling, small animations, and a snappy number-picking interaction.

By day I’m a Go backend developer, so this pushed me into much more frontend/design than usual.

I’d love feedback on both the gameplay and UX. I’m also genuinely curious: how large can the smallest winning number get as the player count grows?

Stack: Nuxt, Supabase and Cloudflare workers

Author here. While the article uses database updates as the example, the same pattern works great for any high-frequency API calls or operations that could benefit from batching. The goal was to make the solution completely transparent - developers just call a normal-looking function, and the batching happens automatically under the hood. I've used several variations of this pattern in production, and it's been particularly effective at reducing costs and latency.

Author here. Recently started a personal dev blog - happy to answer any questions.

Thank you very much for the feedback. I thought about something similar some time ago. Buffer of size one, then measure the average time each item spends in the buffer. But for debugging your approach is simpler and more practical.

Thank you. I "stealed" the name from scala. They have the similar value+error type. Maybe in context it rill the better name could have been "Item"

I have to be honest - I haven't ever heard about it. Just checked and found it's very mature and popular, though it seems to have had no commits in the last 2 years.

After quick scanning, I'd highlight these 2 differences: - Rill is type safe and uses generics - Rill does not try to hide channels, it just provides primitives to transform them

I have some benchmarks in the project's wiki on Github. I can confirm your point: Rill's main bottleneck is channel operations, the library itself adds negligible overhead on top. Of course, for multi-stage pipelines the number of channel operations grows.

To summarize, I believe Rill's performance would be fine for any problem solvable with Go channels. I've used it for a wide variety of use cases - while I'm not sure I can call them high-throughput, I've had pipelines transferring hundreds of GBs of data with no performance issues.

Thank you for the feedback. I agree with your point.

The current solution is to make pipeline stages context-aware (which is often happens automatically) and cancel the context before returning. This is the responsibility of the user and can lead to problems you described.

I haven't yet found a better solution to this. On the other hand, exact same thing happens if your function spawns a goroutine and returns. That goroutine would run until done, unless it's context aware.

Regarding the "Delay" and "infiniteBuffer" functions - these are part of the work on adding support for feedback loops to Rill. I haven't yet found a reliable and user friendly way to do it, so this work is on hold for now.

Thank you! I took a quick look at mgmt, it's quite a large and complex project. I'd need to better understand your DAG-based concurrency patterns to say if Rill would be a good fit. Could you share some examples of the concurrent runners you're thinking about? This would help me understand if/how Rill might be useful for your case.

Thank you! There are two pieces of motivation here. The first one is removing boilerplate of spawning goroutines that read from one channel and write to another. Such code also needs wait/err group to properly close the output channel. I wanted to abstract this away as "channel transformation" with user controlled concurrency.

Another part is to provide solutions for some real problems I had. Most notably batching, error handling (in multi stage pipelines) and order preservation. I thought that they are generic enough to be the part of general purpose library.

The short answer would be: I kept writing a code that spawns goroutines, that read from a channel, do some processing and write results to another channel. Add some wait/err groups to this and we'll get a lot of boilerplate repeated all over the place. I viewed this as "channel transformations" and wanted to abstract it away. When generics came out it became technically possible.

Surprisingly, part of my inspiration came from scala (which I haven't touched since 2014). Back then Scala had transformable streams and the "Try" typethen.

I've just pushed few small changes to the readme that better explain context usage

I've also just pushed few small changes to the readme that clarify this things.

Thank you for the feedback. My design decision is of course a tradeoff. When multiple channels are exposed to the users (not encapsulated inside the lib), this forces them to use "select". And this is very error prone in my experience

Rill might look like it tries to be a replacement for iterators, but it's not the case. It's a concurrency library, that's why it's based on channels

It will. Otherwise the example wouldn't make sense. There's one important detail I haven't clarified enough in that part of the readme.

For proper pipeline termination the context has to be cancelled. So it should have been be Like:

func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel()

    urls := rill.Generate(func(send func(string), sendErr func(error)) {
        for i := 0; i < 1000 && ctx.Err() == nil; i++ {
                send(fmt.Sprintf("https://example.com/file-%d.txt", i))
        }
    })
    ...

One of the reasons I've ommited context cancellation in this and some other examples is because everything's happening inside the main function. I'll probably add cancellations to avoid confusion.

For now, the library is context-agnostic by design. For HTTP timeouts, you'd use Go's standard approaches: either set the HTTP client timeout or pass a context with timeout to each request. Please let me know more about your use case - I'll let you know if Rill isn't a good fit.

No, it's the opposite - the library treats channels as streams, processing items as they arrive without needing to know the total size in advance. This is why it can handle infinite streams and large datasets that don't fit in memory.

I am thinking on it. To be honest, the current design works fine for my use cases: simply put, the function that defines a pipeline should have context.WithCancel() and defer cancel() calls.

I need a feedback on this. What kind of builtin context support would work for you? Do you need something like errgroup's ability to automatically cancel the context on first error?

The lib is based on channels and inherits the channel behavior in terms of backpressure. Simply put if no-one reads on one side of the pipeline, it wouldn't be possible to write anything on the other side. Still, it's possible to add buffering at arbitrary points in the pipeline using the rill.Buffer function.