Go and my realization about what I'll call the 'Promises' pattern | Hacker News Reader

Go and my realization about what I'll call the 'Promises' pattern

utcc.utoronto.ca

88 pointsingve95 comments

I don't understand why you would want an entire new execution scheduling mechanism just to print results in a specific order. The post dismisses the idea of including an index (a "sequence number" if you really want to be snooty about the idea of counting your requests), but "asynchronously waitable objects" is much, much more mechanism than that.

The irony of all of this is that the core thing the post wants to do, of posting results in order to reduce confusion for users, is also probably (I don't know the complete use case, but in general) wrong! Print when you get the results back! Don't block results behind a slow early request! It's a command line program, what's the virtue of queueing fast responses? The only point of the program is to print the output.

> The only point of the program is to print the output.

A consistent structure helps humans parse the output. I have host(1)'s output order in my mental memory.

Another way to think about this: if the only point of the program is to print the output, why not just serialize, or pipeline only one request past the current one?

Or you could just use a sync.WaitGroup and not print anything until all the results are in, which is what the proposed design does anyways in the case where an early request is the slowest to resolve.

Why not serialize / just print one? Parallel is faster.

Why not wait? Because it’s nice to see results as they come in.

Yes, there are pathological possibilities here, if an early request is slow. But this complaint sounds like a complaint about the author’s goals, which are stated clearly enough. “The author has the wrong desires or goals”, your comment seems to say.

Is it faster? The design proposed deliberately withholds fast results in order to ensure they're printed in sequence. Anybody who has ever run "traceroute" knows what this UX is like. If I wasn't going full TUI, I would just print the results as they come in, on stderr (with an index number).

A sync.WaitGroup is as fast as the proposed behavior, for what it's worth, and doesn't have the ordering complexity.

Dude, just accept the premise of the article.

He gives the requirements: Execute in parallel, print in order as quickly as possible. That’s the premise. The point of the article is in how to achieve that, not whether you should.

“Print as soon as possible and ignore ordering” is not a challenge nor an interesting topic to write an article about.

Yeah, this is funny. It's like presenting someone with a thought experiment and them saying "yeah but that would never happen".

From the article:

> (The specific context is that I've got a little Go program to do IP to name DNS lookups (it's in Go for reasons), and on the one hand it would be handy to do several DNS lookups in parallel because sometimes they take a while, but on the other hand I want to print the results in command line order because otherwise it gets confusing.)

If I was using this program, I too would appreciate ordered results and an early return where possible without breaking the ordering.

Edit: and other similar programs might use a limited pool of workers, in which case the user would want to see ordered results as they came in and they would be expected to complete roughly in the same order they were started in.

To be clear: you can still easily get this result without anything nearly as complex as promises. But as someone who has written this program many times in many languages, I'm not so sure you're right that this behavior is desirable. People go out of their way to install CLI network tools that don't do this.

I agree that it seems a bit pointless unless you're running a huge batch and that I don't think reading from channels in a for loop should be referred to as promises, but I've written CLI tools where running batches of jobs in parallel and printing the results in order is desirable. I don't think it's a generally undesirable thing.

Edit: not quite parallel: I mean with a limited pool of workers, or a partial ordering, or something else that means there's a reason to return results early.

I think it's safe to assume that it's a toy problem. A simple example. Picking apart the example is a poor counterargument. It might be difficult to imagine a specific trivial example, but expressing a convincing example concretely can itself be a major diversion. Unless you're saying you can guarantee that your alternative is always better in all conceivable cases?

Similarly, the existence of others wanting different behavior is also a poor counterargument. The fact that someone wants a Bag doesn't mean that someone else is wrong for wanting an Ordered Set.

As you can see upthread, it's not my only argument, just the one responders locked in on. But the problem here is also not lack of an ordered set!

You can print the results when you get them back but also print them in order too. You’ll need a little bit of ANSI escape magic to do it but it’s not a complicated problem to solve. An example of this done well is docker-compose.

Personally, I’d have written this code using a sequence number as well. Albeit I’d have attached the sequence number to a context. To me, this sound like the kind of problem that contexts do well.

The ironic thing about the docker-compose UX, which I agree is probably the optimum here, is that it doesn't benefit from promises.

I’m not going to defend the authors desire for promises because it’s a pattern I’ve never liked in any language. But that’s just my personal preferences.

This depends on your definition of “print”. What if your stdout is not a TTY?

Also, if your stdout is a TTY, but your terminal is not ANSI X3.64? You might be able to look up the corresponding escape codes in terminfo based on $TERM (or simply use tput(1)), but the terminal might not even be able to perform those operations. What do you do then?

> This depends on your definition of “print”. What if your stdout is not a TTY?

If it’s not a tty then you don’t print the ansi escape sequences. Just like how every other well behaving command line utility works.

> Also, if your stdout is a TTY, but your terminal is not ANSI X3.64? You might be able to look up the corresponding escape codes in terminfo based on $TERM (or simply use tput(1)), but the terminal might not even be able to perform those operations. What do you do then?

This feels unnecessarily argumentative but I’ll humour you.

Don’t use $TERM as your only method of fingerprinting a terminal. $TERM is as useless at determining terminal capabilities as the User-Agent string is for determining a web browser’s capabilities.

That all said, the actual ANSI sequences needed to perform the UX I suggested has been standardised since the days of vt100. So even nearly every hardware terminal will support it.

And if you do happen to be running an actual teletype, well you’re in a museum so your use case is very different.

In terms of terminal emulators, most strive for xterm compatibility. And even those that don’t, will still support vt100 sequences. If they didn’t, then even things like readline (and thus Bash) wouldn’t work in their usual expected way.

Sure, the “correct” way to use ansi escape sequences is to query terminfo. But in reality you can get away with hardcoding vt100 sequences for most things. Plus you’ll reduce overhead for your utilities for something that is guaranteed to be supported anyway. In fact perversely, you might actually improve portability because terminfo itself isn’t universal (plenty of non-POSIX systems support ansi sequences but don’t ship terminfo).

> Sure, the “correct” way to use ansi escape sequences is to query terminfo. But in reality you can get away with hardcoding vt100 sequences for most things.

This might be the worst advice I have seen in a long while. All the world is not a VAX, as the tenth commandment for C programmers clearly states. Also, you are quite wrong: Many programs are run with TERM=dumb in order to function via terminals with limited interaction capabilities, like via a Web page, or as a subprocess of an editor, like Emacs eshell.

> This might be the worst advice I have seen in a long while. All the world is not a VAX, as the tenth commandment for C programmers clearly states.

Name me some common terminal emulators that don’t support vt100.

I’d genuinely be interested to know. But as someone who’s an author of a readline library as well as a relatively popular cross platform $SHELL, I’ve never ran into any issues assuming vt100 compatibility.

I can tell you now, such a terminal isn’t still in use.

And this is why I said: if you cannot guarantee vt100 compatibility then you’re likely working in a museum and thus have very different requirements.

> Also, you are quite wrong: Many programs are run with TERM=dumb in order to function via terminals with limited interaction capabilities, like via a Web page, or as a subprocess of an editor, like Emacs eshell.

I didn’t say $TERM isn’t used. I said it’s unreliable for determining terminal capabilities.

Take a look at all the different terminal emulators, 99% of them report themselves as xterm.

Now take a look at all the terminal multiplexers. Most of them report themselves as screen.

Hardware terminals don’t even use env vars, instead they use ANSI escape sequences to describe terminal capabilities.

This is why I compared the $TERM to the User-Agent string.

Perhaps I should also add that I’ve written my own terminal emulator too. And part of that process meant pouring over specifications for various different terminals (hardware and software). So I do have some idea what I’m talking about ;)

> I didn’t say $TERM isn’t used. I said it’s unreliable for determining terminal capabilities.

You did say that every terminal has vt100 capabilities. But TERM=dumb does not, which is why some environments do set it that way.

> Hardware terminals don’t even use env vars, instead they use ANSI escape sequences to describe terminal capabilities.

Firstly, for physical (i.e. non-emulated) terminals, the environment variable is set by getty, who gets it from either /etc/inittab, or now possibly a systemd service setting. So TERM is supposed to always be set. Secondly, how would you know what escape sequences to send to query the capabilities, if TERM is not set? You can’t send VT100 escape codes blindly, because, again, TERM might be “dumb”.

> You did say that every terminal has vt100 capabilities. But TERM=dumb does not, which is why some environments do set it that way.

Fair. However dumb terminals aren't a typical use case because a whole plethora of standard utilities don't work properly on them. eg from pagers to `top`. They're also very easy to detect and do not invalidate any of the other points I've been making.

Dumb terminals aren't exactly common though. They were already superseded by smart terminals as early as the 1970s. In fact the last time I used a dumb terminal professionally was way back in the 1990s for a Bullfrog time sharing system that was already ancient at that point.

Your eshell example is fair, however last time I checked that wasn't even hooked to a TTY and there's ansi-term for when eshell needs smart functionality. If that's still the case then the original mitigation of TTY detection works anyway. So I don't agree that even the eshell example proves your point.

Let's be completely honest here, how many dumb terminals with a TTY attached do you think people are actually using? It's likely a rounding error of 0. I'd expect tools like terminfo to cater for edge cases like these but an internal company tool? I don't think that's a fair ask of anyone's time.

> Firstly, for physical (i.e. non-emulated) terminals, the environment variable is set by getty, who gets it from either /etc/inittab, or now possibly a systemd service setting. So TERM is supposed to always be set.

On modern POSIX systems, yes. It's not on non-POSIX systems nor on many ancient mainframes.

On some non-POSIX systems, it's entirely down to the drivers for the terminal / authors of the terminal emulator to set that env var. Pretty much all of them these days will do but in my career I've actually used several terminals that didn't.

So it's not a guarantee you'll have $TERM, or even a string that's truly representative set in $TERM (I'll go into more below)

But I don't really want to get into the weeds about whether $TERM is set or not. My real issue is what people set it's value to, not whether it's set or not. And how there are conceptually better ways to check for terminal capabilities (in my opinion at least) that unfortunately fell by the wayside.

> Secondly, how would you know what escape sequences to send to query the capabilities, if TERM is not set? You can’t send VT100 escape codes blindly, because, again, TERM might be “dumb”.

Dumb terminals are an extreme edge case and there's already easy methods to detect them. So lets move on from that:

For detecting terminal capabilities on the older hardware terminals, there are escape sequences like the following:

    CSI Ps c  Send Device Attributes (Primary DA).

Personally, I much prefer this approach because it more specifically defines what a terminal can and cannot do. Like how Javascript can query browser capabilities instead of making assumptions based purely on a User-Agent string. But instead we have $TERM and nearly every terminal emulator then defaults that value to `xterm`, or some variation of. Or `screen` if you're a multiplexer.

Clearly that does naff all to help our situation when detecting terminal capabilities.

To be more specific: the $TERM annoyance hampers our abilities to expose new terminal features to applications. For example, if we want to check for things like image support, there's half a dozen different environmental variables that need to be checked to identify what terminal emulator is actually running and whether inlining images will work (ie not get intercepted and broken by a multiplexer).

This isn't just a theoretical example either, I've written tools for working with image data in the terminal and detecting what escape codes to send is a bloody nightmare.

----

Going back to the original reason for this discussion:

For the OPs software, hardcoding vt100 sequences and falling back dumb mode if stdout is not a TTY is a perfectly reasonable suggestion given their requirements.

If their tool explodes in popularity (unlikely because its an internal work tool) and they find people are needing more sophisticated terminal detection, then they can worry about that at that stage. But we both know that my suggestion is good enough to cover even all of the common edge cases. Plus, as it was a work utility anyway, they have complete control over those edge cases to begin with. Thus catering for non-vt100 compatible terminals running regular shells is clearly an overreach of their time.

I also really don't appreciate the tone of your comments. You come off as being antagonistic. Maybe I'm reading your comments too harshly. If I am then I apologize.

[edited: toning down my own comment because it was potentially a bit rude in places]

> It's not on non-POSIX systems

If we cannot assume POSIX, we cannot assume even the existence of environment variables.

> For the OPs software, hardcoding vt100 sequences and falling back dumb mode if stdout is not a TTY is a perfectly reasonable suggestion given their requirements.

Linking to ncurses and (if stdout is a TTY) using ncurses to look up the correct escape sequences is simple and guaranteed to be correct. If ncurses comes back and says “nope, this terminal does not support your requested operations”, in that case you have some leeway in what to do. (I would personally advocate to stopping there; if somebody has a capable terminal, but has not configured their system to advertise it and its capabilities correcly, that is their problem.) If you insist on doing extra work to deal with mis- or unconfigured systems, you should first certainly first handle TERM=dumb specially and not send any escape codes.

But hardcoding terminal escape codes? People worked hard to make libraries to abstract away all these things for programs. Programs should offload as much work as possible to (sufficiently reliable) higher-level libraries.

This feels like advocating for x86-64 assembly in, say, 2016, when the Intel/AMD hegemony was more or less absolute. It might feel like a reasonable assumption at the time, but the pendulum always swings back to multi-platform support being necessary.

> If we cannot assume POSIX, we cannot assume even the existence of environment variables.

That's basically just expanding my point, but yeah, you cannot guarantee $TERM.

For what it's worth though, all the major non-POSIX platforms do support the concept of environmental variables. Even Windows.

> Linking to ncurses and (if stdout is a TTY) using ncurses to look up the correct escape sequences is simple and guaranteed to be correct.

Simple if you want to write in a language that supports C FFIs. Some don't. Furthermore, even those C calls can be sub-optimal. For example C calls break thread safety in some languages.

This particular tool was written in Go. And frankly there isn't any good options:

- cgo creates build and runtime complexities

- the established Go-native ncurses-style libraries come with way way way more than what is needed here. You just dont want nor need that bloat. It's more to audit, it's more to go wrong.

- the Go-native terminfo libraries haven't been battle tested. So you're not really much better off with them than you are hardcoding the tiny number of escape sequences needed (more on that later) and assuming vt100 compatibility for TTYs.

If we were talking about writing a full TUI then I'd agree with you regarding relying on a battle tested ncurses (or equivalent) package. But that's not what this project is. This is literally just printing an ordered list.

> If you insist on doing extra work to deal with mis- or unconfigured systems, you should first certainly first handle TERM=dumb specially and not send any escape codes.

You're the one insisting on doing extra work. Not me :)

Plus I've already put the dumb terminal argument to bed.

> But hardcoding terminal escape codes?

Lets be specific here, I wasn't suggesting every application should hardcode every escape sequence.

I was suggesting that the escape sequences for this specific internal company tool being discussed here are simple enough that they can be hardcoded.

It's literally just 2 escape sequences needed. 3 if you want to get clever.

Having that hardcoded makes more sense than having a 3rd party library that you then need audit (again, this is an internal company tool), just to print 2 escape sequences.

Context matters.

> People worked hard to make libraries to abstract away all these things for programs.

I know. I'm the author of several such libraries ;)

> Programs should offload as much work as possible to (sufficiently reliable) higher-level libraries.

"sufficiently reliable" is doing a lot of heavy lifting there. See my comments about auditing imports for internal corporate tools.

> This feels like advocating for x86-64 assembly in, say, 2016, when the Intel/AMD hegemony was more or less absolute. It might feel like a reasonable assumption at the time, but the pendulum always swings back to multi-platform support being necessary.

That's an absurd comparison and I think even you know that. My entire point is about trying to avoid boiling the ocean -- you're the one advocating creating more unnecessary work, not me.

And if you're worried about vt100 compatibility disappearing tomorrow, actually the opposite is true. We are seeing more non-POSIX platforms implement vt100 compatibility. (to be honest, I'd love to see terminals progress beyond in-band data formats presented as buggy escape sequences, but it's just not going to happen).

---

I don't think we're going to see eye to eye on this. So maybe we just agree to disagree?

Hardcoding an escape sequence or two because you’re using an as-of-yet immature language environment with cumbersome and/or inadequate, or even buggy, libraries is… actually a good argument. Personally, I would have the program run tput(1) as a subprocess and capture (and save) the correct escape sequences from the pipe, for later re-use, but I could imagine how, in some special cases, that might not be practical.

> You're the one insisting on doing extra work.

I am insisting that programs handle terminals correctly. It was you who brought up the rather remote edge case of when TERM is not enough to identify a terminal; i.e. when the operations you want to perform are so advanced that they might differ even when TERM is the same, or be missing even if TERM is set correctly, or not even be present in terminfo. What I wrote was my suggestions to handle that case, which would indeed be extra work, but it was you who asked for that case to be handled.

> I wasn't suggesting every application should hardcode every escape sequence.

> I was suggesting that the escape sequences for this specific internal company tool being discussed here are simple enough

You originally wrote “in reality you can get away with hardcoding vt100 sequences for most things.” But now you want to argue that you were always talking only about this specific use case?

(Also, it seems that for this specific use case, only one escape sequence would be needed: “cursor_up”/“cuu1”.)

> That's an absurd comparison and I think even you know that.

I do not.

> And if you're worried about vt100 compatibility disappearing tomorrow, actually the opposite is true.

The same was said about the x86 instruction set, and it did look completely entrenched for many years. But then it wasn’t. And old ideas and concepts about what can be done with terminals, like MGR, may yet rise again. I see things like sixels only as symptoms of a pent-up demand for this. And people won’t want to go through ANSI X3.64 sequences to render their remote applications, so non-ANSI terminal protocols are likely to emerge again.

> Personally, I would have the program run tput(1) as a subprocess and capture (and save) the correct escape sequences from the pipe, for later re-use, but I could imagine how, in some special cases, that might not be practical.

That's not a bad suggestion generally speaking. I've seen a fair amount of shell scripts that do this too.

In this specific case, they're running DNS lookups in parallel to speed the tool up, so I think forking tput wouldn't be preferable.

But for tools that are more forgiving for latency (and specifically targeting POSIX too), that's definitely an option.

> You originally wrote “in reality you can get away with hardcoding vt100 sequences for most things.” But now you want to argue that you were always talking only about this specific use case?

Fair point.

> (Also, it seems that for this specific use case, only one escape sequence would be needed: “cursor_up”/“cuu1”.)

I was thinking cursor down too, but actually you're right, \n would do that job better for a multitude of reasons.

Edit: thinking about this more, there’s a reverse carriage return ASCII character so you don’t actually need any ANSI escape sequences at all.

> The same was said about the x86 instruction set, and it did look completely entrenched for many years. But then it wasn’t. And old ideas and concepts about what can be done with terminals, like MGR, may yet rise again. I see things like sixels only as symptoms of a pent-up demand for this. And people won’t want to go through ANSI X3.64 sequences to render their remote applications, so non-ANSI terminal protocols are likely to emerge again.

Professionally speaking:

I can't see that happening. More systems are implementing vt100 support, not less. And escape sequences are "easy" to implement and "good enough" that most people have a quick moan about them but then quickly move on to getting whatever job they want done, done.

Plus the decline of x86 was driven by $$$ (like most changes in IT). Apple wanting to own more of the pipeline. Datacentres wanting to reduce infrastructure and operational costs. And ARM was already as old and established as x86 so it wasn't like trying to create a new standard (how many other architectures have fallen by the wayside?). There isn't anything like that happening for terminals right now.

However, personally speaking:

I genuinely hope you're right. The status quo sucks. Just so long as whatever new that comes along is a complete redesign. In-band meta sequences are just horrible.

But I think realistically, those that don't like the status quo with terminals either use web technologies or Microsoft RPCs instead. And those that like the terminal have already learned to live with its many warts.

Just like UTF-8 unified ASCII with extended character sets, maybe ANSI X3.64 can be unified with something more, incorporating features from MGR and/or sixels.

> there’s a reverse carriage return ASCII character

Which one? And if it works, why does ”tput cuu1” not output it, instead of ^[M or ^[[A?

> Just like UTF-8 unified ASCII with extended character sets, maybe ANSI X3.64 can be unified with something more, incorporating features from MGR and/or sixels.

Not another character set please. Terminals are better off UTF8 and we are better off keeping control sequences out of character sets. It made sense in the 60s and 70s but makes zero sense these days. Plus Unicode has enough footguns as it is without introducing control sequences.

In my opinion the only good way to advance beyond escape sequences is to do away entirely with in-band control codes. Control codes should be on a separate channel. This is basically the only thing HTML gets right: separating content from formatting and control sequences.

Having that separation in the terminal would also allow for less hacks in pipe detection. To elaborate, `isatty()`, the C function that's used to detect if a fd is a pipe or TTY, basically just performs a terminal-specific I/O control operation to see if the file descriptor can handle terminal-specific controls, and if it can't, then assumes the fd is not a TTY. Its a complete hack in my opinion. A hack that works and has worked for a great many years. But a hack non-the-less.

Also having a separate data and control channel allows you to incorporate more meta data about the data channel. I actually use this trick in my $SHELL, Murex [0]. It's a typed shell (like Powershell but far less verbose and far more ergonomic to use) however it works perfectly fine with standard POSIX pipes because the type annotations are sent over a different channel. So you have the best of both worlds: raw byte streams for anything classical, but also rich type metadata for anything that understands Murex pipes. While introducing zero boilerplate code for anyone who uses the shell.

> > there’s a reverse carriage return ASCII character

Which one? And if it works, why does ”tput cuu1” not output it, instead of ^[M or ^[[A?

You're right. I was thinking of ^[M [1] (C1 control sequence) and getting confused with C0 sequences (ie ASCII characters). Wasn't helped by me misremembering that modern terminal emulators tend to treat multiple different ASCII characters as LF, such as VT (vertical tab) and FF (form feed) [2]

[0] https://murex.rocks

[1] https://github.com/lmorg/mxtty/blob/main/virtualterm/ansi_c1...

[2] https://github.com/lmorg/mxtty/blob/main/virtualterm/ansi_c0...

> Not another character set please.

Oh no, I was merely using character sets as an analogy. What I was envisoning was a new control scheme, backward compatible with ANSI X3.64, but not merely a bundle of extensions either.

> Control codes should be on a separate channel.

I’m not sure about that. Two separate channels results in synchronization problems (which is why the parallel port went away and why USB and all modern protocols are serial). However, I would not be adverse to having the channel transferring richer data than plain bytes.

> Oh no, I was merely using character sets as an analogy. What I was envisoning was a new control scheme, backward compatible with ANSI X3.64, but not merely a bundle of extensions either.

Interesting.

I'm working on something that's vaguely in that area at the moment: https://github.com/lmorg/mxtty

I'm currently working on getting - full vt100 compatibility (it's mostly there, just failing a couple of tests in vttest which I believe are related to cursor save and restore)

- most of xterm

- plus a few original escape codes to demonstrate it's concept

I plan to add Tektronix 4014 support

and a few other never seen before Terminal features that I haven't yet decided how best to implement.

I'm also going to leverage the fact that I already have a relatively mature $SHELL and tie in some specific support between the shell and the terminal emulator. If just to demonstrate the terminal's capabilities.

It's still pretty alpha but I'd definitely welcome any feedback on the project's mission.

> I’m not sure about that. Two separate channels results in synchronization problems (which is why the parallel port went away and why USB and all modern protocols are serial).

Synchronizing electrical signals is a very different problem to synchronizing data streams. There's plenty of protocols out there that solve the second problem already: TCP/IP (order of packets), video codecs (audio / video synchronization), and so on.

They key isn't to have the zero control sequences in the content stream. It's to reduce those control sequences to just being tags or markers, and shifting the meta data out to the data stream.

An example of this is HTML vs CSS. HTML is the content, CSS is the formatting. Granted they're not streamed, but it's a visual demonstration for how separate concerns can be divided but still connected.

This type of approach would also solve the existing synchronization issues with stdout and stderr. Though if we're already redesigning the terminal in backwards incompatible ways, I'd do away with stderr entirely and replace it dedicated control sequences and first class error handling.

None of this would ever happen but a guy can dream.... (though this is why I started the mxtty project: to see just how far I could push terminal capabilities in a backwards compatible way)

Would a single channel + sequence numbers actually be better, does quantity of channels matter? (I'm genuinely asking, I don't use Go much.)

Agree RE using sequence/index numbers. I would set up a reply array, send indices along with the requests and store each result in the corresponding index.

Keep track of the lowest received index and print when the next one comes in. Loop the same for the next index if it's already completed.

Or write it in JS where this would be trivial and compile to executable with Deno.

I had to solve a similar problem in Go once, partially to scratch an itch and partially to speed things up.

The S3 AWS SDK has support for doing chunked/parallel downloads if you can supply an io.WriterAt [1] implementation.

  type WriterAt interface {
   WriteAt(p []byte, off int64) (n int, err error)
  }

The AWS SDK comes with an implementation that has a Bytes() method to return the contents, so you can download faster but you can't stream the results.

I created a WriteAtReader that implements io.WriterAt and io.Reader. You can read up to the first byte that hasn't been written to the internal buffers. If there's nothing new to read but there's still pending writes, calls to Read will block and immediately return after the next WriteTo call. As bytes are read through Read, the WriterAtReader frees the memory associated with them.

It provides the benefits/immediate results you get with a streaming implementation and the performance of parallel download streams (at the cost of increased memory usage at the outset).

The first "chunk" can be a little slow, but because the second chunk is almost ready to be read by the time the first is done? zoom zoom.

[1] https://pkg.go.dev/io#WriterAt

So: A channel with buffer size 1, so long as it is only written to once and read from once, feels a lot like a Promise.

A variable you can only read once is a lot less useful than a variable you can read as many times as you want. Same goes for promises.

I don’t think I’ve seen a promise implementation that can’t handle multiple reads after the resolve for many years.

It would not take a lot of glue code to alter the contract to make that happen here.

that is a trivial memoizing wrapper on top of what GP described

It's been a while since I did much Go, but I think you can handle this cleanly by making one channel per task, and having an array of channels similar to the array of promises. Each channel takes that task's result, then closes. The caller waits on each channel in sequence.

That's exactly what the author ends up with, too. It's also how you might handle this in any other language that allows for some form of "join", either intended or usable as a join. Waiting on promises, thread joins, channels (as a signal mechanism), it's all the same pattern: Instantiate a bunch of asynchronous activities and queue up a "handler" corresponding to it, wait on the handlers in your desired order.

.NET 9 solves this with

  Task.WhenEach

where you had to use some boilerplate before .NET 9. Nick Chapsas has an interesting youtube video on this explaining what's the problem and what Microsoft did to solve this[1].

1: https://www.youtube.com/watch?v=WqXgl8EZzcs

Good catch. Thanks for posting the video.

Shortcut could be to create an array of sync.OnceValue, immediately invoking each element in a goroutine. Then iterate through the array and call each function again.

Isn't it even simpler in Go? No channels necessary, Each goroutine gets an index or slice, and writes the results to a shared array.

All you need is the goroutines to report when done via a WaitGroup.

That doesn't satisfy the "report the results as they become available" requirement.

The desired behavior is that printing the first result should only wait for the first result to be available, not for all the results to be available.

Would be trivial to show the results live though, especially for atomic values: https://go.dev/play/p/PRzzO_skWoJ

That's not "live"; that's just polling.

Modified the above to https://go.dev/play/p/DRXyvRHsuAH You get the first result in results[0] thanks to `atomic.Int32`.

    package main

    import (
     "fmt"
     "math/rand"
     "sync/atomic"
     "time"
    )

    func main() {
     args := []int{5, 2, 4, 1, 8}
     var indexGen atomic.Int32
     indexGen.Store(-1)
     results := make([]int, len(args))
     finished := make(chan bool)

     slowSquare := func(arg int, index int) {
      randomMilliseconds := rand.Intn(1000)
      blockDuration := time.Duration(randomMilliseconds) * time.Millisecond
      fmt.Printf("Squaring %d, Blocking for %d milliseconds...\n", arg, randomMilliseconds)
      <-time.After(blockDuration)
      idx := indexGen.Add(1)
      results[idx] = arg * arg
      fmt.Printf("Squared %d: results[%d]=%d\n", arg, idx, results[idx])
      finished <- true
     }

     prettyPrinter := func() {
      for range time.NewTicker(time.Second).C {
       fmt.Println("Results: ", results)
      }
     }
     go prettyPrinter()
     for idx, x := range args {
      go slowSquare(x, idx)
     }
     <-finished

     fmt.Println("First Result: ", results[0])
     fmt.Println("So-far Results: ", results)

    }

This will wait up to 1 second before showing a result when a result comes in. I'm pretty sure Chris doesn't want any waiting like that.

This will also print some results multiple times. I think Chris wants to print each result once.

It is utterly clear that the random wait is not intrinsic to the logic - it was only added for demonstration to simulate varying duration of requests.

You can simply comment out the println and just pick the first results[0]. Again, the repeated println for all results was only added for demonstrative clarity.

Frankly, the above satisfies all primary goals. The rest is just nitpicking - without a formal specification of the problem one can argue all day.

>It is utterly clear that the random wait is not intrinsic to the logic - it was only added for demonstration to simulate varying duration of requests.

I wasn't talking about the random wait at all. I was talking about

      for range time.NewTicker(time.Second).C {
       fmt.Println("Results: ", results)
      }

>You can simply comment out the println and just pick the first results[0].

When should we look at results[0]? There needs to be some notification that results[0] is ready to be looked at. Similarly with all the rest of the results.

>Frankly, the above satisfies all primary goals. The rest is just nitpicking - without a formal specification of the problem one can argue all day.

I guess we have to disagree. From my reading of the blog post it was pretty clear what Chris wanted, and the code you provided didn't meet that.

You can completely ignore that pretty print function as it not essential to the goal.

Revised example without pretty-print at https://go.dev/play/p/LkAT_g95BLO

As soon as you have completed `<-finished` in the main go-routine, it means `results[0]` has been populated and is ready for read.

If you want to wait till all results are available, then perform `<-finished`, `len(results)` times. (Or use sync.WaitGroup)

    package main

    import (
     "fmt"
     "math/rand"
     "sync/atomic"
     "time"
    )

    func main() {
     args := []int{5, 2, 4, 1, 8}
     var resultCount atomic.Int32
     resultCount.Store(-1)
     results := make([]int, len(args))
     finished := make(chan bool, len(args))

     slowSquare := func(arg int, fnNum int) {
      randomMilliseconds := rand.Intn(1000)
      blockDuration := time.Duration(randomMilliseconds) * time.Millisecond
      fmt.Printf("(#%d) Squaring %d, Blocking for %d milliseconds...\n", fnNum, arg, randomMilliseconds)
      <-time.After(blockDuration)
      resultIndex := resultCount.Add(1)
      results[resultIndex] = arg * arg
      fmt.Printf("(#%d) Squared %d: results[%d]=%d\n", fnNum, arg, resultIndex, results[resultIndex])
      finished <- true
     }

     for i, x := range args {
      go slowSquare(x, i)
     }
     fmt.Println("(main) Waiting for first finish")
     <-finished
     fmt.Println("(main) First Result: ", results[0])
    }

Chris wants to report the results in the same order as the inputs. So 25, 4, 16, 1, 64. Your code will report the results in an arbitrary order.

Ok, but that is even more simpler and shorter with plain `sync.WaitGroup`.

    package main

    import (
     "fmt"
     "math/rand"
     "sync"
     "time"
    )

    func main() {
     args := []int{5, 2, 4, 1, 8}
     results := make([]int, len(args))
     var wg sync.WaitGroup

     slowSquare := func(arg int, resultIndex int) {
      randomMilliseconds := rand.Intn(1000)
      blockDuration := time.Duration(randomMilliseconds) * time.Millisecond
      fmt.Printf("(#%d) Squaring %d, Blocking for %d milliseconds...\n", resultIndex, arg, randomMilliseconds)
      <-time.After(blockDuration)
      results[resultIndex] = arg * arg
      fmt.Printf("(#%d) Squared %d: results[%d]=%d\n", resultIndex, arg, resultIndex, results[resultIndex])
      wg.Done()
     }

     for i, x := range args {
      wg.Add(1)
      go slowSquare(x, i)
     }
     fmt.Println("(main) Waiting for all to finish")
     wg.Wait()
     fmt.Println("(main) Results: ", results)
    }


    (main) Waiting for all to finish
    (#4) Squaring 8, Blocking for 574 milliseconds...
    (#1) Squaring 2, Blocking for 998 milliseconds...
    (#2) Squaring 4, Blocking for 197 milliseconds...
    (#3) Squaring 1, Blocking for 542 milliseconds...
    (#0) Squaring 5, Blocking for 12 milliseconds...
    (#0) Squared 5: results[0]=25
    (#2) Squared 4: results[2]=16
    (#3) Squared 1: results[3]=1
    (#4) Squared 8: results[4]=64
    (#1) Squared 2: results[1]=4
    (main) Results: [25 4 16 1 64]

As soon as results[0] is ready he wants to print it. Then as soon as results[1] is read he wants to print it. Etc. Not waiting till the end to print everything, and not printing anything out of order.

If results[i] must be printed only AFTER print of results[0]...results[i-1], then you effectively need to wait for max of (time to compute results[0]...results[i]), since even if results[i] is computed earlier you can't print it out if results[0]..results[i-1] are not available. If result[0] takes the longest compute time, then you will definitely need to wait till the end.

Frankly, a simple sequential for loop seems to the simplest solution here :)

Anyways, I think this: https://go.dev/play/p/lFBpzUVVzUj satisfies all the constraints. Only look at the output with the "(main)" prefix, the other prints are for elucidation.

        package main

        import (
         "fmt"
         "math/rand"
         "time"
        )

        type Result struct {
         value    int
         computed bool
         consumed bool
        }

        func main() {
         args := []int{5, 2, 4, 1, 8}
         results := make([]Result, len(args))
         signal := make(chan bool)
         signalCount := 0

         slowSquare := func(arg int, index int) {
          randomMilliseconds := rand.Intn(1000)
          blockDuration := time.Duration(randomMilliseconds) * time.Millisecond
          <-time.After(blockDuration)
          square := arg * arg
          results[index] = Result{value: square, computed: true}
          fmt.Printf("(#%d)   Squared %d, index=%d, result=%2d, duration=%s, sending signal. \n", index, arg, index, square, blockDuration)
          signal <- true
         }

         for i, x := range args {
          go slowSquare(x, i)
         }

         for {
          if signalCount == len(results) {
           break
          }

          <-signal
          signalCount++

          for i := 0; i < len(results); i++ {
           if !results[i].computed {
            break
           }
           if !results[i].consumed {
            fmt.Printf("(main) Squared %d, index=%d, result=%2d\n", args[i], i, results[i].value)
            results[i].consumed = true
           }
          }
         }

        }

Example Output

        (#0)   Squared 5, index=0, result=25, duration=8ms, sending signal. 
        (main) Squared 5, index=0, result=25
        (#4)   Squared 8, index=4, result=64, duration=210ms, sending signal. 
        (#1)   Squared 2, index=1, result= 4, duration=777ms, sending signal. 
        (main) Squared 2, index=1, result= 4
        (#3)   Squared 1, index=3, result= 1, duration=867ms, sending signal. 
        (#2)   Squared 4, index=2, result=16, duration=924ms, sending signal. 
        (main) Squared 4, index=2, result=16
        (main) Squared 1, index=3, result= 1
        (main) Squared 8, index=4, result=64

That code has a race condition. One goroutine can modify an element of results at the same time a different goroutine is reading it. Running with `go run -race` detects the race.

This can be fixed with a mutex.

Now that we've got working code (with the mutex), we have to ask: have we proved Chris wrong? I don't think so. Chris never said it's impossible to implement this in Go. Chris just said that implementing it in Go is uglier than using an array of promises in some other language. And I think this code is uglier than an array of promises.

Although I think I disagree with Chris. He says that an array of channels is significantly uglier than an array of promises. I don't think so. I think an array of channels is fine (and is easier to understand than the signal, signalCount, computed, consumed, +mutex code).

    package main

    import (
     "fmt"
     "math/rand"
     "time"
    )


    func main() {
     args := []int{5, 2, 4, 1, 8}
     results := make([]chan int, len(args))

     slowSquare := func(arg int, index int) {
      randomMilliseconds := rand.Intn(1000)
      blockDuration := time.Duration(randomMilliseconds) * time.Millisecond
      <-time.After(blockDuration)
      square := arg * arg
      fmt.Printf("(#%d)   Squared %d, index=%d, result=%2d, duration=%s, sending signal. \n", index, arg, index, square, blockDuration)
      results[index] <- square
     }

     for i, x := range args {
      results[i] = make(chan int, 1)
      go slowSquare(x, i)
     }

     for i := 0; i < len(results); i++ {
      fmt.Printf("(main) Squared %d, index=%d, result=%2d\n", args[i], i, <-results[i])
     }
    }

Yes, I didn't quite get why he wanted a solution without multiple channels..it is clearly the best way. The only other option without multiple channels is to use one of the atomic types: `sync/Atomic.Int32` (to avoid the mutex) and then a single channel for signalling.

This is a data race between the write to `results[idx]` in the `slowSquare` goroutine and the read of `results` in the `prettyPrinter` goroutine.

> However, this time around I wanted asynchronous requests but to be able to report on completed work in order.

This seems similar to JavaScript's for await...of construct https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe....

A while ago I added a golang version here https://github.com/heavyairship/gopherawaitof.

Yeah this is a one-liner in JS

for await of (const name in ips.map(getName)) console.log(name)

Although you’d want more frills to deal with rejections

The way I would do it without seeing the implementation:

- Atomic to track print order (name it counter).

- Results slice stores items: {index, err, data}.

- Goroutines: do work, store result, then loop-check if counter matches index to print (with tiny sleep).

Edit: Another solution would be to put printing loop outside with a channel to signal work finished but I don't know how I would avoid using nested loops, one for the items in the list and one to check if the result is back.

That is unnecessarily complicated, and you don’t need a sleep or atomics.... Just make an array of result channels, and loop over it awaiting each result. Each result has a producer either working, or waiting for the channel reader to consume it.

Why use an array of channels when you can just have the goroutines write to a regular array?

You'd have to poll something to determine that there was data written to the array, which means you'd need some mechanism (mutex, cv, channel) per entry. If you just constantly spin on the array entries you're wasting time, and you're introducing race conditions (distinguishing partial writes from completed writes). So your array would end up being something like [(bool, str)], with the bool set to true at the end of the write to signal completion.

Now you're handling the race condition (probably) but you're still actively spinning, which is wasteful if the request takes very long. There's a reason people don't use spin locks much and use mutexes (which may spin, but it's been tuned and doesn't spin forever) instead.

Channels end up being effectively a (bool,str) pair but instead of spinning the consumer blocks waiting for the channel to produce a value. This is simpler, and probably much more efficient than spinning on a boolean variable.

NB: The author wants to print results, in order, as soon as possible, not delay until all results are available. If you can wait for all results before printing, then an array of results + waitgroup works fine. If you don't care about the order, then this can be much simpler.

That's what WaitGroup is for. You can only get the full sorted array when every goroutine is done. You don't need to spin...

But then you can't print them as they become available. Which was one of the two things the original author wanted: Print in order, but don't delay for all of them to be finished. He wants to make N requests, and print all N in the order made, not in the order of completion. And he wants to print them as early as possible, minimizing delay to first feedback (except in the worst case that the first request happens to be the last to complete, obviously).

Literally quoting the OP:

> I want asynchronous execution but to report the results in order, as each becomes available.

99% of the comments in this thread are people ignoring those two requirements. WaitGroup does let you get them in order (store in an array in order), but does not satisfy the second requirement. This isn't hard to understand.

That's fair. My issue with the requirement is that, if you fire up enough tasks, you only ever get to print a result or two before being blocked until the whole set is done anyway. But if you really want that behavior, my solution doesn't do that, agreed.

The "in-order" requirement makes this a weird problem to think you have. There's no situation where the channels are going to be heavier then the go-routines you're spawning to handle the processing: in fact the last line lamenting leaving blocked go-routines around is weird, because a blocked go-routine is still using less resources then one actually doing things - it's totally fine for them to block waiting to write to the channel because of the "in-order" requirement.

Your worst case scenario is you spawn N go-routines, and they complete 1 by 1 in reverse order from N so your entire dataset is waiting in memory for in-order completion - so no other concern here matters at all.

Had a similar problem in Elixir - spawn a pool of HTTP requests and notify immediately of the first one that completes with a status of "ok".

   Task.Supervisor.async_stream(
        My.TaskSupervisor,
        my_list_of_urls,
        &assess_url_exists/1,
        ordered: false,
        max_concurrency: 100,
        timeout: 30_000
      )
      |> Enum.find(fn
        {:ok, {:ok, _url}} -> true
        _ -> false
      end)

And - that's it. Do I want it ordered? just change the "ordered" flag. Tweaking concurrency? it's there.

But the really best part is that any pending computation at the point when the correct result is found, it is automatically cancelled. No waiting for its completion. No memory leaks, no file descriptors left open. It just goes away. Boom.

Just sayin'...

Just sayin'... you have solved a different problem to the one the author describes in the article.

Weird. I thought that "similar" meant "different, but kind of".

I love how most of the responses to this article here blame the author for wanting the wrong thing. One could implement a future struct that is awaitable and then the scaffolding to wait on a bunch of them. I have done things like this in other languages, it can be immensely useful. Similarly, when some wanted generics, they were bemoaned for wanting the wrong thing. Go is a wonderful language, even if it is not my cup of tea. But must its users be this defensive?

it is possible they fear being replaced with a few lines of erlang

I don't think this is defensiveness so much as that attempting to model promises in languages like Go will result in bad, nonidiomatic code, and that is a thing that happens semiregularly with Go in particular, as it has so many refugees from other languages with sharply different idioms.

Here, we're discussing implementing future structs just to print DNS results in order. I don't think the UX proposed in the post is good, but stipulate that it is; this is not a hard problem to solve in idiomatic Go.

"refugees from other languages with sharply different idioms"

Reminds me of the time I had to decipher some numerical analysis code written in Common Lisp but in the style of Occam...

> refugees from other languages with sharply different idioms

Mostly Java or JavaScript devs unable to make a dent in their main programming language space looking for a shortcut to jump to a better paying job by pretending to make a huge difference to the other programming language. I cringe every time a frontend dev who hasn't fully grasped React yet utters the words "I hear Rust is the future. I am going to port a JS package to it..." They have already started making inroads into Golang with half-implemented modules.

> this is not a hard problem to solve in idiomatic Go.

Genuinely asking, what would the solution look like in idiomatic Go?

Let's assume for a second that the premise of the article is valid and exactly the behavior we want - "asynchronous execution but to report the results in order, as each becomes available".

Send the results into a channel, with a separate goroutine that polls the channel, stores a buffer of results, and sorts and prints the latest results at a regular interval.

> sorts and prints the latest results at a regular interval

Slightly more complicated than that, because you can only sort and print elements once you have _all_ the elements that came before them. Once you add that layer you've got quite a lot more code (and potential for mistakes) than the promises version.

You could also have N channels, one for each argument, and use reflect.Select to receive the results as soon as available, waiting to print any result until all of its predecessors have come in.

You could also have a mutex-guarded block at the end of every worker goroutine to do the printing and a sync.WaitGroup to follow the workers in main.

No need for reflect.Select, just loop through the channels and wait on each. That'll do your "predecessors" part just by the sake that it's an in-order for loop.

You're right, and for some reason I thought that solution had been ruled out.

No. Channels sent over channel is the correct idiom here. No need for a sort.

Do you find OPs solution nonidiomatic? I do not. The sequence-number solution would probably have been fine too. How would an idiomatic solution look to you?

> But must its users be this defensive?

Go is to programing what Brutalism is to Architecture. It is striped bare of everything down to the most basic of forms. It is structure is laid bare, put on display and free of decoration.

If you come in and try to write Golang like other languages your going to be unhappy, your going to tell us how go sucks because NPM/PIP/Gems is better (a common lament). It's not defensive rolling up the news paper and wacking new devs to Golang and telling them "don't do it like its java/c/ruby/js/python do it like this..."

Embrace the less is more of Go and it makes more sense and gets much more pleasant.

I don't think that is a fair comparison for Brutalism. Programming's brutalism is more like Scheme: Simple, composable, sound and principled design. Golang is a "modern" McMansion: it tries to copy a modern style by unintelligently stripping away all ornamentation, but it doesn't understand the underlying theme of modernism and ultimately fails to achieve a coherent design.

This is the HN thread I didn't know I needed until I saw it.

I often wonder how many of Brutalism's ardent fans grew up in an environment dense with Brutalist structures. I'm not here to bash the movement (the finest examples are wonderful) but just a reminder that the median Brutalist building is often a bleak, bleak affair.

There's probably an analogy in there somewhere for programming too. Perhaps the best of Golang is wonderful but the median just doesn't compare.

ETA: Either way, thanks for the taking the discussion down this direction. Certainly made me chuckle.

Garbage collection screams: "look elsewhere if true brutalism is what you seek; maybe C or Pascal deserve that mantle more than a high level language like Go."

Reminds me of how I once wanted to implement a doubly linked list in idiomatic Rust but then people started telling me that I shouldn't want to write that.

How did idiomatic Rust make it hard to implement a doubly-linked list? I don't know Rust, so asking out of curiosity.

You can't very easily have two equivalently powerful pointers that point at the same node in Rust. It became somewhat of a meme in Rust community to the point that there is even a series called "Learn Rust With Entirely Too Many Linked Lists" [0].

[0] https://rust-unofficial.github.io/too-many-lists/

Rust's ownership and borrowing rules really want tree-shaped data, i.e. where there are no cycles. But a doubly-linked list is exactly the opposite of that, each pair of adjacent nodes create a cycle.

The memory ownership kinda expects all structures to be in a DAG (ie no cycles since ownership tracking doesn't understand that).

I postponed my properly getting into Rust for way too long after inadvertently choosing to implement a doubly-linked list as an early learning excercise to familiarize myself with Rust ownership and mostly ending up confused and frustrated. Anything not involving recursive cyclic data structures turned out a lot more straightforward.

I think there are two parts to it. First, hating Go has been very popular with a certain segment who feel it’s not complicated enough to be useful, so maybe people are conditioned to be defensive. Second, Go had a huge hype wave, so a lot of clumsy, hapless users got into it, thinking it would make them into good programmers. These people tend to write bad articles and give Go a bad name as a “noob language” that’s hard to do useful things in. That makes clever, elite Go users want to signal how they’re not like the hapless bozo Go users who can’t get anything right.

This phenomenon exists more broadly in computing and in general, but for some reason Go is a particularly concentrated microcosm of it.

Lordy, I clicked on this, fully expecting a discussion of a shape-pattern in the board game of Go, and somehow didn't have a thought in my mind for the real, completely different, meaning.

This reminds me of "trompe l'oeil foods"... dishes that appear to be one thing, but are another entirely. Or maybe optical illusions more generally.

Reply on news.ycombinator.com