Even in Go, concurrency is still not easy

Have another goroutine receive and output the "found" results, rather than doing it in the first loop. Now you have a proper pipeline.

The big win with goroutines is that they're green threads. They can block, and you can have lots of them. With regular threads, too many will use up resources. With "async", anything that blocks or just takes too long locks up the whole system. Go has the best of both worlds, which is very useful for the case of a server holding open connections to a huge number of human-paced clients.

I figured this out in Go about 30 minutes after I realized I needed to limit concurrency, and I'm a dumbass, basically.

So, I'm not sure that "still not easy" is accurate, especially given that I have a lot of trouble conceptualizing async and await, as used in C#. In fact, I don't think I've ever successfully used that paradigm in C#. I've been told that C# async and await are "easy" and this article says Go concurrency is "still not easy" and I've had the exact opposite experience in both cases.

In fact I really don't understand why async and await is even a thing anymore, given the paradigm that Go uses and the example it sets. It is almost supernatural in its ease, for me.

If you're using .NET in your projects, you can start taking advantage of such tools _today_. I would like for such tools and testing techniques to become more and more common place in the industry as concurrent and distributed systems are _hard_ and we should use all the help we can get.

Any modern language allows you to create threads (semantically equivalent to goroutines) and pass messages between them. Most of them even have high-level concurrency support such as parallel map, structured concurrency, reactive extensions, type-safe generic collections...

We could say that Golang makes concurrency using thread semantics scale to a large number of threads. When this paradigm is appropriate, then indeed Go is nice. The select statement is nice. But this is not always the right approach.

Yes, it is true that go needs slightly fewer characters to launch a naked thread than in C# or Java. In that sense, it is easier. Anyone who considers that a major selling point is a walking race condition.

As a tongue in cheek analogy, what if we consider my new language G, in which G is an alias for goto. G is going to finally make control flow easy. I've done away with complicated control flow constructs such as for or while - users wouldn't understand them anyway. Users are encouraged to use the G keyword liberally. After all, it's only 1 character. Indeed, by that metric, it's even simpler than go's go keyword. What could go wrong?

Simplified:

    var wg sync.WaitGroup
    wg.Add(jobs)
    ch := make(chan workItem, jobs)
    for i := 0; i < jobs; i++ {
        go func() {
            defer wg.Done()
            for item := range ch {
                processItem(item)
            }
        }()
    }
    for _, item := range items {
        ch <- item
    }
    close(ch)
    wg.Wait()

Languages that _actually_ make concurrency easy, such as Haskell, Scala, OCAML or distributed concurreny easy (Erlang/Elixir) are way ahead of Go but are not as hyped and are not marketing themselves as well.

It is the same story as with garbage collection. Garbage collection prevents some kinds of errors and in general makes it way more easy to deal with dynamic memory allocations. However, garbage collections does not mean you don't have to think about allocation patters and object lifetime for example.

Go does make concurrency much easier by the tools it provides. Most importantly are goroutines, which are very lightweight, so within reasonable limits, you don't have to be concerned about the number of goroutines you spawen - but as the example shows, you shouldn't try to spawn more goroutines as there are file handles available to your process, if every goroutine allocates a file handle. Not only are goroutines very lightweight with small growable stacks, but as they are part of the language specification, the compiler can generate code which helps with scheduling, in most cases you have efficient cooperative sheduling, so thread-based preemption is less frequent.

On top of that, channels provide a very easy to use abstraction for communication between goroutines. There are a lot of use cases, where goroutines and channels give you very easy and safe concurrency. It does not save you from understanding concurrency issues and especially deadlocks. Which in a general sense is impossible to achieve, because to prove that a program is deadlock-free would require to prove that all goroutines return and that is equivalent to the halting problem, which is unsolveable.

It should also not be underestimated, that due to goroutines and channels being part of the language spec, these are very commonly used features. Concurrency is present in most Go programs. As a consequence, when one reads a lot of Go code, there will be plenty examples of their usage. And any library needs to be thread-safe as the likelyhood is very high that it does get called from goroutines.

Like if I have 3 definitions in a row that set variables to the result of methods that don’t share memory, that seems pretty obviously parallelizable. Why isn’t anyone doing these sorts of optimizations?

I would have thought it would have been solved by now, but I was wrong.

See the playground: https://play.golang.org/p/pXJaGQ0efe8

Source Code is available on GitHub: https://github.com/go-training/training/blob/2ddb95d08c654a6...

I can’t recommend it high enough. It makes working with highly concurrent go so much easier and is a blessing for anybody with prior experience in Erlang or Akka.

For example, Go has no generic support for atomic primitives, arrays, slices, or maps. It has sync/atomic, but it's not generic. I always reach for Uber's atomic library, which has typesafe atomic wrappers such as atomic.Bool. They're so common, you'd think a CSP-aware language would provide keywords declare things atomic:

  var b atomic[bool]
  if cas(b, false, true) {
    ...
  }

But as you use more and more of these primitives, your code (or at least my code) becomes more and more obscured by the layers of error-handling, cancel-detection, retrying, and so on. I really wish some of this was built into the language, especially cancellation. For example, channels don't support contexts. So if you have a loop like this:

  for evt := range taskCh {
    handle(evt)
  }

  for {
    select {
      case evt := <-taskCh:
        handle(evt)
      case <-ctx.Done():
        return ctx.Err()
    }
  }

Another option is to spawn a goroutine whose only job is to abort the channel:

  go func() {
    <-ctx.Done()
    close(ch)
  }()
  for evt := range taskCh {
    handle(evt)
  }
  return ctx.Err()
  

As a real-world example, here [1] is some code I've been working on lately. A controller starts N workers that need to process a task queue. If the controller is stopped, all the workers need to stop. So I use the pattern above. But I can't use a for loop! That's because I use a task queue abstraction that needs to support features that Go channels don't.

I can't use a for loop or select block to block on the task queue, because only channels support that. So that's another problem with Go: Often, you can use raw channels as your data processing primitive if things are simple enough, but in practice, channels are best for coordination, and you have to build your own primitives — and yet, by building those primitives, you lose language expressiveness.

To address the concrete example, I'd love to do this instead:

  // Range over task queue (custom object with blocking
  // semantics), automatically cancel if ctx is cancelled
  for task := range taskQueue with ctx {
    handle(task)
  }

What we see with golang is a phenomenon where people just parrot what some well known figures said at some point in time, without proper evidence or any basis (and in some cases, even when the evidence is counter to those claims).

I'm looking forward to Java's green thread implementation (project Loom), as it has proper ways to manage cancellation and deadlines and hierarchies, all of which have are quite verbose and error prone, or not supported at all in golang.