Updating the Go Memory Model

Java may be non-deterministic, or exhibit unintended behaviour in response to a race condition. UB is strictly worse because it could do either of those things... or indeed anything else, segfaulting being the least of your concerns.

A few years back someone at Dropbox mentioned tricky prod-only races were a real problem for them sometimes, for instance: https://about.sourcegraph.com/go/go-reliability-and-durabili...

That's where I start dreaming about best-effort detection of races in production binaries or even just reducing the chances of torn reads. Years back there might have been other options, like explicit private vs. shared heaps with more controls on the shared heap, but there seems to be a more restricted set of choices now.

Reminds me a lot of the Mac forums where every poor feature in an Apple product is explained with a "let me help you understand why you're thinking about it wrong" kind of answer -- up until Apple fixes the bug / implements the feature to the applause of the same people who were talking down why it would ever need to get fixed.

And just the tone of the article here turns me off in the way it begins with a bunch of quotes and philosophy. I actually agree entirely with “Don't communicate by sharing memory. Share memory by communicating" but I figured that out myself in probably 2003 writing crappy perl scripts that utilized parallelism but I wanted to aggressively avoid all the concurrency pitfalls. Actor models and Erlang later made completely intuitive sense to me. The principle is entirely correct, but its fucking weird that a programming language needs to have a list of "proverbs".

That said, what I have learned from watching this space for a decade is that even formal proofs are no match for the complexity of this general problem. You have to get the definitions right, you have to prove the right statements, and so on. There is still plenty of room for error. And even a correct, verified proof is not really an insight as to why things work.

Experts were telling us in 2009 that OOTA had been resolved to everyone's satisfaction, only to discover that it wasn't, really. Maybe Promising Semantics is the final answer, but maybe not. We need to get to the point where everything is obvious and easy to explain, and we're just not even close to that yet.

Looking at citations of Promising Semantics in Google Scholar I found this Pomsets with Preconditions paper from OOPSLA 2020: https://dl.acm.org/doi/abs/10.1145/3428262. It contains this sentence:

"As confirmed by [Chakraborty and Vafeiadis 2018; Kang et al. 2018], the promising semantics [Kang et al. 2017] and related models [Chakraborty and Vafeiadis 2019; Jagadeesan et al. 2010; Manson et al. 2005] all allow OOTA behaviors of OOTA4."

I take that to mean the jury still seems to be out on excluding all OOTA problems. Maybe the canonical example has been taken care of, but not other variants. And again we don't really understand it.

[1] unless there is an additional indirection, I don't really know much about go.

https://blog.stalkr.net/2015/04/golang-data-races-to-break-m...

You'd probably hit other walls. I'm def. not trying to say it's straightforward!

https://www.jtolio.com/2016/03/go-channels-are-bad-and-you-s...

https://www.datadoghq.com/blog/go-performance-tales/

* Use `reflect.Value.Select`[1]. Having to reach for reflect feels ugly for such a common case, and the performance of the reflect-select is much lower than the native select.

* Create a single channel owned by the reader, pass it to each writer, and arrange for this channel to be closed when the final writer exits, through a waitgroup. There is an example under "Parallel digestion" in the Go Concurrency Patterns blog post[2]. Note the little details to get right. We must launch a separate goroutine to monitor the waitgroup / channel closure. If we accidentally do it in-line at the wrong level, everything will work fine if the total number of items written to `c` is less than `c`'s capacity, but will hang once a worker becomes blocked on `c`. Additionally, the waitgroup is threaded directly into the writers, which may be more difficult if those are implemented in some other generic package.

* Wrap the above pattern up into a `merge` function, such as the one under "Fan-in, Fan-out" in the Go Concurrency Patterns post[2]. The lack of generics means we will have to copy-paste this function everywhere we want to use it. Additionally, this launches a goroutine for every channel being watched, which strikes people as "expensive" for such a simple operation.

* We can construct a function that takes two channels and launches a goroutine that selects between the two and writes to a merged output channel. By constructing a tree of these we can merge an arbitrary number of channels. This is really just an optimization of the above.

None of these options are particularly intuitive. Too often I've instead seen developers create a single channel owned by the reader and either:

* Assume it is never closed and the reader doesn't terminate until the application does

* Rely on some external mechanism to know when to stop reading. If the reader can stop reading without confirming that the writers have stopped writing, this can lead to the writers becoming blocked on sending into this channel, which may prevent them from performing necessary cleanup actions (signaling `.Done()` on a waitgroup, for instance) that cause hangs in other areas.

* Thread a cancellation ctx through every reader and writer. This ensures that nothing hangs, but can result in messages that are sitting in the the channels being dropped. If other areas of code have an assumption like, "every accepted request will receive a response", this can break that.

In addition, many developers have a gut instinct to add some amount of buffering to their channels, which usually results in these backpressure / channel issues being papered over during low-load unit tests, only to rear their head during higher load integration tests or in production, when the debugging story is much more difficult.

[1]: https://pkg.go.dev/reflect#Select

[2]: https://blog.golang.org/pipelines

I was specifically trying to explain why I say that Go's channels are not intuitive, because they require studying and memorizing these other arbitrary complications.

I'm also curious about what docs you're referring to, exactly. Here's what I found when looking for golang channel docs:

https://golang.org/ref/spec#Channel_types

If the documentation's described behaviour, along with code patterns to accommodate that behaviour, are intuitive to you after reading this, then you have a very different perspective on the world than I do.

I also found these: https://tour.golang.org/concurrency/2 https://golang.org/doc/effective_go#channels

Unless I've missed it in my reading, I don't see any of these clearly stating that the single-return form of channel receive will fabricate zero values when misused, or describing how you need to replace a closed channel with a nil when selecting on multiple channels to avoid spinning the CPU when it's been closed.

I agree that this stuff is learnable. I have learned it, and so have you. I agree that there are learning resources out there that help with learning the nuances of using Go's channels well.

Hopefully this can help you feel less shocked the next time someone says that Go's channels are not intuitive. If you disagree, can you explain more about how Go's channel management choices are more intuitive than the alternatives to you?

[Edit: I found the documentation on producing a zero value when reading from a closed channel here: https://golang.org/ref/spec#Receive_operator]

(The only language I know where this is nominally "solved" is Java if you confine yourself to its checked exceptions, but nobody considers that solution "satisfactory", and indeed can be taken as a rough-and-ready proof that there may not even be a "static" solution to this problem that is usable. I'm beginning to think of it as "Errors don't compose, because they contain every possible pathology that would prevent values from composing." At this point, based on the consistent failures trying to create composable error solutions despite substantial effort poured into it across multiple languages, I am going to operate under the assumption there is no such solution until someone produces one.)

This is just one kind of pipeline. If this is all you have done, you may have the impression that concurrency (in Go) is easy.

BTW, this is worth reading:

Understanding Real-World Concurrency Bugs in Go https://songlh.github.io/paper/go-study.pdf

It really feels like the design was driven by this one example only.

Many people wish C++ memory_order_relaxed had no effect on synchronization and could be optimized like a normal access. It's not. https://internals.rust-lang.org/t/unordered-as-a-solution-to...

Barrier-based abstractions could handle that, if they support more than one barrier. For loads, this would allow efficient dependent load reordering, by essentially enforcing the ordering only where needed for concurrency reasons (this mostly helps speculating loads before the address is confirmed, and not needing to snoop for invalidations of the cache line that contains the speculated address/killing the load), and similarly taking pressure of the store buffer by being less strict about the order in which it commits to L1D$.

RISC-V's propose WMM has such weak default ordering, but due to using fences, it's overly strict to the point where it performs worse on heavy concurrent code that's littered with atomics, compared to a TSO version (of the same softcore) that "just" prefetches exclusive access for writes. Even when turning RMW into relaxed semantics, so it's just due to the overly-strict load fence that effectively trashes all shared-state L1D cachelines.

I still find data races (sometimes crashes) in the wild on a regular basis. For example, RSS Guard accesses shared data unsynchronized when syncing settings from a server, so performing two types of syncs at once on 2 different threads will crash when they reallocate the same vector. Qt Creator intermittently crashes (or at least used to) in some tricky CMake handling code with multithreaded copy-on-write string lists. And I see apps now and then that perform unsynchronized writes to memory concurrently read in another thread, and it usually doesn't misbehave.