[dead]
This level is rocket science. If you can't tell why it is right, you fail. Such a failure, which was just a singular missing synchronized block, is the _worst_ 3-6 month debugging horror I've ever faced. Singular data corruptions once a week on a system pushing millions and trillions of player interactions in that time frame.
We first designed with many smart people just being adverse and trying to break it. Then one guy implemented, and 5-6 really talented java devs reviewed entirely destructively, and then all of us started to work with hardware to write testing setups to break the thing. If there was doubt, it was wrong.
We then put that queue, which sequentialized for a singular partition (aka user account) but parallelized across as many partitions as possible live and it just worked. It just worked.
We did similar work on a caching trie later on with the same group of people. But during these two projects I very much realized: This kind of work just isn't feasible with the majority of developers. Out of hundreds of devs, I know 4-5 who can think this way.
Thus, most code should be structured by lower-level frameworks in a way such that it is not concurrent on data. Once you're concurrent on singular pieces of data, the complexity explodes so much. Just don't be concurrent, unless it's trivial concurrency.
- Performance on game servers
- Thoughts on concurrency in general, including whether you'd use the same primitives to achieve what you did back then
- Areas where you felt the tooling was lacking, or if the language could have constructs that would make that problem easier
That's it on a surface level! Your experience seems really interesting, and I'd love to read more and learn from it.
The idea that making things immutable somehow fixes concurrency issues always made me chuckle.
I remember reading and watching Rich Hickey talking about Clojure's persistent objects and thinking: Okay, that's great- another thread can't change the data that my thread has because I'll just be using the old copy and they'll have a new, different copy. But now my two threads are working with different versions of reality... that's STILL a logic bug in many cases.
That's not to say it doesn't help at all, but it's EXTREMELY far from "share xor mutate" solving all concurrency issues/complexity. Sometimes data needs to be synchronized between different actors. There's no avoiding that. Sometimes devs don't notice it because they use a SQL database as the centralized synchronizer, but the complexity is still there once you start seeing the effect of your DB's transaction level (e.g., repeatable_read vs read_committed, etc).
Same thing with goto and pointers. Goto kills structured programming and pointers kill memory safety. We're doing fine without both.
Use transactions when you want to synchronise between threads. If your language doesn't have transactions, it probably can't because it already handed out shared mutation, and now it's too late to put the genie in the bottle.
> This, we realized, is just part and parcel of an optimistic TM system that does in-place writes.
[1] https://joeduffyblog.com/2010/01/03/a-brief-retrospective-on...
In our present context, most mainstream languages have already handed out shared mutation. To my eye, this is the main reason so many languages have issues with writing asynch/parallel/distributed programs. It's also why Rust has an easier time of it, they didn't just hand out shared mutation. And also why Erlang has the best time of it, they built the language around no shared mutation.