Hacker News Reader

My deep interest is multithreaded code. For a software engineer working on business software, I'm not sure if they should be spending too much time debugging multithreaded bugs because they are operating at the wrong level of abstraction from my perspective for business operations.

I'm looking for an approach to writing concurrent code with parallelism that is elegant and easy to understand and hard to introduce bugs. This requires alternative programming approaches and in my perspective, alternative notations.

One such design uses monotonic state machines which can only move in one direction. I've designed a syntax and written a parser and very toy runtime for the notation.

https://github.com/samsquire/ideas5#56-stateful-circle-progr...

https://github.com/samsquire/ideas4#558-state-machine-formul...

The idea is inspired by LMAX Disruptor and queuing systems.

And your approach can be built into a system that does multi-threading away from Python, thereby achieving parallelism without requiring that Python supports it as well.

That's basically what all machine learning code written in Python does. It calls out to libraries that can themselves parallelize, use the GPU, etc. And then gets the answer back. You get parallelism without any special Python support.

Just to add a bit of my opinion after reading your comment in the context of this thread and not to the merit of your idea. You are precisely the type of person I'd keep very very far away from multithreading in any business software project and also why I advocate the GIL to stay. If you want to do that, go solo in your own time, or try apply for a research position in some giant tech Co.

>My point of view is that anyone who wants to write multithreaded code, shouldn't be trusted to. Making it easier for people to justify this kind of footgun is a problem.

Out of curiosity, have you done any Rust programming and used Rayon?

It's hard to convey how easy and impactful multi-threading can be if properly enclosed in a safe abstraction.

Python is dynamic AF and Rust's whole shtick is compile-time safety. Python was built from the ground up to be dynamic and "easy", Rust was meticulously designed to be strict and use types to enforce constraints.

It's hard to convey how difficult it would be to retrofit python to be able to truly "enclose multithreading in a safe abstraction".

I have only read about and played a tiny bit with Rust. But as I noted at https://news.ycombinator.com/item?id=36342081, I see it as fundamentally different than the way people want to add multithreading to Python. People want to lock code in Python. But Rust locks data with its compile-time checked ownership model.

See https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.h... for more.

This is a good point. With tsan and helgrind multithreading becomes manageable in C/C++, nothing like that exists (or will exist) for Python.

>My point of view is that anyone who wants to write multithreaded code, shouldn't be trusted to. Making it easier for people to justify this kind of footgun is a problem.

It's 2023 already. 1988 called.

Did you know that Coverity actively REMOVED checks for concurrency bugs?

It turns out that when the programmer doesn't understand what the tool says, managers believe the programmer and throw out the tool. Coverity was finding itself in situations where they were finding real bugs, and being punished for it by losing the sale. So they removed the checks for those bugs.

I'll revisit my opinion of multithreaded code when things like that stop happening. In the meantime there are models of how to run code on multiple threads that work well enough with different primitives. See Erlang, Go, and Rust for three of them. Also, if you squint sideways, microservices. (Though most people set up microservices in a way that makes debugging problematic. Topic for another day.)

> Did you know that Coverity actively REMOVED checks for concurrency bugs?

Source?

https://cseweb.ucsd.edu/~dstefan/cse227-spring20/papers/bess...

Specifically, on the last page of that:

> As an example, for many years we gave up on checkers that flagged concurrency errors; while finding such errors was not too difficult, explaining them to many users was.

(And thanks; I was also wondering about that)

This is very witty and all, but what does it mean?

I think he was subtly Rick Rolling you.

https://en.wikipedia.org/wiki/Never_Gonna_Give_You_Up

On 12 March 1988, "Never Gonna Give You Up" reached number one in the American Billboard Hot 100 chart after having been played by resident DJ, Larry Levan, at the Paradise Garage in 1987. The single topped the charts in 25 countries worldwide.

Actually it just meant that this is a tired old argument when C/C++ programmers were new to multithreaded code.

Now we have languages and language facilities (consider Rust, Haskell, and others) to make it much safer. Same with green threads and what Go and now Java does.

And yet, the actual proposals for how to do multithreading in Python are essentially the same as the old C/C++ approaches. And since that hasn't changed, there is no reason to expect better results.