Hacker News Reader

Comment by cratermoon | Hacker News Reader

Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs

View on HN

I refuse to believe that summing up weights for each of 300 planes is so hard that it can't be made correct by a team of moderately competent developers operating under sane management. There might be some complexity, but it's 2023 and we have enough knowledge and tools to solve that kind of stuff routinely and reliably. Somebody fucked up big time and should be fired never to work in software again.

I can give you a simple example on how this can happen, from a real production system in a different regulated environment: all the SQL SELECT statements are WITH (NOLOCK) on MS SQL server, resulting in non-committed (dirty) reads. Under very light load it works fine, on higher loads it reads some dirty data. If you calculate the weight of the plane by adding the weight of the checked in luggage plus passengers read from a shared database, some differences will appear and they will increase with the load as more transactions will be in flight and possibly read or ignored.

I agree that it's hard to see from the outside how calculating weight figures for an airplane could be software load-dependent. But this all smacks of the point that Brooks makes in 'The Mythical Man Month' all those years ago. Two guys in a garage can make a program that does 'x', but to make a programming system product that does 'x' is more than an order of magnitude more work. The complexities introduced don't relate to the 'x', but the intricacies of large systems. Deciding that 'someone has fucked up and should never work again'? Sorry, competent and conscientious people regularly make such mistakes. Which is why we need tests, software review, engineering processes et al to catch those mistakes.

Boeing itself (unsure about this company though) was widely reported to have offshored their senior engineers a few years ago. eg:

https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-... (paywalled)

https://archive.is/vdg9S (paywall workaround)

Note that there were several articles about it around the time, the above one is just the first reasonable seeming one from a quick search.

This wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them.

"the bug was missed because it only presented when many aircraft at the same time were using the system"

There is certainly not enough information in that quote to say one way or another what the bug was. I've definitely seen concurrency bugs under load because data that wasn't supposed to be shared actually was, e.g. I posted this serious GitHub bug in a comment above, https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare....

Obviously sessions should be independent and not sharing data, but that's why it was a bug.

I believe any bug that only happens under load is a concurrency bug by definition. The shared resource is the thing under load. If it weren't shared, then the load from one computation would have no effect on another.

Should have used Rust.

A proper transactional database would suffice. You could do this safely and performantly on '00s hardware in PHP3 for crying out loud, serving thousands of planes per second.

Bad coding happen even in transactional databases, see my example in the other comment.

Note that I didn't say you couldn't write bad PHP3.

(This was a joke!)

Reply on news.ycombinator.com