Parallel programming is hard. Right?
lbrandy.com
lbrandy.com
For people who don't do "This thing I'm an expert at and claim is not so hard" as often as the expert in question it really is that hard. Things get easier, and people better, with practice. Now, if this is something you plan on using a lot in your job / particular programming field then you shouldn't be afraid of multi-threaded programming but if this is going to be a small passing item that you work with rarely if ever you should probably use existing libraries / abstractions for this type of thing.
Lots of people think they understand multithreading because they understand the synchronization primitives. "I'll just share this synchronized HashMap among my threads and everything will be fine". Not so. Because your code makes implicit assumptions about that HashMap all the time, e.g. that you can check if an element is in the map and if not, add it (but another thread added it after you checked); that the map won't change and throw an exception while you iterate over it because my loop doesn't change it, etc.
I don't think granularity is the hardest thing about multithreaded programming, it's these implicit assumptions. In an imperative language we just tend to think about our code as if it is single threaded and nothing changes until you change it. Then you get rare and irreproducible bugs in your program.
If we focus on the first part, in my experience it is wrong: you can't rely on the data structure to do synchronization for you, because these structures only guarantee that internally they will work consistently, not that using them you can forget about synchronization and threading. Quite often you need to do your own synchronization on top of them, and it is easy to miss some cases. That's what my examples are about.
Well yeah. But the primitives aren't "synchronized hash maps", they're "transactions".
Haskell might be a turn off to some people because it doesn't allow the quick and dirty solution for people who know what they're doing, and I'm sure there are those who have been permanently turned off to "protecting the programmer from himself" by such limp-wristed languages as Java, but you've got to admit there's merit to a language where instead of spending 5 years going through the school of hard knocks with production bugs, the programmer is baffled from the get-go and has to go ask an expert right away, resulting in much more solid code from the first version.
Respectfully, I think the usual claim is that concurrent programming is hard, not that parallel programming is hard. So, while the article is interesting in its discussions of the performance trade-offs involved in implementing data-parallel algorithms, it's arguing against a strawman, I think.
How do you verify correctness? How do you test? A bug might only present one in ten billion loops, perhaps when you push the cores really hard. Or fail on the new processors with the slightly different barrier/cache/coherency semantics. Or fail in ways that send you down a million heisenbug gullies. Without any proper tools or diagnostics (yet).
A raw performance increase is not the most common good reason for concurrency, for multiple threads or processes in an application though it's a common bad reason. Achieving a decrease in latency is one common good reason to for concurrency on a single machine.
His basic argument at the top seems correct if you limit it to data parallelism. But if you read the rest of the article the author doesn't seem to follow the thread of his own argument.
At its core the notion that data parallelism is easier to debug than control parallelism is obvious. It is, and its often hardware supported. But unfortunately data parallelism is harder to write and often requires skill in math, so it's not used nearly so much.
Just look on google for: "13 dwarves" parallel or "13 dwarves" berkeley
If you're just trying to do some number crunching across sixteen cores, it might be pretty easy to figure out what the expected result is, since you can compare it with a single-core version of the algorithm in question to make sure your results are right. But what if you look at some real-world applications that are highly parallel?
For example, both World of Warcraft and Aion have suffered from multiple highly visible race conditions in their game mechanics. This might sound simple, but in practice players are able to actually identify those races and find ways to exploit them - sometimes it is possible to exploit them in such a way that the player is able to earn gold or other in-game rewards more rapidly than other players. Since you can often convert in-game rewards into real-world ones, I think you can see where that goes...
The most likely explanation in the above cases is that the people involved in writing the code just didn't have a good understanding of concurrency, but that can't be all there is to it, because if you watch closely, the same concurrency related issues crop up in games and related software all the time.
A significant issue here is that in complex systems, it can be impossible to distinguish between a race condition and expected behavior. If your game has a button labelled 'win match', and two players press the button, which one should win the match? Do you compare the timestamps on the TCP packets that say they pressed the button, ignoring the impact of network latency? Do you try and compare local time data from their machines, ignoring the impact of clock drift? What if, by some fluke of nature, they both send in a request with the exact same timestamp? In a simple case, you'd never see this, but when you have 11+ million customers playing on a daily/weekly basis, such things can happen.