Hacker News Reader

How is this nonsense or anything to do with "language runtimes with disabilities"? An OS running on a single core processor cannot be parallel but it may be concurrent: it can never physically do two things at the same time, but it might be able to logically interleave different tasks.

Parallelism is a physical thing, concurrency is a logical thing.

You have just repeated the nonsense I was talking about.

The claim you repeat is meaningless. A program is either parallel / concurrent or not. The situation you describe (when there's a single processor core) isn't parallel or concurrent. In some sense, it emulates concurrent / parallel execution because it imitates the unpredictable ordering of code execution, which sure has its uses... but the whole point of dealing with this unpredictable ordering is that we actually want parallelism / concurrency. The emulation on its own is worthless.

No it's not? How do you characterize running two programs on a single core without calling it concurrent but not parallel? There is a clear distinction between logical multitasking and physical multitasking that I think is useful to taxonomize.

bruh your OS scheduler wants a word with you

a program (as written) is either concurrent or not, a program (as executed) is either parallel or not

a program which is not written as concurrent can never be executed as parallel

a program which is written as concurrent can be executed as parallel, or not

> The situation you describe (when there's a single processor core) isn't parallel or concurrent.

a program running on a single core can never be parallel, but it can be concurrent

concurrent is a logical property, parallel is a physical property

"a program which is not written as concurrent can never be executed as parallel"

Unless of course you're running multiple independent instances of it, each with different parameters, which I gather is a pretty common way of running long running CPU- intensive operations on large data sets. Presumably there's often some final separate step that may be needed to combine the results once they're all finished, which if run manually obviates any concurrency concerns at the software level.

in which case that program is being executed N times, so none of this applies

("a program [as executed]" is a single instance (process) of a binary)

Yes, concurrency on a single core CPU is simply not possible. That's why multitasking OS's didn't exist until multicore CPUs became a thing.

If only these "Gophers" knew anything about computing history!

So before 2001 your OS didn't have a scheduler and was incapable of handling more than one process at the same time? The user interface was simply hanging when you told your CPU to do something?

That was the point I was attempting to get across. Single core computers multitask fine, which proves that concurrency is not the same thing as parallelism.

Even in modern times, we have something like the first generation raspberry pi zero, it has a single ARM core, and it multitasks fine.

Your "single core" computer contains a lot of other concurrently operating hardware like storage and network cards.

Parallelism is a physical thing, concurrency is a logical thing.

Fundamentally the difficulty is all about synchronization. People can try to split hairs and say there are two terms for two different things but ultimately it doesn't matter because the underlying problem is the same.

Single core concurrency doesn't have to deal with hardware memory synchronization.

What point are you trying to make now? You just said:

Parallelism is a physical thing, concurrency is a logical thing.

So by your own definition, wouldn't multi-core concurrency be parallelism?

Yes? Parallelism is inherently concurrent but concurrency is not inherently parallel. But single core concurrency still exists as a form of non-parallel concurrency.

The point I'm trying to make is down to brass tacks, single core concurrency is a lot simpler than multi core concurrency, because you don't need any hardware cooperation.

Before you were saying they are two different things, now you're saying they are the same when you have multiple cores and not the same when you have a single core. If they're the same when you have multiple cores, isn't there just single and multi-core concurrency by your own definitions? If so, why are you making a distinction of parallelism and concurrency?

Parallelism is concurrent.

Concurrency isn’t necessarily parallel.

There are still plenty of single core systems out there running concurrent software, even today, for example embedded systems.

> Single core concurrency doesn't have to deal with hardware memory synchronization.

Yes it does. If you have two threads which (for whatever reason) are sharing memory and they can be run in an arbitrary order, you have to deal with memory synchronization. If it's a single core machine, you still need to synchronize your memory access. Mutexes and semaphores predate systems with multiple processors.

Yes, but those are not hardware memory synchronization mechanisms. You do not have to deal with memory fencing to coordinate reads/writes at the CPU level, because there is only one core that can read memory at a time.

>> don't know who invented this nonsense distinction ...

It not nonsense. In C or C++ lots of code can be made parallel using OpenMP and inserting some #pragma statements above for loops. This does not work for things like running a UI in one thread and some other work in another thread, perhaps displaying results as they are found. These are quite different types of parallelism.

I think it's nonsense to invent new terminology for this tiny minor difference in how you use threads and whether you wait on results or you don't wait.

but it's just not a tiny minor difference. The problems you end up dealing with are entirely different.

The terminology used might be terrible since they're originally synonyms. Differentiating between the two is very useful though.

how are the problems different? In all cases you want the maximum performance. The only decision here is how you schedule your tasks. Ideally entirely in parallel and you only schedule them to wait if you can't find a way to make them 100% parallel

If there is no dependency between your results, you do not need synchronization. Parallelizing it just means putting them into a seperate stream of execution. That can be a thread or an entirely different computer and the two different execution streams need to synchronization or communication between each other.

That is an entirely different set of problems than having to deal with a computation that has close dependencies and now you need synchronization and communication to progress the compuation. You don't only have to think about how to synchronize your computation, you also have to think about how to distribute your data to begin with to minimize the need for synchronization. An entire set of problems that just don't exist in the former case.

To pick up a word you used: In one case you have many independent tasks and you want to run them as quickly as possible. In the other case you have a single task and you think about how to split that up to make it faster.

Parallelism and concurrency are different models. It is not a needless distinction.

With just concurrency, you can have a system, e.g. an event processing system, where each event happens, is processed to completion, and then the next event is processed. Events can be of different types and can have different handlers. Events can be arrive (or be delivered) in different orders. The context-switching points are known (in this case, the beginning and end of event processing). As such, any computation between context switching points appears to happen atomically (i.e. no partial update, no internal reordering).

Parallelism is a different model.

There are no well-defined, limited set of interaction points between computations. A computation could literally be interrupted at any machine-level instruction and another computation start running. Suddenly, the number of possible interactions is huge--almost beyond comprehension. Locks, transactions, or other synchronization mechanisms are necessary in order to create larger atomic regions (and defend against race conditions). Worse, with weak memory models, which most modern multi-core hardware have, means that interleaving alone is not enough to explain the possible interactions between threads. It's possible with weak memory models that writes from another computation appear in different orders to different threads. Weak memory models make avoiding race conditions absolutely paramount.

This is word salad. Sorry.

What are you modeling?

The way you use "concurrency" it's indistinguishable from code without any signs of parallelism. From your "definition", concurrency is just any code. Such definitions are called "trivial" if you don't want to offend the author, and "worthless" if you are honest.

Your attempt at defining "parallelism" is even worse... You start by calling it undefined, and then proceed throwing poorly connected verbs and nouns...

Let me make it simple:

Parallel or concurrent code is code that doesn't require time interval between instructions (this is in contrast to Von Neumann model, where it's necessary to have a non-zero time span between instructions).

Here. That's it. No "but hardware", no "there are no well-defined" etc.

a program which is concurrent expresses logically independent paths of execution, which can be executed one after the other, or at the same time, or any mix thereof -- it is a logical property of the program as expressed, and not necessarily exploited, or observable, when the program is run

a program which is parallel actively demonstrates multiple paths of execution during runtime -- it is a physical property of the program as executed

tl;dr: every parallel program is concurrent, not every concurrent program is parallel

Your comment is subtractive from the discussion and its tone and attitude is not OK. If you came here to make no attempt to understand what people have written, even when they are explaining words, defining terms, and instead attack them and shout past them, we're better off if you didn't.

I wouldn't normally put such a fine point on it, but in this thread you've managed to make a lot of noise and calls lots of things "nonsense". If you're after emotional responses and heated discussion, that's tantamount to trolling, and you should stop.

> In all cases you want the maximum performance.

Not necessarily true. In many cases of concurrent programming, you don’t actually care about performance, you care about multiple different things happening seemingly at the same time (sure faster is nice, but not stuttering or freezing is more important). For example on a single core multitasking system (either in the past or on an embedded system today), you want multiple different tasks all running together, but this is achieved by giving them all a little chunk of execution one after the other — time slicing. The tasks are running concurrently, they all make progress together, but they share the same single thread of execution, they don’t happen in parallel. Yet one doesn’t block another, which is the important thing here.

I mean may e you “want” maximum performance, but that’s not what’s most important. Not having one task block another is what’s important.

Parallel programming is pretty much always about making things faster, by running then in parallel at the same time. The main benefit of doing that is that they both complete faster.

I'm sorry but you are betraying a myopic view here. Concurrency as a computer science concept has existed at least since the existence of co-routines which have been around for 60+ years. Parallelism as a concept has also existed for about as long. I find it fascinating that you take issue with "concurrency" when that one is the older concept in the engineering zeitgeist. If anything, parallelism is the odd one because it only became sounding average engineers started thinking about when consumer CPUs started becoming multi threaded.

Concurrency didn’t necessarily require threads. Concurrency is not a difference in how you use threads, its logical computer science concept. Parallelism is a physical property and parallelism is almost always concerned with doing things a faster by doing them in parallel, concurrency can be done for other purposes (eg to not have long running tasks block work). All parallel programming is concurrent but not all concurrent programming is parallel.

The distinction matters because the focus is different: they’re different topics. With concurrent programming you deal with synchronisation and topics like lock-freedom and wait-freedom. In parallel programming the main focus is about work distribution and scheduling to process it faster.

OpenMP versus std::thread/pthread/etc isn't this distinction. Both are thread-based parallelism; that OpenMP implementations do so cleverly via a threadpool approach is secondary. Any loop you can write using OpenMP you could dispatch manually via an explicit thread-based approach, and long-running parallel task you'd normally implement via std::async/std::future/std::thread could be dispatched via OpenMP. There are good conventions about what kinds of operations are better expressed via one or the other mechanism, but they are effectively equivalent in capability.

What does this have to do with anything?

I mean, great... you discovered a somewhat useful library: OpenMP... so what? How does this factoid affect the validity of the definition of code parallelism?

I'm not sure I can identify when it started, but these were already the concepts commonly in use when I did my CS undergraduate work in the early 90s. I.e. it was in textbooks and course titles as established jargon.

Concurrency was the kind of thing worried about in OS design or Unix programming styles whether on a time-sharing system or some small scale multi-processing system. Coordination of heterogeneous sequential programs on some shared resources.

Parallelism was the topic of high-performance computing with combined use of many hardware resources to accelerate a single algorithm.

Of course these are simplifying abstractions, and real systems can get into the murky gray area that is both concurrent and parallel.

Rob Pike has a discussion on the distinctions between parallelism and concurrency. Concurrency is closely related to co-routines which are a distinct invention from threads/processes which are more related to parallelism.

Just because you didn't know about the concept doesn't mean the distinction is nonsense. I think they are similar but not the same, exactly for the reasons laid out in the comment you are replying too. Just because you hate the languages that support concurrent programming doesn't mean concurrent programming is meaningless. Any language using async/await (basically all of them these days) support concurrent programming, including languages such as Swift and C# which are nothing like Python or JavaScript.

> In such environments programmers are offered a mechanism that has many downsides of parallel / concurrent programming (eg. unpredictable order of execution) without the benefits of parallel / concurrent programming (ie. nothing actually happens at the same time, or only sleep is possible at the same time etc.)

From the developer's perspective it's a massive upside to not have to manage low-level details and just define how the event loop will call their code.

Any modern web browser has plenty of parallel execution behind the scenes, but the developer (and user) will just see concurrency which is much simpler to reason about. The order of execution doesn't matter if the things being executed aren't dependent. What matters more is that there's only one thread to think about. If they are dependent they shouldn't have been parallelized in the first place, so they're not.

What does this have to do with low or whatever other level?