Hacker News Reader

Comment by sitkack | Hacker News Reader

Moore’s law is far from over and multithreading is not the answer. Your opening sentence is spot on tho.

> Moore’s law is far from over and multithreading is not the answer.

Wut? We hit the power wall back in 2004. There was a little bit of optimization around the memory wall and ilp wall afterwards, but really, cores haven't gotten faster since.

It's been all about being able to cram more cores in since then, which implies at least multi-threading, but multi-processing is basically required to get the most out of a cpu these days.

Moore's law is "the observation that the number of transistors in an integrated circuit doubles about every two years". For a while clock speed was a proxy for that metric, but it's not the 'law' itself.

Yeah, but today number of cores is the rough proxy for that metric.

How do you operate in that world if "multithreading isn't the answer"?

Modern CPUs contain a lot more computing units than cores. For a while, hyperthreading was thought to be a useful way to make use of them. More recently, people have turned to advanced instruction sets like SSE and AVX.

Those things aren’t mutually exclusive. Also firstly, I suspect there’s more “low hanging fruit” in making more software make use of more cores. We’re increasingly getting better languages, tooling and libs for multi threading stuff, and it’s far more in the realm of your average developer than writing SIMD compatible code and making sure your code can pipeline properly.

Threads are of course appropriate to implement high-level concurrency and parallelism. But for fine-grained parallelism, they are unwieldy and have high overhead.

Spreading an algorithm across multiple threads makes it more difficult for an optimizing compiler to find opportunities for SIMD optimization.

Similarly to how modern languages make it easier to safely use threads, runtimes also make it easier to take advantage of SIMD optimizations. For example, recently a SIMD-optimized sorting algorithm was included in OpenJDK. Apart from that, SIMD is way less brittle at runtime than GPUs and other accelerators.

Care to elaborate?

No idea about the future of the Moore's law precisely. Yet recent research results show that there is still room for faster semiconductors, as discussed on HN [0].

[0]: https://news.ycombinator.com/item?id=38201624

I mean why isn’t multithreading the answer?

Multithreading has the following disadvantages:

* Overhead: the overhead to start and manage multiple threads is considerable in practice. Most multithreaded algorithms are in fact slower than optimized serial implementations when n_threads=1

* Communication: threads have to communicate with each other and synchronize access to shared resources. "Embarrassingly parallel" problems don't require synchronization, but many interesting problems are not of that kind.

* Amdahl's law: there is a point of diminishing returns on parallelizing an application since it quite likely contains parts that are not easily parallelized.

Well, yes. But assuming the question is how to reduce overall latency, what alternative is there besides algorithmic improvements or increasing clock frequency?

Recently, SIMD has been increasingly used to make use of the increased number of compute units in a CPU. Sure, it's not as easy and straightforward to use as multithreading (I never expected to say that about multithreading), but libraries and programming languages make more and more use of them.

Edit: latency is difficult. But accellerating CPUs and using GPUs for compute was never about latency. Most I/O bottlenecks are because CPUs have sped up so much and left the rest of the platform in the dust. Much of it is also due to fundamental limitations due to the speed of light. Increasing throughput is always easier that reducing latency.

I guess I was thinking more of multithreading as an umbrella term for parallelism but I see your point. If you squint, SIMD and multithreading are the same problem. What I mean is, if you have a computable function you want to evaluate on a finite amount of bits, the problem of how to physically layout and schedule the computation to minimize latency is very similar to maximizing throughput.

From a circuit complexity standpoint, you could evaluate a wide but shallow circuit to evaluate the function in a nanosecond, or a deep but narrow circuit that takes eons. Whether parts of that circuit are evaluated synchronously or asynchronously is immaterial, although synchronous computation does seem easier to reason about and the UX is more user-friendly from a programmability standpoint.

I agree with you the fundamental limitation is the speed of light if you are width-bounded (i.e., if physical space is the dominating constraint).

You forgot Wirth's law: software complexity compensates increase in speed of hardware.

Reply on news.ycombinator.com