Hacker News Reader

Comment by diptanu | Hacker News Reader

Can you please explain that more?

I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this.

The better question is: why has the Linux Foundation been silent about it?

If this were true -- "a decade of wasted cores", with losses of "13-24% for typical Linux workloads" (for a decade, as the title suggests) -- then companies like Netflix would have lost many millions due to our choice of Linux, and due to the Linux scheduling maintainers and community failing to identify such egregious problems. The industry as a whole -- including every device and server that runs Linux -- would have lost many BILLIONS. It would be one of the most costly failures in technology EVER.

And the Linux Foundation is silent about this? Seriously?

>"I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this."

Then why make a cryptic comment like you did? Cryptic comments kind of invite questions.

I've already answered it elsewhere -- more than once -- and I'm trying not to waste any more time on it. I think the Linux Foundation is best positioned to coordinate a detailed written response.

There's some extra research I did that I haven't shared yet, including, for example, when the bugs were introduced (still needs double checking):

  - Bug 1: Mar 2011, for Linux 2.6.38
  - Bug 2: Apr 2012, for Linux 3.4
  - Bug 3: Dec 2009, for Linux 2.6.32
  - Bug 4: Feb 2015, for Linux 3.19

This paper was published in early 2016, with the title "A decade of wasted cores".

Ah, but they were very intense years, so they counted more than one!

The tl;dr is that unless you had an HPC workload with a NUMA box, you couldn't observe this bad behavior.

Which means in reality, you could name approximately everyone that ran into this issue on a single list: top500.org.

Many HPC workloads run 1 process per core, and pin them. So they can't observe this bad behavior.

The Linpack benchmark is an example of an HPC code that should run one process per core and be pinned.

Is NUMA that rare? Back in 2007-2008 or so, my company bought some rack servers fitted with 48-core AMD Opteron (Magny-Cours), which weren't particularly expensive, and had a NUMA architecture.

We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result.

> We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result.

I'd bet, but not too much, that that was more due to a) postgres' internal locking implementation scaling horribly at that time b) zone_reclaim_mode leading to bad behaviour around IO.

It's possible, but we weren't testing 48 clients at that time, just our normal workload, which had much less parallelism than that. The person in charge of setting up the systems explained the performance issues as being due to NUMA.

You may find this a fascinating read. It's regarding how SQL Server has it's own threading scheduler to bypass the Windows scheduler. When ran on a machine with HyperThreading enabled it will only use 50% of the CPU, I guess they're going after CPU Cache utilisation.

It's really hard to find a decent article on the subject, here's two of the best I've found. https://www.mssqltips.com/sqlservertip/4403/understanding-sq... https://docs.microsoft.com/en-us/sql/relational-databases/th...

The paper has a clickbaity title, but the actual performance impact of the scheduler behaviors discussed therein is negligible on many workloads. YMMV.

Reply on news.ycombinator.com