A 20 Year Old Chipset Workaround Has Been Hurting Modern AMD Linux Systems | Hacker News Reader

A 20 Year Old Chipset Workaround Has Been Hurting Modern AMD Linux Systems

phoronix.com

185 pointsrbanffy32 comments

From the linked kernel thread:

  Kernel        : baseline      baseline + C2 disabled   baseline + patch

  Min (MB/s)    : 2215.06       33072.10 (+1393.05%)     33016.10 (+1390.52%)
  Max (MB/s)    : 32938.80      34399.10                 34774.50
  Median (MB/s) : 32191.80      33476.60                 33805.70
  AMean (MB/s)  : 22448.55      33649.27 (+49.89%)       33865.43 (+50.85%)
  AMean Stddev  : 17526.70      680.14                   880.72
  AMean CoefVar : 78.07%        2.02%                    2.60%

The origin of the bug appears to be here as well:

https://lore.kernel.org/lkml/YytrV2UMah8555+s@zn.tnic/

If you dig into opensource code it's not that unusual that you'll find code with a comment saying something like "for compatibility with [version of foo that is 20 years old by now and we stopped supporting 10 years ago]". There are probably a lot of CPU cycles lost to such code.

Here's my favorite (from GNU tar):

      /* Parse base-64 output produced only by tar test versions
         1.13.6 (1999-08-11) through 1.13.11 (1999-08-23).
         Support for this will be withdrawn in future releases.  */

<http://git.savannah.gnu.org/cgit/tar.git/tree/src/list.c?id=...>

One of my favourites, archive filenames containing a colon ':' character cause GNU tar to execute a command, historically rsh (remote shell), but thanks to the Linux alternatives system, is often aliased to ssh. This was added many years ago to support remote tape drives.

GNU tar might randomly make network connections, so sanitize your archive names, I guess..

I had to check that because it sounds too "execute arb shell scripts on dhcp lease" to still be there... but there it is... decades later and --force-local not the default?

It's almost worse that rsh isn't installed on most systems.

pledge(2) is the correct model.

No no, don't remove that. You might not use the test tar version, but you may still have tarballs produced by that version.

I don't know many examples but it's not uncommon in proprietary code either. Sometimes you have to do ugly workarounds for bugs and you don't know when the bugs will be fixed, so you can't trivially prevent it affecting future versions.

FreeBSD just recently [0,1] removed support for booting with the FreeBSD 1.x-4.x bootloader.

[0]: https://twitter.com/bsdimp/status/1570987021031866369

[1]: https://reviews.freebsd.org/D36550

This is a problem with all sufficiently advanced software, honestly. The NT and XNU kernels deal with this all the time, with plenty of exploits and UB simply caused by abusing zombie code. Windows didn't even have proper code for interfacing with NVME hardware at full speed until a few months ago (and very few developers have picked up that DirectStorage API). Similarly, there's a lot of idiosyncratic MacOS behavior that gets abused by projects like OpenCore.

The main difference is that when someone balks at the open source code, they can fix it. When someone finds a weird bottleneck on commercial software, our only option is to curse the name of whoever wrote it and move on.

It's a shame that all AMD users had to suffer for so long because of VIA's horrible chipsets. It was well known back in those days to avoid motherboards with VIA chipsets due to instability/BSODs/etc.

If I had a dollar for every Win9x blue screen due to Via chipsets...

I suspect there are hundreds to thousands of such workarounds in a lot of hardware, driver and software considering the billions of lines of code they could be in, much of which is proprietry. I have done quite a few workarounds that fixed odd problems but ultimately lead to worse performance once its fixed. I did my best at the time to try and limit when they applied but there is no easy way to detect the platform in a lot of cases to limit the application or to see into the future, especially when its working around a bug in the operating system and no fix has been released in any reasonable period of time. You can document them as well as you like we don't have a standard for how to deal with these types of things when it impacts common code. So I suspect there are lots of legacy workarounds causing performance degradation all over the place.

Great find and always excited to see performance improvements from discoveries in older code!

On the other side, I'm always sad / unsettled to see the comments on Phoronix articles like this, the tribalism, the accusations, etc.

The Phoronix forum is notorious for its trolls and agressive/dismissive behavior from posters.

A trollish and aggressive board would be 4chan, Something Awful, Fucked Company, or Twitter. Phoronix strikes seems to be a niche board with strong opinions; basically the same as any BBS I used to visit. Calling it tribalism and accusatory, based on the thread related to this topic, is pearl clutching.

The negative comments I saw in the thread are of the style below.

> That also tells us that AMD has not bothered to fix this for 20 years. On the other hand, I welcome their stepped-up efforts to improve the Linux experience for their end-users. There are still some areas where more participating could yield some benefits though, e.g. glibc or better upstream compiler support/tuning.

and

> It seems Linux competition is so slow AMD didn't even notice such suboptimal performance.

Unsettled? This is milquetoast criticism at best.

Link to the forum below in case I missed something. I've never visited the forum before and am not deeply involved in Linux, other than happily using it for my home computer.

https://www.phoronix.com/forums/forum/phoronix/latest-phoron...

Look a few other threads you will see. The ones about Wayland, gtk or KDE are notorious. But yeah it is nowhere near 4chan levels.

I wonder if there are more heuristic workarounds of this kind just lurking out there and hindering major performance wins.

On the face of it, I'd expect the major players (RedHat/MAGMA) to be profiling the heck out of common SW/HW combos, but I can imagine there are a lot of scenarios they don't cover, and I'm not sure who else would.

A systemwide sampling profiler would have quickly found this right?

So, could the impact only be substantial in very specific workloads? Because otherwise someone would have found it already?

This one is tricky because the wait op is accounted in C-state, not something obviously spinning in C0 that is the typical signature of a bottleneck.

Unrelated but I would love to figure out why my AMD system will completely lock up (including the mouse) under certain tasks with high CPU loads (in ubuntu). Rendering with Kdenlive is a great way to lock up my entire system for example. It is so irritating it makes me want to go to an intel system even though I have a very nice CPU and mobo.

I might have the same issue, although it's hard to say.

My current suspicion is it's somehow related to power saving, because I can't reproduce it after a fresh reboot. It only starts happening once I go through a few power states (e.g. by closing the laptop lid).

Since you mentioned a mobo I assume you're on a desktop. Do you use sleep/hibernate?

Ah yeah I remember there being an open issue for the kernel that had something to do with sleep states, but if I recall they supposedly patched that bug. I don't use sleep/hibernate. I'm not sure what my system settings are for those modes. Yes on desktop.

I have ’pcie_aspm=off’ in my boot params, although I don’t recall the exact reason why it was needed. Likely something stability related, have you tried it?

Although sounds like your problem could also be caused by I/O saturation?

Doesn't happen to me. I'm on ZorinOS (Ubuntu based) but no such issues.

My first gen ryzen 1700 had a silicon bug that was fixed by RMA and c6 sleep states bug can be fixed in software https://wiki.archlinux.org/title/Ryzen#Soft_lock_freezing

What is IBS in "certain workloads with IBS"?

I was going to guess the Instruction Based Sampling, but that appears to be only 15 years old.

It's a little ambiguous, but the word before is important -- "Sampling certain workloads with IBS" -- so that "with" clause is talking about the sampling method, not the workload style.

I think you are right. From the context the issue was found using IBS, not that the 20 year old code was doing IBS.

The issue is a dummy op used to work around an issue in old hardware gets counted as C-State residency, and "A large C-State residency value can prime the cpuidle governor to recommend a deeper C-State". So this dummy op used during switching C States makes the CPU look more idle than it is, causing more low power mode than there should be.

Irritable Board Syndrome?

less cat5 more fiber

> (...) leading to performance degradation on workloads that rapidly switch between busy and idle phases.

So umm e.g. long compilation jobs of complex software? What else?

Reply on news.ycombinator.com