Hacker News Reader

Comment by mping | Hacker News Reader

On the comments: "Intel CPUs were tested with Meltdown/Spectre patches, that's probably the discrepancy you're seeing."

So every other reviewer is not using those patches (isn't Windows installing them automatically?), and the performance impact is that huge that it completely negates every performance advantage Intel had for now almost a decade?

I'm not 100% sure about every other reviewer, but many times tests will be done with a base windows 10 install (as to avoid the need to look at comparisons between patches).

Regarding performance impact, short answer is - yes. Basically, it removes most of the advantage of the cache for Intel.

> many times tests will be done with a base windows 10 install [without any updates/patches]

So, even without Spectre & friends, these benchmarks are essentially meaningless?

It's a difficult problem. It's not practical to re-test all of the old CPUs or GPUs every time you review a new part. Either you test all components against a baseline OS and driver set, or you test components with whatever OS and driver releases are contemporaneous. Neither case is ideal.

Reviewers have generally settled on the former option; OS and driver updates tend to increase performance, so testing with contemporaneous software might artificially inflate the apparent performance deltas in favour of newer components. Nobody wants to buy a new processor thinking that it's 40% faster than their old one, only to discover that most of the difference was just OS optimisations. You'd rather have your readers be pleasantly surprised than bitterly disappointed, so it makes sense to choose a benchmarking methodology that errs on the side of favouring older parts.

Spectre is a weird exception to business as usual, because we saw a huge performance decrease in a single update that affects one particular optimisation in one particular processor generation. In this case, it probably makes sense for reviewers to re-run all of their old benchmarks and set a new baseline.

> isn't Windows installing them automatically?

I thought the OS update was only part of the patch and that OEMs also needed to update their BIOS, which means users have to manually update their BIOS.

Microcode is indeed distributed through Windows Update as well. And packages on other OSes. And applied at boot time.

It takes a long time to run a comprehensive test suite against a single processor. Reviewers often have to ship the hardware back to the company afterward. The result is that they very frequently re-use the old benchmarks in newer tests.

New software doesn't affect existing benchmarks.

Which is wrong. Video Games don't actually make very many kernel system calls. Meltdown / Spectre patches have virtually no effect on video games.

The majority of DirectX stuff in the innermost-loops automatically DMAs to the video card without even needing a kernel call. Video games care a lot about performance and did their best to never touch kernel-space anyway.

Compilers, Web Servers, Databases, etc. etc. were affected very strongly. Compilers open files. Web Servers open Sockets. Databases communicate with mutexes / semaphores. All of these are OS-level features and require a kernel-mode switch. Meltdown means that every kernel-mode switch clears out the TLB-cache.

So each time a compiler opens a file (MMap or otherwise), it basically loses its entire TLB cache. Every. Single. System call. That's why its a big deal for servers, but not a big deal for video games.

I wonder if moving to a different file format for compilation might minimize the impact of this? For example, what if source code were stored as a single SQLite file? What impact would that have?

If anything, it'd be slower. SQLite locks the database file (which would be another OS-level system call).

Simple files are more straightforward. You do one MMap and then there wouldn't be any other system calls made. I'd bet that compilers are "slow" because they have to make many, many, many mmaps to do anything. Each #include turns into another MMap, which Meltdown mitigations cause a full TLB flush each time.

For reading many blobs of approx 10KB each, it is often faster to read them out of an SQLite database than from separate files on disk. See https://sqlite.org/fasterthanfs.html for details and to download code that you can use to test this yourself on your system.

I don't know if SQLite would be faster or slower in the context suggested here. But I'd like to suggest that perhaps it is worth running the experiment before reaching a conclusion.

Mmap uses page faults instead of syscalls to read more data. Both these mechanisms require switching to the kernel and back, so it's not clear to me why it should be less affected by Meltdown patches.

" Meltdown / Spectre patches have virtually no effect on video games."

The amount of speculative-execution that games rely upon for many of their components is insane. With the patch applied, I went from a solid 60FPS at 1440x900 in GTAV down to about 48 FPS (as measured by the Steam FPS counter) on a 7th-gen i3.

I could be mistaken, but I recall seeing benchmarks that came out during the whole meltdown/spectre fiasco showed that the effects on games were pretty negligible.

I thought it was mainly workloads relating to virtualization etc... that were effected significantly

Here are two benchmarks supporting this: https://www.eurogamer.net/articles/digitalfoundry-2018-does-..., https://www.tomshardware.com/reviews/gaming-performance-melt.... I remember it like that as well.

There have been many more updates since (for instance the first article you linked was from 01/10/18):

January 29th Microsoft had to revert the Intel patch discribed:

https://www.zdnet.com/article/windows-emergency-patch-micros...

There's also implementations in March for the Spectre variant 2:

https://www.digitaltrends.com/computing/microsoft-windows-pa...

There's much more in here.

it looks like anandtech's methodology was flawed: https://www.anandtech.com/show/12678/a-timely-discovery-exam...

Reply on news.ycombinator.com