The majority of DirectX stuff in the innermost-loops automatically DMAs to the video card without even needing a kernel call. Video games care a lot about performance and did their best to never touch kernel-space anyway.
Compilers, Web Servers, Databases, etc. etc. were affected very strongly. Compilers open files. Web Servers open Sockets. Databases communicate with mutexes / semaphores. All of these are OS-level features and require a kernel-mode switch. Meltdown means that every kernel-mode switch clears out the TLB-cache.
So each time a compiler opens a file (MMap or otherwise), it basically loses its entire TLB cache. Every. Single. System call. That's why its a big deal for servers, but not a big deal for video games.