What's "much slower"? 10%? 30%? 60%? Are the speeds that don't work in an x4 configuration even within spec, or "overclocked"?
Is that why everyone is complaining about ddr5 prices? Atm I'm fine with my ddr4 box and will probably still be for the next few years.
Tbh the last time I've seen a noticeable difference from memory speed was when I overclocked my ram (and system bus!) from 66 to 100 Mhz. Back in the pentium 1 / amd k5 era, i believe.
I do have a feeling what you're describing only means 5 more fps in some benchmarks...
And last, a friendly reminder. If your working set doesn't fit in ram you're screwed no matter what your ram speed is :)