I had an issue with Comcast last year where I would have large latency spikes for ~2 hours every night, and speedtest.net always remained at normal latency: https://i.fluffy.cc/kNCxLMbkF2wx7NHFJxbzcgX35Bsb5nvl.png
I couldn't find any non-speedtest sites where my latency was less than 100ms, but Comcast's own speed test, several other public speed tests, speedtest.net, and even ookla.com (the company behind speedtest.net) were perfect (the ~20ms you see on the graph).
What you describe is probably congestion in the network somewhere that isn't the last mile. Speed tests aren't a good indicator for any issues occurring anywhere but the last mile, as they won't take the same path with fairly high probability.
Really, my point was that congestion is often localized in a network. The fact that it isn't present in a path doesn't mean it's an ISP gaming speed tests - though perhaps they'd be more keen on peering directly and avoiding congested transit if it makes them look better, and I would absolutely believe that would happen.
On a separate note, what tool did you use to create that graph?
[1] https://github.com/cfallin/pingd (in Rust, requires liboping)