I had an issue with Comcast last year where I would have large latency spikes for ~2 hours every night, and speedtest.net always remained at normal latency: https://i.fluffy.cc/kNCxLMbkF2wx7NHFJxbzcgX35Bsb5nvl.png
I couldn't find any non-speedtest sites where my latency was less than 100ms, but Comcast's own speed test, several other public speed tests, speedtest.net, and even ookla.com (the company behind speedtest.net) were perfect (the ~20ms you see on the graph).