Not only, that, modern CPUs have transistors that switch in 0.1 ns. So even if they got to that speed, it would be 100,000x, not 1,000,000x.
And, if they only got to switching in 10 femtoseconds, it would be 10,000x, not 1,000,000x.
You might ask, what's two orders of magnitude between friends? But a job that takes a minute is quite a lot different from one that takes going on two hours.