Actually, Moore's law says nothing about speed, just the number of transistors you can stuff into the thing. Plus progress has been leveling off.
Make no mistake, he knew he worked on the production of CPUs and memories, it was just not his focus. Also, he certainly cared about switching time, but his observation was about transistor count, not switching time.