There are a large number of assumptions built up around the singularity concept and CPU's are not getting exponentially faster so at least one of them seems incorrect. Honestly, the hardest problem IMO is the idea that intelligence is fungible and twice as much of it will provide anywhere near twice the result. The real world is fuzzy and thinking faster/deeper only get’s you so far.

Let's say we can build a machine smarter than a single person, that’s a long way from building something smarter than humanity so there is not a quick kick off to infinity. Now let's suggest we build a billion of the things and they are all smarter than the average person. Nothing says you can get anything close to a linear feedback between the intelligence designing an AI and how smart the AI is so there would not necessarily be a feedback loop to keep making more intelligent systems. etc.