Yes, I get these too. Pretty much every time I talk to someone who takes an interest in my startup, they raise some issue that I either hadn't thought of at all or hadn't thought about in the same way. That doesn't even cover the hacking-related 'oh crap' moments, but most of mine have been conceptual so far. It's to be expected: there are tens of startups in my space, I assume that some are technically better and many are worse.

As you point out, the hardest thing may not be the 'oh crap' moment but rather what conclusions you should draw from it, if any. In most cases I have elected to go simpler when in doubt. Other times I've decided to re-think various aspects of my project, but these are in the minority.