We know that Neanderthals may have had bigger brains, and may have been pushed to extinction by homo sapiens. A bigger brain will require more food and take longer to develop, potentially putting Neanderthals at an evolutionary disadvantage to humans.
Assuming the laws of Physics do not vary throughout the universe and that evolution, if "run more than once" will eventually lead to similar outcomes, that could mean aliens capable of space travel are not significantly more intelligent than us, they've just had longer to develop technology.
And given their apparent tendency to crash their spaceships while visiting earth, you might have a reasonable case for "alien idiots"...