Just speculation (I am not a physicist or astrobiologist), but this is my thinking also. If anything is traveling between the stars, it's either inorganic (an AI), or it has a natural lifespan several orders of magnitudes longer than humans. Without FTL travel, to conquer space you also have to conquer time.

Organic life is fragile and ill-suited for space travel, at least organic life as we know it. Perhaps the destiny of all sufficiently advanced organisms is either to die off or replace itself with AI.

Anyway, while I find it impossible to believe there's nothing out there, I find it nearly equally impossible to believe anything that's ever visited our rock resembled little green men.