Actually, he pushes this idea even further: in the book, our solar system was already engulfed in a few spheres millions of years ago. He suggests that this why Venus is such a hellscape: the aliens came, took the resources they wanted, and left behind a polluted mess. In the case of Venus, they left lots of greenhouse gases behind as the result of some chemical process used to extract resources; as a result, Venus quickly became the warmest planet in the solar system. It's a fun twist on the Fermi paradox: signs of aliens are actually all around us, we are just too dumb to notice them.
Another interesting idea he explores a bit is "ownership" of resources. Do the resource-rich asteroids in our solar system really belong to us? Or are they available to any alien race who happens to pass through? In the book, we first notice aliens by observing unexplainable infrared radiation from the asteroid belt (later revealed to be thermal emissions from their resource extraction). He suggests that these aliens will potentially crowd out humans; even if they are not overtly hostile, they could gobble up all the resources we would have used to expand our civilization.
Highly recommend this book.