This line of logic doesn’t hold.
Star spanning structures requires the ability to move mass at scale which requires enormous but diffused in time and space amounts of controlled energy.
Moving between stars requires immense energy focused on a single thing. The amount of acceleration required to any meaningful mass to speed necessary to travel between stars and maintain a coherent civilization is extraordinary and implies many things about the mass being accelerated as well. Most things would simply be crushed instantly, including any organism. The speed of light is also a major impediment as is time dilation, as is the need to decelerate, which require equal and opposite energy away from the destination. Ok - accelerate with large lasers based at home - how do you then stop? You first need to build large lasers both at home and destination. The problems expand, and this only helps for very local travel between close stars.
It’s difficult but plausible with today’s technology scales up to build star spanning structures because they’re built incrementally and the need for near light speed travel doesn’t exist. Further the existence of a star spanning structure might enable interstellar travel, but not intergalactic.
I’d also note that SF is meant to entertain with a story that is possible given some assumptions of progress or hidden nature of reality or small modifications to what is. It’s not meant to be built on magic and other absurdities (not meant as a pejorative!). Many science fiction stories become true and many more will. Often science fiction inspires us to achieve what doesn’t exist today but could. To couch it in something that’s dismissible as say Gandalf the Wizard fighting a Balrog is to greatly underestimate what science fiction is. It’s a roadmap to the future that helps us understand what to avoid on the road.