The Fermi Paradox is predicated on the principle that aliens would actually want to come here. I'm not sure this is a valid assumption.
For a hot and fast civilisation like us to exist for serious amounts of time in space, it's going to have to adapt to living in space --- because, as you say, it's intensely hostile. One of the key adaptations is to stop being hot and fast. If you can live on small energy budgets you're way more likely to survive. If you live slowly then travel becomes much easier, as well as costing less.
But if you become a cold and slow civilisation, inner solar systems aren't going to be very hospitable --- hot, radioactive, space is so bent that travel is enormously difficult, everything's at the bottom of cripplingly expensive gravity wells... why bother? The gas giants are rich, but are even more expensive (to get from low orbit around one of the moons of Jupiter to low orbit around another takes about 10 km/s. You can launch from the surface of Earth for that). But further out... the Kuiper and Oort clouds are peaceful, cheap, hospitable, and chock full of light elements with interesting (if slow) chemistry.
Estimates of the Oort cloud density predict that there'll be a 20km body every 30 light seconds or so, and that there's about five Earth masses in Sol's Oort cloud alone. And Oort clouds of neighbouring stars overlap...
So I'm expecting any successful civilisation to start colonising the spaces between the stars, rather than the star systems themselves.
In this model, our own solar system could already be colonised, and we'd never know. They'd probably have noticed our radio transmissions by now, but it may not have occurred to them to answer yet. They may not bother; if we survive long enough to adapt to space, we'll be out there soon. Or they may not be able, depending on their energy budget. Or maybe they are, and we just haven't noticed. (New Horizons has a 12W transmitter and we would have to be looking in exactly the right direction to spot it. And it's way closer than the Oort Cloud.)
Of course, this doesn't explain why our hypothetical civilisation has built a hypothetical megastructure around this star. I wouldn't know why they'd want to do that (except for noting in passing that interstellar travel for hot-and-fast creatures like us basically requires a Dyson sphere for the energy requirements).
I'm planning a book. Can you tell?