The Fermi paradox is "why a highly developed civilization isn't behaving like a Western colonialists or tourists toward primitive natives of some remote wilderness".
Would they eschew expansionism? Maybe. Would they always eschew it 100% of the time? Definitely not. And this is a Darwinean process - the ones that decided to stay home are the ones you don't meet.
And they've had a billion years to change their minds. Consider that the likelihood is numerous billion-year-head-starts, one species would inevitably try it.