There's the question of motivation. It seems possible that an advanced civilization could gravitationally disrupt a small planet like mercury, turn it into solar collectors and particle accelerators and create multiple-ton quantities of antimatter and use that for interstellar travel, but they would likely choose to do something else with those resources.
People have a hard time thinking on a timescale of half a lifetime, never mind a lifetime or several lifetimes. The realistic version of the Bussard ramjet is not that you aim for c=0.1 or more, but rather with D-D fusion you can make hops of 1,000-10,000 au to Oort cloud bodies, you could possible gravitationally disrupt an object like Pluto and turn it into little ringworlds or other colonies. It might take people like that 10,000 years to make it to the next star but if they can make a comfortable lifestyle along the way why would they bother? And why would we even care enough to send them out on such a quest?
Even colonizing Mars is problematic in the sense that it's inconceivable that you could bring anything back from Mars that would give us flatlanders any economic incentive to do it. It might be a gift to a population of a billion or so people that will live there, but profitable to us? No way.
So even if you could colonize the galaxy with Von Neuman probes, it won't happen in your lifetime, your children's lifetime, etc. Our civilization might not be there to get the data from the first probes to reach other stars and the more steps removed the more the probability drops. Still there's just that tiny probability of a 'dark forest' scenario where somebody finds one of those things and decides to do something about it.