Direct interaction isn't needed for havoc. A supermassive object sweeping by the Solar System could destabilize Jovian orbits. In the Nice model, Neptune flung Kuiper belt asteroids sunward, gifting the inner planets with a late heavy bombardment.
Rogue gas giants, brown dwarfs accelerated to relativistic speeds, giant asteroids approaching from the Sun's direction, Carrington Events, an ill-directed gamma ray, etc. So many ways life on Earth can see its 250 million remaining years cut short, and those are only a few of the cosmic threats we can imagine.
A black hole with a Schwarzschild radius of 20 km would weigh about 6.8 Solar masses. It wouldn't even need to get super close to affect the Solar System.