Given the ratio of one black hole per 1000 stars, which is the best we can guess right now, the average distance to the closest black hole is 36 lightyears.
I don't know if we'd have expected to have seen lensing effects from our whole-sky surveys, but it seems to me that this would also be probable.
Finally, if you send a line through our galaxy, you'd expect it to pass near a lot of these black holes -- and thus you'd expect to see a lot of microlensing effects which the article states don't seem to occur nearly often enough.
Normal black holes are really small, much smaller than a planet, which we can barely detect now. And those planets must be orbiting a star and in the correct inclination to be detected.
Primordial black holes are posited to be possibly so small as to be able to pass through the earth without interacting with more than a few atoms, which would make visiting them both difficult and not enormously interesting.
Edit: clarification and added info on primordial black holes which I looked up.
In other words, if you are outside a spherical mass the pull of gravity gets stronger as you get closer (~1/r^2). But once you are inside the mass, the pull gets weaker. Hence there is no black hole at the center of the earth, for example.