A. Mercury is small.
B. Sun is close, and huge objects like sun tend to clear their neighbourhood of small rocks. (There are comets coming your way, though.)
C. Even if something hits Mercury, there are no oceans, so zero risk of a tsunami wreaking havoc 1000 km from the original point of impact. This is one of the disadvantages of having a liquid ocean on our surface.
That said, asteroids are not the main problem. The radiation on the surface of Mercury would be a challenge. Humans wouldn't be able to work outside during Mercury day at all, unless we can construct much better protective suits. Even getting there would be a challenge, you would get a lot of exposure during the flight.
But Mercury is rich in minerals, so perhaps we can find something of value in mineable concentration there. And building a space elevator on Mercury should be possible with current materials.
So in my case, the jury is out.
Cargo, unlike humans, can take its time on the way. In case of minerals, it likely won't be damaged by vacuum or radiation exposure. The only thing that really interests you is that it arrives on schedule.
Earths atmosphere protects us from all but the biggest rocks. Micrometer impacts on Mercury are a real danger, but that's also true in space/moon/mars