The humans will end up spending 99% of their aggregate time in locations that are large enough to bore habitat out of solid rock. Moons, planetoids, substantial asteroids.
Some of the other large, iron-rich asteroids like Ceres, Vesta, Pallas, and Interamnia are more like protoplanets than rubble piles.
Besides the concern for structural integrity/stability, they also have reasonable amounts of water ice, volatiles, metals, ad other resources needed to supply an outpost.
"The spacecraft risked being swallowed whole by the asteroid as it collected the sample. As OSIRIS-REx touched the ground in October 2020, pebbles began flying about. The timely ignition of its thruster kept it safe and led to the creation of a puzzling large crater 8 meters (26 feet) across." https://www.iflscience.com/nasas-osiris-rex-was-almost-swall...
Wow.