Velocity is important though.
A very rough calculation of mine involving a hypothetical asteroid in a elliptical orbit extending as far as Jupiter and right down to Earth, assuming no difference in orbital inclination to Earth and no significant gravitational perturbations, would result in a relative speed of 5km/s. The actual impact speed would be greater due to Earth's own gravity, adding an extra 11km/s.
Not all asteroids are from the asteroid belt, but I am under the impression that visitors from the outer solar system (which could be as fast as the upper bound that ddahlen mentions) are much more infrequent than stray asteroid belt objects, so the median impact speed would still be relatively slow.
Earth's orbital diameter is: ~30 x 10^7 km
number of seconds in a year is: ~ π x 10^7 s
so, Earth's orbital velocity is: ~30 km/s
speed of light, c, is: ~30 x 10^4 km/s
so, Earth's orbital diameter is: ~ 1,000 light seconds
and, Earth's orbital velocity is: ~ 0.0001 cJust doing some back of the envelope calculations, looks like Omuamua was moving about 165,000km/hr (relative to Earth) when it was about at Earths orbital distance.
This speed is not actually a crazy number, it is a lot faster than the majority of things which could hit us, but there are geometries of things in our solar system which can reach these relative velocities. (For example things in retrograde, IE: reverse orbits) can lead to basically escape velocity + earths velocity.
Looks like there is not a significant amount of variance in asteroid speed so mass would be the biggest deciding factor.
Tiny stuff burns up completely in the upper atmosphere, where the pressure is low, because they have low surface area per mass -- the atmosphere can stop them entirely. Their terminal velocity is low. (That is, when the velocity through air is high enough that the drag prevents gravity from speeding up the object any further.)
Medium objects have a higher terminal velocity get deeper into the atmosphere before exploding. Fragments from these (which now have higher surface area per mass) can then be slowed further by the atmosphere and make it to the surface, but not so dramatically. Bits of the Chelyabinsk impactor fall into this category.
Big objects have a high terminal velocity. They make it to the ground largely intact... and without being slowed as much by the atmosphere. That gives you craters and bad days for being a dinosaur.
These are just random guesses though, so I could be completely wrong.