These pieces are in orbit at many thousands of miles per hour, and have a random radar cross section, depending again, on whatever random impacts have already done to them, their orientation, composition.
The radar itself to track them must pass through the ionosphere, which itself causes variable time delays and phase shifts. Smaller objects return signals just above the noise after correlation in the receiver, most of them are just at the noise floor.
Observations only happen when the objects are in site of the tracking ground stations, which only cover a small portion of the sky, not including the expected collision volume.
All of that is just what I could recall might effect predictions off the top of my head... I'm amazed they can make predictions with any level of certainty.
And that's the difficulty with the simplifying (and wrong) assumption that that outside forces don't alter the orbit- even 2021, we have a lot of room for improvement when it comes to predicting many of the other factors that come into play, like solar activity and the Earth's climate.
Radar resolution of civil orgs might also be a limit.