Do you think China would have hesitated to shoot down an American "weather" balloon, the size of three school buses, if it entered China's airspace? I doubt it would have been allowed to float for thousands of miles before being downed.
The US launches rockets primarily from Kennedy Space Center and Vandenberg AFB so they spend minimal time flying overland as they fly out to sea. This is so failures or aborts pose as little risk as possible to people on the ground.
China launches rockets from inland and have dropped failed and aborted rockets on its citizens on the ground numerous times.
So yes, it is "really that hard" to shoot something down without causing collateral harm on the ground. And yes, China gives less fucks about human life than the US does.