If Spiderman can't move a bit of rubble (he doesn't really do dirt) then he can't pull something into or out of place to free a friend or trap an enemy.
It's (all) the little things that are in the way of a game feeling immersive and appealing to people who aren't just into the main mechanic (ie shooter on rails).
I'd really prefer for the developer to focus on the gameplay and plot rather than make a perfect simulation of earth. What you describe sounds like a cool sandbox, like Flight Simulator, but isn't appropriate for most games.
In the real world you can't enter or demolish any building you want. So it makes sense for games not to offer this either until it's an actual plot point to do so.
In real life, you can punch holes in and through buildings and trees, given enough bullets and a high enough caliber