Not only overdraw but also having to be careful with how you craft the scene. SM pretty much allows for arbitrary scenes.

Also small nitpick: you don't filter the shadow map, but the resulting shadow response.