It's also a lot more noise pollution than normal aircraft. I lived under the reentry path of the space shuttle back when it landed at Edwards Air Force base, and the sonic booms were loud as hell and shook the entire house, but only once every couple of months thankfully. We're not talking super close, either. Near middle of Los Angeles, almost 100 miles south.
I guess they could wait until they've very far away to actually go supersonic, though, and slow down while still really far from the landing strip, but if they're flying over populated regions at all supersonic, it's going to mini earthquake everyone underneath the flight path.