Short of a lazier universe than we're willing to accept, it almost certainly did cause an effect, including harmonic vibrations, ones that are replicated around like echos, etc. It's just unclear if it's sound if there's nobody there to interpret it.
Not saying I buy it one way or the other, just that it's the reason I never thought it was as stupid as you do.
Extending this take back to thread root and then the objection, I always understood "speed of sound" to be more or less "speed of causalty filtered through a substrate", i.e. if you bump into this side, how long does it take that side to be affected by it? "Sound" is just one potential type of bump to be traveling, if we subjectively interpret the vibrations that eventually make it through to the other side that way.