On the upside they're not quadcopters, the noise from dozens or hundreds of those flying by daily would drive people completely insane.
Cars and trucks are really loud
Just be as far away as you'd like an actual highway, but in this case vertical space works too.
So a typical 80db commercial drone flying at 300m (CAA minimum) would be 50db at ground level (closest point), or about a loud conversation (but less than quiet road traffic).
Source: I did drone noise research a few years ago. It's almost impossible to detect via noise with decent distance (>100m) unless directly under.