I feel the same. I dry my hands on my trousers and wear headphones in the toilets to avoid the blast of white (?) noise from other users of the high-speed dryers at work.
It wouldn't surprise me if the UK regulations on these devices also only measure the noise levels without hands in the airflow... there's no way these are legal when they are actually in use.