It's probably because you have not heard it often enough to just not think about it. If I were to describe a sound as being rectangle, it just doesn't make sense and I have to think about it and try to interpret it. The extra effort in trying to decode the meaning is just annoying. In contrast, you've probably heard the phrase sharp sound. It's equally as technically meaningless as crisp sound, but you probably get it and aren't annoyed by it.

I'm not entirely certain - I use it to describe the opposite of "dull/distorted" sound, kind of a synonym of "fresh". I really have no idea where I picked it up, other than I grew up on the west coast.

Not sure why it bothers you. Is English your first language? This kind of thing is often difficult for non-native speakers.

Crisp is a word with a few meanings. In this context, it's a metaphor assigning physical qualities to sound. (A crisp dollar bill is smooth and without creases)