By the same logic you might as well say: "why are we even kerning fonts, who cares if there's a few gaps when i write »irl«."
The fact that using different dashes does encode meaning in a subtle sense does have relevance for semantics -- but that's, imho, almost secondary to this argument, as it's not as grammatically relevant as commas and. periods, for example.
The primary importance of using the correct dashes is that it preserves a good flow for reading and is paramount to micro-typographic balance:
- A longer dash to link words that belong together is visually perceived as an interruption and doesn't feel like those two words are one
- In reverse, a shorter dash when switching context -- or interjecting another idea within a sentence -- doesn't slow the pace of the text flow enough, and your brain will read/intonate it the same way as when linking words.
- And at last, either of them won't preserve optical balance when displaying a numerical range, as numbers are wider than a hyphen, but narrower than an em space, which would result in either insufficient visual separation compared to spaces following said numbers, or too much of an optical gap within an entity that belongs together.
That's the barebones set of dashes that are relevant for a balanced typographical appearance, not made up pedantic complexity to annoy people. Otherwise we'd be taking about half and quarter em dashes and the likes.