You can see that the market deviates from the model and accounts for the fact that the Brownian motion model of Black-Scholes underestimates the probability of big moves: typically, options for the same security and the same expiration date have different IV, with options ATM having a lower IV and options deep ITM or deem OTM having larger IV.

If the market believed in the model, options for the same security and the same expiration date would all have the same IV, which would be whatever volatility the market thinks the security is going to have.

"and yet, in practice, it doesn't matter that much (most of the time)"

It does not matter? https://en.wikipedia.org/wiki/Long-Term_Capital_Management

Options far out of the market are underpriced. Mandelbrot, investing on the stockmarket is riskier than you think. But then, the opposite must also be true. It can be more lucrative than expected. I currently hold some far out of the money options. Unfortunately, the underlying stock goes against me :-(