The problem is most people were taught math wrong and this typically only gets corrected when you get to high level mathematics like abstract algebra.

Imaginary numbers aren't "imaginary" they are how to do consistent math in a 2D framework. Quaternions? Well that's about 4D. Poincaré once said that math isn't about numbers, but relationships between numbers. For some reason we don't talk about mathematical structure (explicitly) until the late stages. I wouldn't necessarily call these concepts "abstruse" but they are a bit more abstract. A big part of the problem though, we often ignore the ground we are building upon and so when you finally look at it, it is new and confusing. But then again, the success of Bourbaki's New Math is arguable[0]

[0] https://en.wikipedia.org/wiki/New_Math#In_other_countries