The easiest conceptual handle is geometric: volume expansion, but seeing how this is related to the combinatorial sum over all permutations, or how those two point of views are related to the algebraic one (that a set of equations having a solution or not), is not easy to see even in the 2D case.
I won't be surprised if math professors don't have this issue like you said (especially if someone is comfortable with wedge products), but the vast majority of newcomers who are interested in understanding why something works rather than just how to use it struggle all the time with determinants.