But they do obscure the meaning in a lot of proofs. If you start using them immediately, it is also hard to motivate where they come from, what's the intuition behind them and how the students could arrive at the definition themselves.
And even if you use them and they produce short one-line proofs, you should also prove all the properties behind them first, which is where all the complexity is hiding.
The nice point about Linear Algebra Done Right is that it explains how Linear Algebra works without resorting to "determinant magic". In the end, you will also understand why determinants work so well in so many proofs.
In any case, his book is really suited for a second course in Linear Algebra, so maybe it is beneficial to start using determinants right away. Personally I understood them better with his book.
Disclaimer: I don't know Sheldon Axler, but I read his book.