Many quantum algorithms aren't actually written as matrix math using gates - you can use something ZX as an alternate representation, and if your algorithm is composed of other algorithms you can abstract them and use recursions just fine (and that is in fact done fairly often). There are also other models of writing down quantum computation that don't use logic gates.
Pretty often you'll find that papers on the higher-order applicability of quantum computing, which tweak in simple ways or glue together existing basic algorithms, won't use the circuit model for their algorithms at all.
The reason the matrix math persists is twofold. Firstly, we don't have good quantum computers, and gate depth is severely limited, plus it can quickly became intractable to simulate inefficient circuits without using tricks, even if they are doing something simple.
So we are at a state in quantum computing where writing the basic algorithms requires absolutely extreme optimization, down to the level of logic gates, because it's not feasible to implement or even rigorously prove much any other way.
The other issue is that quantum computing offers you an infinite degree of freedom on the operations you can do. While in classical computing there are only two operations you can do on 1 bit, there is an infinite number of operations you can do on 1 qubit. These operations are easily described by a set of orthogonal normalized vectors (as a change of basis), so of course a matrix is the most natural way to describe them. The infinity of basic operations really is the problem here, unfortunately, so simpler ways of describing basic operations aren't really possible. Of course, that doesn't excuse the circuit based approach - that is however due to a limitation in technology.