Here are the list of gates, go build your algorithm
https://en.m.wikipedia.org/wiki/Quantum_logic_gate
(Hint, the hardest part is hardware, not the math)
I don't want my quantum computer to constrain me to building up circuits with matrix math operations. Most algorithms you do today are very poorly represented as matrix math that way. As a toy example, write up a sorting algorithm as matrix math using classical gates for me, and come back. Then I will start treating logic gates as a serious way to write algorithms.
Pretty often you'll find that papers on the higher-order applicability of quantum computing, which tweak in simple ways or glue together existing basic algorithms, won't use the circuit model for their algorithms at all.
The reason the matrix math persists is twofold. Firstly, we don't have good quantum computers, and gate depth is severely limited, plus it can quickly became intractable to simulate inefficient circuits without using tricks, even if they are doing something simple.
So we are at a state in quantum computing where writing the basic algorithms requires absolutely extreme optimization, down to the level of logic gates, because it's not feasible to implement or even rigorously prove much any other way.
The other issue is that quantum computing offers you an infinite degree of freedom on the operations you can do. While in classical computing there are only two operations you can do on 1 bit, there is an infinite number of operations you can do on 1 qubit. These operations are easily described by a set of orthogonal normalized vectors (as a change of basis), so of course a matrix is the most natural way to describe them. The infinity of basic operations really is the problem here, unfortunately, so simpler ways of describing basic operations aren't really possible. Of course, that doesn't excuse the circuit based approach - that is however due to a limitation in technology.
By the way, I have been a professional FPGA developer for a while, and every few years, vendors think "this will be the time that FPGAs get broad adoption." AI inference is (right now) a perfect problem for FPGA use (literally 100-1000x more efficient than GPUs), but almost nobody cares, despite how powerful the computers are. The reason nobody cares is that FPGA programming is about constructing classical circuits - thinking about algorithms that way is REALLY hard. For example, hash tables were invented in the 60's, but they came to FPGAs in the 2010's. Sure, you can kind of do recursion and other similar ideas, but it's generally really annoying to cram an algorithm into a representation that is FPGA-friendly.
I don't want quantum computing to wedge itself into the same trap, particularly when it doesn't look like it needs to on any fundamental level. Maybe I'm wrong that quantum computers will eventually overcome the limitations on depth, etc.