When you progress from arithmetic to abstract, it hardly gets easier.
Moreover, memorizing. You go from memorizing multiplication tables to other kinds of tables, like tables of equations giving various identities, rows of coefficients in series, and the like. Contents of various kinds of matrices.
The need to be precise and avoid mistakes never goes away. Manipulating a complex math equation is still a form of arithmetic. And it's harder because the underlying semantics means that something which is mechanically correct at the syntax level (easy to check) could actually be meaningless and wrong.
The simplistic notations used in math don't "keep up" with the increasing complexity of what is going on. They just get harder semantics. Notation which looks like multiplication or addition in such and such domain is just "sort of" like it, but, oh, here are the ways in which it isn't.