I am not arguing there is not math involved, but gradient decent is basic calculus MSE, MAE, etc. This is something you take in high school. This stuff is pretty basic. Even in gradient decent you might have multiple minimums. The minimum you get will dependent on which set of points you start at. Imagine a 3d curve you might have different valleys.

Let me quote François Chollet: "Neural networks" are a sad misnomer. They're neither neural nor even networks. They're chains of differentiable, parameterized geometric functions, trained with gradient descent (with gradients obtained via the chain rule). A small set of highschool-level ideas put together