Hacker News Reader

Comment by samfisher83 | Hacker News Reader

I am not sure when you are using neural nets how much knowing the underlying math helps you. Yes we are doing matrix multiplies and using gradient decent. That is stuff you probably learned in high school. Usually you just learn partial derivatives in two dimensions instead of 10s or 100s of dimensions.

Even if you understand the math you are training tens of thousands of parameters inside your model. You are also passing them through non linear elements like sigmoid of ReLUs. I am not sure what insight knowing calculus or linear algebra will provide you in building the model unless you can process more than 3 dimensions non linear elements in your head. I am sure there are people that can do it, but how many can do it?

View on HN

Actually very much. Think of it from an industry perspective.

If you are a software engineer that needs to use a network as a plug in module, then you may not need much understanding of linear algebra. But then you also don't need to know much ML either. It is simply a software engineering job.

However things change for anyone who gets their hands dirty or actually builds (even just implements know networks ) from scratch .

It is common to find that error caused by transferring a known model to your problem, was because of making key mistakes in how the question was posted mathematically. I am currently implementing Neural nets in my job, and have made many errors because of misunderstanding the mathematical operation of a layer.

PGMs are almost entirely math and so are embeddings and matrix factorization models. VAEs and GANs also need a solid grasp of the math behind them. Want to touch your loss function ? -> Math. Want to change optimization methods ? -> Math.

The visual designs of almost all neural nets are grounded in math, and this fact makes it vital to be good at it, if one wishes to gain anything beyond a surface level understanding of it.

I am not arguing there is not math involved, but gradient decent is basic calculus MSE, MAE, etc. This is something you take in high school. This stuff is pretty basic. Even in gradient decent you might have multiple minimums. The minimum you get will dependent on which set of points you start at. Imagine a 3d curve you might have different valleys.

Let me quote François Chollet: "Neural networks" are a sad misnomer. They're neither neural nor even networks. They're chains of differentiable, parameterized geometric functions, trained with gradient descent (with gradients obtained via the chain rule). A small set of highschool-level ideas put together

You need a great knowledge of math to internalize the complexity of neural nets. One particular area of mathematics that is not in the set of highschool-level ideas is topology.

Quick question, do you do much machine learning in your current job on complex and large datasets? I find that the people who claim that machine learning is not math heavy are not solving the actual hard ML problems. Ofcourse it will be easy if your problem maps well to some framework, where jamming your data into a model produces useful outputs, but theres plenty of problems where you cant do that, and the second that happens, you need to understand the algorithms. Its similar to using some javascript framework, where you can plug in solutions great, until you need to do something outside the framework.

Calculus is helpful for understanding backprop.

Reply on news.ycombinator.com