I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.
(One of) The value(s) that a math grad brings is debugging and fixing these ML models when training fails. Many would not have an idea about how to even begin debugging why the trained model is not working so well, let alone how to explore fixes.
Why didn't the training converge
Validation/test errors are great but why is performance in the wild so poor
Why is the model converging so soon
Why is this all zero
Why is this NaN
Model performance is not great, do I need to move to something more complicated or am I doing something wrong
Did the nature of the upstream data change ?
Sometimes this feature is missing, how should I deal with this
The training set and the data on which the model will be deployed are different. How to address this problem
The labelers labelled only the instances that are easy to label, not chosen uniformly from the data. How to train with such skewed label selection
I need to update model but with a few thousand data points but not train from scratch. How do I do it
Model too large which doubles can I replace with float32
So on and so forth. Many times models are given up on prematurely because the expertise to investigate lackluster performance does not exist in the team.
But the examples you quoted were not my examples, at least not their primary movers (the NaNs could be caused by overflow but that overflow can have a deeper cause). The examples I gave have/had very different root causes at play and the fixes required some facility with maths, not to the extent that you have to be capable of discovering new math, or something so complicated as the geometry and topology of strings, but nonetheless math that requires grad school or advanced and gifted undergrad level math.
Coming back to numeric overflow that you mention. I can imagine a software engineer eventually figuring out that overflow was a root cause (sometimes they will not). However there's quite a gap between overflow recognition and say knowledge of numerical analysis that will help guide a fix.
You say > "literally every single example"... can be dealt without much math. I would be very keen to learn from you about how to deal with this one, say. Without much math.
The labelers labelled only
the instances that are
easy to label, not chosen
uniformly from the data.
How to train with such
skewed label selection
(without relabeling properly)
Consider the case that the labelers creates the labelled training set by cherry picking those examples that are easy to label. He labels many, but selects the items to label according to his preference.
First question, is this even a problem. Yes, most likely. But why ? How to fix it ? When are such fixes even possible.
This can be partially solved if we make some assumptions about your labeller:
1. they have still picked enough challenging samples.
2. their preferences are still based on features you care about.
3. he labelled the challenging samples correctly.
And probably some other assumptions should hold for distribution of labels, etc. But what we can do in this situation is first try to model that labeller preferences, by training a binary classifier - how likely he would choose this sample for labelling from the real-world distribution? If we train that classifier, we can then assign its confidence as a sample weight when preparing our training dataset (less likely samples get more weight). This would force our main classifier to pay more attention to the challenging samples during training.
This could help somewhat if all assumptions hold, but in practice I would not expect much improvement, and the solution above can easily make it worse - this problem needs to be solved by better labelling.
How did you solve it?
For this to work the two measures need to be absolutely continuous with each other.
This is close to your pre-penultimate paragraph and that's mathy enough. This done right can take care of bias but may do so at the expense of variance, so this Radon Nikodym derivative that is estimated needs to be done so under appropriate regularization in the function space.
Thinking of the solution in these terms requires mathematical thinking.
Now let's consider the case where some features may be missing on instances at the time of deployment but always present in training and the features are uncorrelated with each other (by construction).
More often than not this is duplicated work (mathematically speaking) and there is a lot to be gained by sharing advances in either field by running it through a "translation". This has happened many times historically - a lot of the "we met at a cafe and worked it out on a napkin" inventions are exactly that.
Math proficiency helps a lot at that. The level of abstraction you deal with is naturally high.
Recently, the problem of actually knowing every field enough, just cursorily, to make connections is easier with AI. Modern LLMs do approximate retrieval and still need a planner + verifier, the mathematician can be that.
This is somewhat adjacent to what terry tao spoke about, and the setup is sort of what alpha evolve does.
You get that impression because such advances are high impact and rare (because they are difficult). Most advances come as a sequence of field-specific assumption, field-specific empirical observation, field-specific theorem, and so on. We only see the advances that are actually made, leading to an observation bias.
Instead of people "hacking" university education to make them purely fotm job training centers. The real hack would be something that really drills down at the fundamentals. CS, Math, Physics, and Philosophy to get an all around education in approaching problems from fundamentals I think would be the optimal school experience.
So in my specific case I stopped thinking about mathematics as: how to interpret a sequence of symbols
But instead I decided to start thinking about it as “the symbols tell me about the multidimensional topological coordinate space that I need to inhabit
So now when I look at a equation (or whatever) my first step is “OK how do I turn this into a topology so that I can explore the toplogical space the way that a number would”
Kind of like if you were to extend Nagle’s “what it’s like to be a bat” but instead of being a bat you’re a number
The way I like to look at it is that I'm engaged in the ice trade and they are about to invent everything else that will end mine and every other current trade. Which leaves me with two practical options: a) deep despair. b) to become a Jacks of all trades, master of none, but oftentimes better than a master of one. The Jacks can, for now, capitalize in the thing that the Machines currently lack, which is agency.
Once the fundamental concepts are understood, what problem is being solved and where the key difficulties are, only then the equations will start to make sense. If you start out with the math, you're making your life unnecessarily hard.
Also, not universally true but directionally true as a rule of thumb, the more equations a text contains the less likely it is that the author itself has truly grasped the subject. People who really grasp a subject can usually explain it well in plain language.
That's very much a matter of style. An equation is often the plainest way of expressing something
You give a physicist an equation of a completely unrelated field in mathematics and it will make zero sense to them because they lack the context. And vice versa. The only people who can readily read and understand your equations are those that already understand the subject and have learned all the context around the math.
Therefore it's pointless to try to start with the math when you're foreign to a field. It simply won't make any sense without the context.