Feels like we're going to see a lot of headlines like this in the future.
"AI comes up with bizarre ___________________, but it works!"
"AI comes up with bizarre ___________________, but it works!"
Imagine these headlines mutating slowly into “all software engineering performed by AI at certain company” and we will just dismiss it as generic because being employed and programming with keyboards is old fashioned. Give it twenty years and I bet this is the future.
People have been posing examples of similar "weird non-human design" results throughout here that are more than a decade old.
Of course today call something "AI" and suddenly interest, and presumably grant opportunities, increase by a few orders of magnitude.
OTOH, AI is very much a search in multidimensional spaces, it is so into it, that it would probably make sense to say that gradient descent is an AI tool. Not because it is used to train neural networks, but because the specialty of AI is a search in multidimensional spaces. People probably wouldn't agree, like they don't agree that Fundamental Theorem of Algebra is not of algebra (and not fundamental btw). But the disagreement is not about the deep meaning of the theorem or gradient descent, but about tradition and "we always did it this way".
The researchers in this article didn't do that. They used gradient descent to choose from a set of experiments. The choice of experiment was the end result and the direct output of the optimisation. Nothing was "learned" or "trained".
Gradient descent and other optimisation tools are used in machine learning, but long predate machine learning and are used in many other fields. Taking "AI" to include "anything that uses gradient descent" would just render an already heavily abused term almost entirely meaningless.
It is a simple iterative algorithm that goes from one point to the next. It doesn't even have memory of previous steps (caveat, the authors used BFGS which approximates the Hessian with previous gradient iterates, but this is still not AI). There is no finding weights or any such thing.
If every for loop is AI, then we might as well call everything AI. Can you pass me the AI, please?
This conflation of everything with AI is precisely why people say things like "gradient descent is most often used in ML" without evidence, and this likely being wrong. No, instead it is 1) ML is the currently most prominent (to the public) use of mathematical optimization and 2) everything else is called AI to the public so they conflate that with ML even when it isn't.
Take a random employee at an engineering or applied sciences (non experimental) lab, ask them if they ever use mathematical optimization, chances are a majority will tell you they do. The vast majority of these are not using or devising ML algorithms.
This matters because of what is clear from this thread. Some people devise a classic algorithm that requires intimate knowledge of the problem at hand, the press calls it AI, the public thinks it's AI, registers one more case of "AI as the tool to replace all others". The Zeitgeist becomes that everything else can go to the bin, and AI (by the more restrictive definition) receives disproportionate attention and funds. Note that funding AI research would not fund the people in the headline, unless they do like the minority of bandits that rebrand their non-AI work with AI keywords.
When the equation is too complex for analytic methods but good enough for gradient descent that equation is overwhelmingly the majority of the time characterized as AI.
A great deal of applied mathematics is related to finding a minimum or maximum quantity of something. There are not always constructive methods, sometimes (often) there's no better way than to step through a generic optimization method.
Some quick examples clearly unrelated to ML, and very common as they relate to CAD (everywhere from in silico studies to manufacturing) and computer vision:
- projecting a point on a surface
- fitting a parametric surface through a point cloud
Another example is non-linear PDEs. Some notable cases are Navier-Stoke's equations, non-linear elasticity, or reaction-diffusion. These are used in many industries. To solve non-linear PDEs, a residual is minimized using, typically, quasi-Newton methods (gradient descent's buff cousin). This is because numerical schemes only exist for linear equations, so you must first recast the problem as something linear (or a succession of those, as it were).
By the way, I might add that most PDEs can be equivalently recast as optimization problems.
Yet another is inverse problems: imaging (medical, non destructive testing...), parameter estimation (subsoil imaging), or even shape optimization. Similarly, optimal control. (similar in that it is minimizing a quantity under PDE constraints)
To summarize, almost every time you seek to solve a non-linear equation of any kind (of which there are many completely unrelated to ML), numerical optimization is right around the corner. And when you seek to find "the best" or "the least" or "the most" of something, optimization. Clearly, this is all the time.
I think I've provided a broad enough set of fields with ubiquitous applications, that it is clear optimization is omnipresent and used considerably more often than ML is. As you see, there is no association from optimization to ML or AI, although there is one the other way around. (much like a bird is not a chicken).