Comment by agentcoops | Hacker News Reader

Comment by agentcoops | Hacker News Reader

Parent Full thread

I agree with your point, but I think it's worth noting that there's a real problem of language today both in popular and scientific communication. On the one hand, in popular understanding, there's the importance of clearly separating the era of "machine learning" as let's say Netflix recommendations from the qualitative leap of modern AI, most obviously LLMs. This article clearly draws on the latter association and really leads to confusion, most glaringly in the remark you note that the AI probably took up some forgotten Russian text etc.

However, scientifically, I think there's a real challenge to clearly delineate from the standpoint of 2025 what all should fall under the concept of AI -- we really lose something if "AI" comes to mean only LLMs. Everyone can agree that numeric methods in general should not be classed as AI, but it's also true that the scientific-intellectual lineage that leads to modern AI is for many decades indistinguishable from what would appear to be simply optimization problems or the history of statistics (see especially the early work of Paul Werbos where backpropagation is developed almost directly from Bellman's Dynamic Programming [1]). The classical definition would be that AI pursues goals under uncertainty with at least some learned or search‑based policy (paradigmatically but not exclusively gradient-descent of loss function), which is correct but perhaps fails to register the qualitative leap achieved in recent years.

Regardless -- and while still affirming that the OP itself makes serious errors -- I think it's hard to find a definition of AI that is not simply "LLMs" under which the methods of the actual paper cited [2] would not fall.

[1] His dissertation was re-published as The Roots of Backpropagation. Especially in the Soviet Union, important not least for Kolmogorov and Vapnik, AI was indistinguishable from an approach to optimization problems. It was only in the west where "AI" was taken to be a question of symbolic reasoning etc, which turned out to have been an unsuccessful research trajectory (cf the "AI winter").

[2] https://arxiv.org/pdf/2312.04258

"AI" is just a misleading and unhelpful term, exactly because it causes people to assume that there are properties we associate with intelligence (abstract thought, planning, motivations, emotions) present in anything given the term. That is easier to correct when someone is referring to a logistic regression. I think that "AI" has clung to LLMs because they specifically give the illusion of having those properties.

I would distinguish between:

- methods that were devised with domain knowledge (= numerical methods)

- generic methods that rely on numerical brute forcing to interpolate general behaviour (= AI)

The qualitative leap is that numerical brute forcing is at a stage where it can be applied to useful enough generic models.

There's a fundamental difference between any ML based method and, say, classic optimization. Let's take a simple gradient descent. This solves a very specific (if general) class of problems: min_x f(x) where f is differentiable. Since f is differentiable, someone had the (straightforward) idea of using its gradient to figure out where to go. The gradient is the direction of greatest ascent, so -grad(f) comes as a good guess of where to go to decrease f. But this is local information, only valid at (or rather in the vicinity of) a point. Hence, short of improving the descent direction (which other methods do, like quasi-Newton methods, which allow a "larger vicinity" of descent direction pertinence), the best you can do is iterate along x - h grad(f) at various h and find one that is optimal in some sense. How this is optimal is all worked out by hand: it should provide sufficient decrease, while still giving you some room for progression (not too low a gradient), in the case of the Wolfe-Armijo rules, for example.

These are all unimportant details, the point is the algorithms are devised by carefully examining the objects at play (here, differentiable functions), and how best to exploit their behaviour. These algorithms are quite specific; some assume the function is twice differentiable, others that it is Lipschitzian and you know the constant, in others you don't know the constant, or the function is convex...

Now in AI, generally speaking, you define a parametric function family (the parameters are called weights) and you fit that family of functions so that it maps inputs to desired ouputs (called training). This is really meta-algorithmics, in a sense. No domain knowledge required to devise an algorithm that solves, say, the heat equation (though it will do so badly) or can reproduce some probability distribution. Under the assumption that your parametric function family is large enough that it can interpolate the behaviour you're looking after, of course. (correct me on this paragraph if I'm wrong)

To summarize, in my (classic numerics trained) mind, classic numerics is devising methods that apply to specific cases and require knowledge of the objects at play, and AI is devising general interpolators that can fit to varied behaviour given enough CPU (or GPU as it were) time.

So, this article is clearly not describing AI as people usually mean it in academia, at least. I'll bet you a $100 the authors of the software they used don't describe it as AI.

Reply on news.ycombinator.com