I don’t think that this reply takes into consideration just how inefficient RL and the likes are. In fact, RL is so inefficient that current SOTA in RL is … causal transformers that perform in-context learning without gradient updates.
Depending on the approach one takes with RL, be it policy gradients or value networks, it still relies on gradient descent (and backprop).
Policy gradients are just increasing the likelihood of useful actions given the current state. It’s a likelihood model increasing probabilities based on observed random walks.
Value networks are even worse because one needs to derive not only the quality of the behaviour but also select an action.
Sure enough, alternative methods exist such as model based RL, etc, and for example ChatGPT use RL to train some value functions and learn how to rank options, but all of these rely on gradient descent.
Gradient descent, especially stochastic, is just garbage compared to stuff that we have for fixed functions that are not very expensive to evaluate.
With stochastic gradient descent, your loss landscape depends on the example or mini batch, so a way to think about it is that the landscape is a linear combination of all the training examples, but at any time you observe only some of them and cope that the gradient doesn’t mess up too bad.
But in general gradient descent shows linear convergence rate (cf Nocedal et al Numerical Opt, or Boyd and Vanderberghe’s proof where they bound the improvement of the iterates), and that’s a best case scenario (meaning non stochastic, non partial).
Second order methods can get quadratic convergence rate but they are prohibitly expensive for large models, or require hessians (good luck lol).
None of these though address limitations imposed by loss functions, eg needing exponentially higher values to increase a prediction optimised by cross entropy (see the logarithm). Nor do they address the bound on the information that we have about the minima.
So needing exponentially more steps (assuming each update is fixed in length) while relying on linear convergence is … problematic to say the list