The limitations of gradient descent as a principle of brain function

The "money" quote is right here,

> Steepest descent or gradient descent depends on a choice of ruler (or Riemannian metric) for the parameter space of interest. The Euclidean metric is rarely a natural choice, especially for spaces with dimensions that carry different physical units, such as the example shown here.

But I think this, to many ML researchers or biophysicists, is true by what I lovingly call the NSH[0]. The argument posed (in a slightly anthropomorphized way) goes as follows:

-

We have some function, f, which we want to minimize/maximize: usually f will be the negative log of some probability distribution, or whatever. Now, hey, look, this function could be anything that's differentiable, all we know is that the minimum of this function is really useful.

But, what happens if we replace this function by another function, g, whose minimum at x is the same, but g(x + ∆) ≈ g(x) for ∆ being fairly large, then we run into a problem! Minimizing g (which is, admittedly, an unnatural representation of f) is bad because it's really hard for us to distinguish points around x from x itself (which is the point we want) using only the gradient information, since it's very close to zero. In other words, while the function g preserves many of the same properties as f (in fact, the only one we care about, the minimum), the trajectory of gradient descent on g is going to be mostly pretty dumb as compared to what it could be for f.

In the case that f is the negative log-likelihood of some empirical distribution, then the Fisher information turns out to be a natural choice for the curvature of the distribution (see Information Geometry, if you're interested in this idea) and running what is essentially a Newton method on f will yield similar trajectories to running it on g, which is what we want.

-

The paper makes no particularly strong argument that is of note, except that, yes, of course, "the brain" doesn't actually "run" gradient descent as stated on paper, which apparently is an assumption some neuroscience models make use of to make predictions. This is... not really a claim anyone is arguing for, as far as I know (the paper has citations [12,13], but taking a quick peek, only 12 mentions this idea directly, but even there invokes the "natural gradient" argument that the authors here propose, and only mentions, in passing, that this mechanism is "biologically plausible").

---

[0] More specifically, the No-Shit Hypothesis.

Noted that the example they use from Rui Costa et al. 2017 (a great paper btw recommended reading) is not any kind of machine learning result, it is a fitting of data from previous experiments showing pre- and post-synaptic changes during short term LTP experiments. So i 'm not sure what the authors are rambling on about here.

I really can't believe anyone argues that brains can just be gradient descent. It's like people have never heard of a fitness valley or a sparsely connected graph region.

I can't see a PDF here, I'm shown a 46 byte file to download.

tl;dr ordinary gradient descent is sensitive to the parametrization of the problem but natural gradient is not. This is an important fact (and one that is fairly well-known within the ML community), but it is not totally clear to me why it should be particularly relevant to neuroscience.