Hacker News Reader

Comment by tlb | Hacker News Reader

If you're surprised by this result because you've used kernel machines and didn't find them very good at generalization, keep in mind that this assumes a kernel function that accurately reflects similarity of input samples. Most work with kernel machines just uses Euclidean distance. For instance, in an image recognition model it would have to identify 2 images with dogs as more similar than an image with a dog and an image with a cat.

With a sufficiently magical kernel function, indeed you can get great results with a kernel machine. But it's not so easy to write a kernel function for a domain like image processing, where shifts, scales, and small rotations shouldn't affect similarity much. Let alone for text processing, where it should recognize 2 sentences with similar meaning as similar.

View on HN

This may not have been your intention but I think you're underselling the utility of this result a bit. Yes you need a good kernel function to get good results but this paper literally provides a way to create such a kernel.

With a way to turn a gradient descent into a roughly equivalent kernel machine you can turn gradient descent into an embedding, so rather than merely seemingly learning features you can recover the actual feature space.

It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. This is somewhat counterintuitive when for the equivalent gradient descent only the end result is seemingly important.

> It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result. This is somewhat counterintuitive when for the equivalent gradient descent only the end result is seemingly important.

I don’t understand. What do you mean here?

I only (quickly) skimmed the article but it says that the limit is equal to a (not the) zero of your gradient.

The result should be the same for every trajectory leading to the same zero. Put another way, it doesn’t depend on the path if it leads to the same zero.

I would say here that not all zeros are equal though. :)

> It's also interesting how the result heavily depends on the path taking during the gradient descent rather than merely the end result.

After reading just the linked article, not the original paper, my interpretation is that the kernel isn't calculated during the gradient descent process, but from the final model after training by gradient descent. If that's right, it's not path-dependent.

No, the kernel is a mixing of gradients from the training. The proof depends completely on the particular form of optimization, in this case gradient descent in the limit of infinitesimal "timestepping", and suggests that gradient descent optimization doesn't cleverly pick good internal representations of the data.

> suggests that gradient descent optimization doesn't cleverly pick good internal representations of the data.

How so? The existence of equivalent kernels doesn't imply they aren't clever or good.

The final kernel is an integral taken over the path of the parameters during the gradient descent.

Basically what they're doing is just y = y0 + \int dy/dt dt and then rephrasing this equation as a kernel machine. Which I guess should also work for other forms of learning where you slowly change the parameters.

Well, we have the sufficiently magical kernel function from the gradient descent process -- seems like we could save a lot of computation by extracting out that function and running it as an SVM rather than holding it independent.

Interesting idea! Is there any research in this direction?

>With a sufficiently magical kernel function, indeed you can get great results with a kernel machine. But it's not so easy to write a kernel function for a domain like image processing, where shifts, scales, and small rotations shouldn't affect similarity much. Let alone for text processing, where it should recognize 2 sentences with similar meaning as similar.

I think the key issue at hand is that gradient descent is easier to train than a model using kernel functions. Someone could absolutely devise a mechanism for back propagation of errors with kernel functions, but at that point, it is basically a neural network.

Wouldn't that (similarity matters) be very useful information to encode in the NN though? Kinda like how you can do non-convolutional deep-learning on images, but it takes exponentially more training data than convolutional deep-learning does.

> If you're surprised by this result because you've used kernel machines and didn't find them very good at generalization,

This statement is a bit strange, because most of the theoretical work around NTK assumes that the tangent kernel doesn't change sufficiently during training. This seemed to make this closer to "extreme machines" than deep learning.

There was a paper from Francis Bach's group which seemed to be saying as much, but that criticism was no match for the meme-fication of this line of research by over-enthusiastic amateurs.

I may or may not be surprised by the result, I'm definitely not surprised by the 'Thing A is thing B' in machine learning.

Every tom, machine-learner, and harry, is an expert on proving to the whole world that thing-A is thing-B. The only problem is people don't hire them with a million dollars a year total compensation.

But the converse is not true in this case, so still interesting.

is an expert == is an "expert"

Reply on news.ycombinator.com