Hacker News Reader

You're not making sense. If the loss function is convex, adding L1 regularization is still convex. So iterative methods for convex problems (which include SVMs, linear regression, and logistic regression) will find the global optimum.

View on HN

1) Neural Network Loss functions are not convex. But that isn't the issue here.

2) When you use actual numerical optimization techniques with floating point arithmetic, you don't find an exact minimum (global or local). And you don't get exact zeros.

Have you tried this on a real problem? I wouldn't consider MNIST a real problem, but even there you will not get _exact_ zeros. Try it.

If you rolled your own naive numerical approximation of L1 regularization, you might not have gotten exact zeros. If you use e.g. LARS or cyclical coordinate descent for the L1-regularized parameter cohort, as suited to the problem, you will get exact zeros, as prescribed by the mathematics of L1.

I've never seen anyone optimize a neural network using LARS or cyclical coordinate descent. I thought that's what this entire discussion was about - not arbitrary optimization theory.

Reply on news.ycombinator.com