Don't Cite the No Free Lunch Theorem | Hacker News Reader

Don't Cite the No Free Lunch Theorem

peekaboo-vision.blogspot.com

61 pointsanother15 comments

Good! This article is about the NFL for machine learning (Wolpert), and also mentions NFL for optimisation (Wolpert and Macready). My recent paper [1] is about NFL for optimisation and mentions NFL for ML. The overall message is the same: most people misunderstand it, and you probably shouldn't cite it. And the anthropic principle is a sufficient assumption for this.

[1] https://arxiv.org/abs/1906.03280

If the NFL doesn't apply in the real world, then where is the one algorithm to rule them all?

There probably isn't one! But as argued in the paper, our reasons for thinking so are not based on NFL.

GLORIOUS VARIATIONAL BAYES

The first thing that came to my mind was that Milton Friedman quote "there's no such free meal, someone has to pay it".

My first thought was that this headline was referring to the idea in economics too. Today I learned that there's a "No Free Lunch" idea in ML.

> Milton Friedman quote

This meme was actually popularized by Robert Heinlein in The Moon Is a Harsh Mistress, ten years earlier than Milton Friedman started using it.

I learnt it from Milton, but it's nice to still discover other people figured out this.

Even though the NFL only holds absolutely when problems are closed under permutation, that doesn't imply that most problems are a good fit for a particular algorithm: https://arxiv.org/pdf/1609.08913.pdf

>What it actually (vaguely) says is “You can’t learn from data without making assumptions”.

That's what they taught us it means in our graduate machine learning class. What were people citing it to mean?

This is pretty much equivalent to 'you can't learn from random noise', isn't it?

It's a little more like, "If you assume nothing is more complicated or costly than anything else, then any set of examples you see could just as easily be generated adversarially to trick you as not."

That's also what they taught us in casual reading and HN, even before ML when it was a theorem about search algorithms and compression. I don't know where the author got their basis for being contrarian.

TANSTAAFL?

I think you're asking if this is about "there ain't no such thing as a free launch." It isn't, I thought it was at first myself, coming from an Economics background. It's about a machine learning theorm.

Reply on news.ycombinator.com