Hacker News Reader

I spoke w/ Marvin Minsky once back in the 1990s and he told me that he thought "emergent properties" were bunk.

He said the same thing about perceptrons in general. When it comes to bunk, Minsky was... let's just say he was a subject matter expert.

He caused a lot of people to waste a lot of time. I've got your XOR right here, Marvin...

Minsky is an example of the power of credentialism. He was an MIT professor who researched AI. I think history has pretty much demonstrated he didn't have any special insight into the question of AI, but for decades he was the most famous researcher in the field.

Hard to say. Perceptrons was an early mathematically rigorous book in CS that asked questions about what you could accomplish with a family of algorithms. That said, it is easy to imagine that neural networks could gave gained 5 years or more progress if people had had some good luck early on or taken them more seriously.

I doubt it would take very many suitable examples in the training dataset to fix problems like that.

I think this problem is mostly already solved with more capable models.

Here's the shoe-identification experiment with GPT-4 and it's not confused by a suit jacket:

https://imgur.com/a/cPP6pH7

It's also not confused by a tuxedo:

https://imgur.com/a/mWMxZbA

Also tried the same examples with Google Gemini 1.5 Pro and it doesn't hallucinate either.

https://imgur.com/a/3Yv7wuP

If it sees 90% of photos of basketball players wearing sneakers it is still going to get the idea that basketball players wear sneakers. I guess I could make up some synthetic data where I mix and match people's shoes but it's a strange way to get at the problem: train on some image set that has completely unrealistic statistics to try to make it pay attention to particular features as opposed to other ones.

Problems like that have been bugging me for a while, if you had some Bayesian model you could adjust the priors to make the statistics match a particular set of cases and it would be nice if you could do the same with a wide range of machine learning approaches. For instance you might find that 90% of the cases are "easy" cases that seem like a waste to include in the training data, keeping there gives the model the right idea about the statistics but may burn up the learning capacity of the model such that it can't really learn from the other 10%.

I talked w/ some contractors who made intelligence models for three-letter agencies and they told me all about ways for dealing with that come down to building multiple-staged models where you build a model that separates the hard cases from the easy cases with specialized training sets for each one. It's one of those things that some people have forgotten in the Research Triangle Park area but Silicon Valley never knew.

Your example is really simple to fix.

Just add another model asking it if there are shoes visible or not.

That's the right track but you have to go a little further.

You probably need to segment out the feet and then have a model that just looks at the feet. Just throwing out images without feet isn't going to tell the system that it is only supported to look at the feet. And even if you do that, there is also the inconvenient truth that a system that looks at everything else could still beat a "proper" model because there are going to be cases where the view of the feet is not so good and exploiting the bias is going to help.

This weekend I might find the time to mate my image sorter to a CLIP-then-classical-ML classifier and I'll see how good it does. I expect it to do really well on "indoors vs outdoors" but would not expect it to do well on shoes (other than by cheating) unless I put a lot of effort into something better.