An "AI Breakthrough" on Systematic Generalization in Language?

aiguide.substack.com

47 pointspicometer4 comments

Very interesting write up and analysis of a significant paper. The blog's author has herself made fascinating contributions in AI (especially on abstraction). I find it very helpful to read these kinds of in depth reviews.

1) https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

2) Study used 25 people

3) via Mechanical Turk

Interesting topic but I don't see a breakthrough.

Why would it matter? They wanted data generated for comparison and got it. They wanted a common type of human error and got it too. What's the criticism exactly?

Yes, they got some data, and got some errors. But they are using 25 samples to characterize "human reasoning". Sure, it's a place to start.

The bigger issue is that this pattern - small samples, via Mechanical Turk - is a frequent flier in papers that make claims that end up failing further scrutiny. It's more common in sociology and psychology than AI research, but I think we know this isn't a solid foundation to build a lot of extrapolation on.

Reply on news.ycombinator.com