Comment by matusp | Hacker News Reader

Comment by matusp | Hacker News Reader

What a great write-up, kudos to the author! I’ve been in the field since 2014, so this really feels like reliving my career. I think one paradigm shift that isn’t fully represented in the article is what we now call “genAI.” Sure, we had all kinds of language models (BERTs, word embeddings, etc.), but in the end, most people used them to build customized classifiers or regression models. Nobody was thinking about “solving” tasks by asking oracle-like models questions in natural language. That was considered completely impossible with our technology even in 2018/19. Some people studied language models, but that definitely wasn’t their primary use case; they were mainly used to support tasks like speech-to-text, grammar correction, or similar applications.

With GPT-3 and later ChatGPT, there was a very fundamental shift in how people think about approaching NLP problems. Many of the techniques and methods became outdated and you could suddenly do things that were not feasible before.

> Nobody was thinking about “solving” tasks by asking oracle-like models

I remember this being talked about maybe even earlier than 2018/2019, but the scale of models then was still off by at least one order of magnitude before it had a chance of working. It was the ridiculous scale of GPT that allowed the insight that scaling would make it useful.

(Tangentially related; I remember a research project/system from maybe 2010 or earlier that could respond to natural language queries. One of the demos was to ask for distance between cities. It was based on some sort of language parsing and knowledge graph/database, not deep-learning. Would be interesting to read about this again, if anyone remembers.)

Are you sure? I wrote an essay at the end of 2016 about the state of AI research and at the time researchers were demolishing benchmarks like FAIR's bAbI which involved generating answers to questions. I wrote back then about story comprehension and programming robots by giving them stories (we'd now call these prompts).

https://blog.plan99.net/the-science-of-westworld-ec624585e47

bAbI paper: https://arxiv.org/abs/1502.05698

Abstract: One long-term goal of machine learning research is to produce methods that are applicable to reasoning and natural language, in particular building an intelligent dialogue agent. To measure progress towards that goal, we argue for the usefulness of a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human.

So at least FAIR was thinking about making AI that you could ask questions of in natural language. Then they went and beat their own benchmark with the Memory Networks paper:

https://arxiv.org/pdf/1410.3916

Fred went to the kitchen. Fred picked up the milk. Fred travelled to the office.

Where is the milk ? A: office

Where does milk come from ? A: milk come from cow

What is a cow a type of ? A: cow be female of cattle

Where are cattle found ? A: cattle farm become widespread in brazil

What does milk taste like ? A: milk taste like milk

What does milk go well with ? A: milk go with coffee

Where was Fred before the office ? A: kitchen

That was published in 2015. So we can see quite early ChatGPT like capabilities, even though they're quite primitive still.

Reply on news.ycombinator.com