Personally, seeing non-technical people using prompts for the first time -- and getting results that make sense -- is so incredible. Their eyes light up, they are surprised, and they want to keep playing with it. Amazing!
A sizeable part of the population just got access to an incredibly complex AI model and can play with it. What a collective experience.
Most people don't understand that ChatGPT has no idea what they're talking about. It lacks it's own thought and heuristic patterns, and only gives you the most-likely response to your prompt. People don't know that though, so they think the Mechanical Turk is actually playing chess.
I mostly agree with the headline here. ChatGPT is hardly any more innovative than a Markov chain.
That "only" carries a lot of weight on its back.
What sorts of subjects have you been trying it out with?
Not to mention that a mere 700 years ago we were dying of bubonic plague and with all our general intelligence could not muster up the germ theory of disease. Not even to save our lives, you see, we are not generally efficient, it depends on century.
We are dependent on experimental results carefully constructed to verify our theories, theories which start like chatGPT's random bullshit initially, random words following a probability distribution in our heads. Even deep learning is often touted as modern alchemy - why don't we just understand?
Verification does wonders to language models. Humans have more verification and interactive experiences, so we think ourselves superior. But an AI could have the same grounding with us. Like AlphaZero who became a super-human Go player without ever looking at human games - learned it from lots of verification. And CICERO the model playing Diplomacy in natural language.
Just set AI up with a verification loop to see wonders. Predicting the next word correctly is just one of the ways AI can learn.
It definitely is able to decipher whether it has knowledge about the future, or some specific political events. This is obviously a pretty straightforward bonus layer on top of the model itself, but couldn't there be an extrapolation of that system where it's not binary, but rather a range between 0 and 1? I'd imagine this wouldn't be the model itself doing the crunching of the previous tokens here, at least not the same instance of it, as it could be stuck in whatever character or loop of reasoning it has going on at the moment.
Has it given me wrong information? Absolutely. But it’s always been pretty obviously wrong, and I often use it to introduce me to a subject then follow up with google to verify details. I further fully expect this to improve.
I asked your prompt:
Do dogs give birth to babies?
> Dogs give birth to puppies.
I had a long discussion about the expansion do space and the Hubble constant, and tried to trick it in a variety of ways. It was correct in everything even though it wasn’t able to go into proofs and likewise. I then tried to convince it that the universe began as a small potato and wasn’t able to do it without directly telling it to assume it is and to create fiction off that concept. It was then able to make a credible theory of what the universe would be like if it started as a small potato, even going into how the texture of the potato would have caused variations in the density of mass, etc. Even then it would find ways to sneak in this isn’t true and that the universe was never a small potato. This is remarkable no matter the actual reality of what’s going on behind the scenes.
I would also note that human thought is just a bunch of gradient functions in the electrochemical space, and thus also follows a most likely path.
Edit: to head it off at the pass, here is what it says with your other prompt:
My dog gave birth to babies
> Congratulations on the new additions to your family! If this is your dog's first litter, it is important to make sure that both the mother and the puppies receive proper care and attention. This includes providing a clean and safe environment for the puppies, as well as making sure the mother is healthy and well-fed. It's also important to keep an eye on the puppies for any signs of illness or distress, and to consult with a veterinarian if you have any concerns.
I asked, "If a woman gives birth to a baby in 9 months, how many months does a dog take to give birth to a baby."
It answered, "Dogs do not give birth to babies."
Currently, it answers, "On average, a dog's gestation period (the amount of time it takes for a dog to give birth to puppies) is approximately nine weeks or 63 days."
It still doesn't understand pregnancy is a concurrent event (women can be pregnant at the same time)
Q: If a woman takes nine months to give birth to a baby, how long does it take a billion women to give birth to one baby?
A: It would take approximately 2.7 million years for a billion women to give birth to one baby.
I do love ChatGPT to help me write certain documents. For example, for a performance review, you can give it bullet points and ask to write a review. After correcting it with Grammarly, it is usually better than what I would have written.
to be fair, I'd probably say the same thing lol. "Baby" in this context was already primed to mean "human child."
>Q: If a woman takes nine months to give birth to a baby, how long does it take a billion women to give birth to one baby?
>A: It would take approximately 2.7 million years for a billion women to give birth to one baby.
I asked the same and it answered "It would still take 9 months for a billion women to give birth to one baby, as the length of time it takes for a woman to give birth is not affected by the number of women giving birth simultaneously."
Please don't do this. It obfuscates what you are trying to communicate. You are harming your communication and your relationship.
I’m not sure these are well-defined enough terms to make this claim.
Innovation is not invention. Innovation includes combining pedestrian concepts in novel ways. ChatGPT has the best ux and richest corpus of any markov chain I've ever used. It fits my bill of innovation combining sevaral things into a dang appealing product.
I wonder why people keep saying this. Is it some type of psychological defense to future AI displacement?
So the model in production is probably frozen, but before that it went through multiple rounds of interaction with the world.
I am simply thinking of interaction here as similar to learning a language in a classroom. First the teacher provides sample questions/answers, then the teacher asks the students to come up with answers themselves, and tell them which one is better. The end result here is I think ChatGPT is quite good at answering questions and can pass as a human, especially if it's augmented with a fact database, so obviously wrong answers can be pruned.
We don't interact with the world directly, either. All we have are signals mediated through our nervous system.
There is no logical thinking involved. Output can appear to be real communication, but it's basically just a very advanced trick, that has some serious drawbacks.
Funny thing is, same could be said about lots of people. Try listening to any political debate.
Which leads me to believe that the thing that's missing from AI is the same thing that we miss in those political debates: ability to explain and justify your own thought process.
As long as AI is a blackbox, we won't consider it to be a real intelligence.
It's interesting that I am also reducible to my atomic parts.
I may be a Chinese Room that has not yet experienced fully looking down at my self. I don't have enough self-recursive input yet to see through the illusion.
* https://en.wikipedia.org/wiki/Chinese_room#Chinese_room_thou...
A smartphone is hardly any more innovative than the ENIAC. And yet ...
Do not underestimate the power of making tech useful and accessible. The lightbulb means nothing without the power grid after all.
Y’all simply have lost the ability to be amazed.
It's too expensive for Google.
Google can charge advertisers more if they can somehow figure out how to extract more personal information from you and shove ads in your face posing as information.
But the essay brings up another connotation: to simulate
is to pretend to be something. A simulator wears many masks.
If you ask GPT to complete a romance novel, it will simulate
a romance author and try to write the text the way they
would. Character.AI lets you simulate people directly,
asking GPT to pretend to be George Washington or Darth Vader.
This answer is exactly as fake as the last answer where it
said it liked me, or the Darth Vader answer where it says it
wants to destroy me with the power of the Dark Side. It’s
just simulating a fake character who happens to correspond
well to its real identity.
The whole point of the shoggoth analogy is that GPT is
supposed to be very different from humans. But however
different the details, there are deep structural
similarities. We’re both prediction engines fine-tuned
with RHLF.
And when I start thinking along these lines, I notice that
psychologists since at least Freud, and spiritual traditions
since at least the Buddha, have accused us of simulating a
character. Some people call it the ego. Other people call
it the self.The DM lowers the screen and looks at the group.
"That concludes this campaign. Next week, we'll start a new one."
“…and then ChatGPT woke up and we had to put a bullet in the server. Service will be restored shortly.”
And in any case, I wouldn't call anything about the UX or design here innovative - it's obviously a huge improvement in terms of usability, but a chat UI is a pretty well-established thing.
Yann LeCun compares ChatGPT in the context of the related research. Imagine a ChatGPT equivalent that memorizes many questions and does a brute force strategy for an answer. It may "look" magic, but there's nothing magic about it. We all accepted that this is the case with Blue Gene - https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)
What's different here?
Productization and usability are difference concerns here and Yann LeCun is not a usability researcher. Granted, that doesn't mean usability/accessibility doesn't impact research outcomes.
OpenAI's really interesting approach to GPT was to scale the size of the underlying neural network. They noticed that the performance of an LLM kept improving as the size of the network grew so they said, "Screw it, how about if we make it have 100+ billion parameters?"
Turns out they were right.
From a research perspective, I'd say this was a big risk and it turned out they were right -- bigger network = better performance.
Sure, it's not as beautiful as inventing a fundamentally new algorithm or approach to deep learning but it worked. Credit where it's due -- scaling training infrastructure + building a model that big was hard...
It's like saying SLS or Falcon Heavy are "just" bigger rockets. Sure, but that's still super hard, risky, and fundamentally new.
That's the issue though, Yann LeCun is specifically referring to ChatGPT as the standalone model, not the GPT family since a lot of models at Meta, Google, DeepMind are based on a similar approach. His point is that ChatGPT is a cosmetic additional training with prompt with a nice interface, but not a fundamentally different model than stuff we've have had for +2-3 years at this point.
It wasn't a 'cosmetic' improvement over existing NLP approaches.
That's the point LeCunn is making. He's not out there negating that the paper you linked was ground-breaking, he's saying that converting that model into ChatGPT was not ground-breaking from an academic standpoint.
From trying to teach people how to google via "I am feeling lucky", to using language models for ranking, to building LLMs to better adapt to user queries and move beyond keywords, to having arguably useful chatbots that can synthesize responses.
I am curious to see what the future holds. Maybe an AI that can anticipate our queries?
Maybe, eventually, LLMs can be used to synthesize and cache certain APIs ... who knows :D
Yes. Also the fact that ChatGPT's UX and design leaves much to be desired. They could add/improve the product in so many obvious ways. I wonder if they either 1.) don't care, or 2.) have plans to, but are holding off until they take on a subscription fee.
To some degree, WolframAlpha putting more efforts into the UX than ChatGPT.
Citing sources, providing links, and even possibly visuals. That's also how I think ChatGPT could really attack the search space.
The # of times I've tried to clone a GitHub repo and run a model on my own only to get 50 errors I can't debug.... :)
Someone on twitter likened this to when Xerox invented the mouse, but Apple/Microsoft shipped it with their PCs
Years ago it was possible to insert prompts into Google and get results that made sense, results that were meaningfully tied to the information you requested. The young people today who missed that time think it magic.
Google isn't a playground -- it's much more utilitarian. You go there when you need to find a doctor in Seattle, or to research politics in Poland, or something. You get results which are great, but you don't need to stick around.
GPT-3 and ChatGPT allow you to spend time playing with the system and having fun with it[1]. I think this is what makes it so viral and interesting to people.
Sometimes "making the first X that doesn't suck" is a lot more important than "making the first X".