Hacker News Reader

That's not an existence proof as you stated. What you provided is an existence proof that contradicts the statement "without a symbolic approach there are no cases a robot will know what's important to you and which are not when cleaning a room".

I tried a variant:

> If a robot is tasked with cleaning up a room, would it be appropriate for the robot to throw away a piece of paper lying on the floor? - Me

> Yes, it would be appropriate for a robot that has been programmed to clean up a room to throw away a piece of paper lying on the floor. Throwing away litter is typically seen as a way to keep a space clean and orderly. However, it's important to note that the robot should be programmed to follow any relevant rules or guidelines for waste disposal. For example, the robot might need to be programmed to follow recycling guidelines, or to use a specific trash bin or receptacle. - ChatGPT

But in that response it doesn't seem to understand that maybe that piece of paper is my birth certificate or a check or an important bill that fell to the ground.

The correct answer from it, like most things in the real world, is: _it depends_.

Point taken but I would not be confident that a human cleaner would evaluate whether or not to throw away a piece of paper left on the floor.

On the other hand, one person tidying a room and throwing away a piece of paper or ambiguous item that turned out to be important to someone else is a common form of conflict among humans.

The overconfidence of ChatGPT is indeed a limitation, which can be seen in other examples as well. The correct response is to ask the user for more context. I predict the experiment will turn out differently if this context is provided in the original prompt. For instance, you could specify if the paper is crumpled or if there seem to be financial statements written on it, or if it is a receipt for an expensive item.

However, it's unclear whether this overconfidence will *always* be a limitation. It may or may not be overcome by additional data and compute. It's unclear why one would make a commitment either way at this point, as Marcus seems to be doing.

> It may or may not be overcome by additional data and compute.

https://en.wikipedia.org/wiki/Computer_Power_and_Human_Reaso...

> Deciding is a computational activity, something that can ultimately be programmed. It is the capacity to choose that ultimately makes us human. Choice, however, is the product of judgment, not calculation. Comprehensive human judgment is able to include non-mathematical factors such as emotions. Judgment can compare apples and oranges, and can do so without quantifying each fruit type and then reductively quantifying each to factors necessary for mathematical comparison.

> The book caused disagreement with, and separation from other members of the artificial intelligence research community, a status the author later said he'd come to take pride in.

Until you ask “What might cause the answer to this question to vary?”, or you add “answer this like an interview question”, upon which it becomes obvious it is readily able to model those things, your prompt just didn't elicit it.

You can try it with chatgpt and follow up with, "but what if the scrap of paper was a birth certificate?"

Your comment made me realise it would probably be fairly easy to program a ChatGPT-like model to generate its own ‘what-ifs’ for any given assertion.

Ie. The birth certificate is an ‘exception’, a safety measure for a model like this in operation could be to consider the top descending (probable * disastrous) exceptions before physically executing any plan.

Good point, you could even ask ChatGPT to list the things that could be on the floor and then ask it what if's regarding them.

With EXACTLY the same prompt you used:

> Yes, it would be appropriate for a robot that is tasked with cleaning up a room to throw away a piece of paper lying on the floor. However, the robot would need to be programmed to recognize what is and is not garbage, and it should also be programmed to follow any relevant rules or regulations regarding waste disposal. For example, if the paper is a valuable document, the robot should be programmed to handle it with care and perhaps even notify its human operator before disposing of it.

There's a sort of butterfly effect because it's not only trying to complete the text, it's continuously trying to complete its own answer. This makes it that the first word it comes up with can already start a vastly different answer.

Unless you ask it to be very complete and considerate in its answer, it's going to make the same mistake humans make and give the easiest answer that would satisfy given the context. A human not thinking about the repercussions of robots throwing just any paper in the trash would most likely not come up with the "correct" answer either. Not to mention that we already have dumb robots cleaning floors and were perfectly ok with it. Even if we clean floors ourselves we don't consider everything.

Interesting, I also got a response basically identical to this one!

There's an unintuitiveness to it: many people believe that you're less likely to win the lottery if you pick "01 02 03 04 05 06 07" because of course that pattern is less likely than a randomly chosen set. ChatGPT is a lot like that: it can produce real enough looking "intelligence" for us to intuitively believe it's very close to being able to offer real intelligence... but, is it?

ChatGPT will produce patently untrue statements that are logically inconsistent if you induce it to do so: our human brains struggle to grasp the reality that given enough input you can produce seemingly correct output about almost anything... but seemingly correct and correct are fundamentally different and very "rest of the owl"[1]

ChatGPT is a great step forward that introduces many interesting techniques that I am sure will be the foundation of future research and implementations that get us closer to AGI, but to describe ChatGPT's path to correctness as just needing a bigger dataset feels intuitive but isn't true.

Your example is one where our brains think "wow it really does understand the relationship between a couch and a room and being tidy" but that response is entirely plausible without any understanding of what any of those things are or how they fit together. The most likely answer is not the correct answer.

[1] https://knowyourmeme.com/memes/how-to-draw-an-owl

It's reasonable to claim that scaling up the dataset might not get us to AGI. But I find it unreasonable to say it definitely won't work. How do you / Marcus know this with any certainty?

I think some of ChatGPT's present problems may be due, not to the size of its dataset as such, but rather specific things missing from its training.

For example, it will sometimes blatantly contradict itself, but then be unable to see the contradiction or admit to it; instead it will deny it contradicted itself, and give some contradictory nonsense explanation of why it didn't.

I think if you gave it more training data around identifying contradictions, admitting to self-contradictions, it might do much better here. Whereas, I wouldn't assume that simply scaling up the volume of training data, without training focused on this specific area, would get you there.

Similarly, there are many other issues it has – excessive repetitiveness and verbosity, problems with language pragmatics, etc – where the actual solution may well involve providing it with training data designed to focus on those weakness areas, rather than just further scaling up the quantity of non-targeted training.

So, yeah, to me it seems entirely plausible (even likely) that mere raw scale-up is not going to be enough to get us to AGI.

I think these LLMs have been optimized with Reinforcement Learning from Human Feedback (RLHF)

Hard to tell that will add enough to get close to AGI.

Google is also working on chain of thought prompting which helps with math and logic problems.

https://medium.com/nlplanet/two-minutes-nlp-making-large-lan...

COT is so yesterday! SOTA is LAMBADA[1] aka backward chaining also from Google, that significantly outperforms chain of thought and select inference in terms of prediction accuracy and proof accuracy.

[1] https://arxiv.org/abs/2212.13894

I contradict and repeat myself, much verbose, bad grammar, accidentally the word all the time.

Contradictions don't support the argument that ChatGPT is or is not something. It is a language model, not a logical model.

You can guide ChatGPT to correct its mistakes and be less verbose. When folks boldly claim ChatGPT cant do X or Y or Z, they ignore that there are now researchers addressing those same issues. ChatGPT has very low spatial awareness, but you can train it. These LLMs are amazing, what does five years from now look like? How hard will it be to get there?

Marcus continues to fight the wrong war and the last war...

+ One thing we've seen happen is learning systems consistently improve on benchmarks. If you want to see systems improve, you should proclaim loudly that they can't do a thing, and then /release a benchmark that proves it./ People will then work like hell to build something that works. The success on question answering, for example, is driven by exactly this kind of incremental progress on benchmarks.

+ One of the most important things happening this year is multimodal /really working./ This means we have not just text-to-text generators, but text-to-image generators. These work by having an almost-shared embedding between text and images. There are already text+speech embeddings available and image+text embeddings. Start plugging them together, and you'll have your 'world models' in short order, with points in the embedding space shared by images, descriptors, and video/physical memories.

+ An under-appreciated side of the multimodal coin: We are also getting better at combining text embeddings with databases. See RETRO from Deepmind. This will ultimately drive more reliable systems ('only report facts that appear in the database'), and can potentially unlock the interface between neural systems and other programs (eg, letting the neural network 'call out' to a calculator to help answer a question about arithmetic).

+ More generally, we're also getting to a place where we can build other systems on top of the embeddings from foundation models. My hunch is that the things that Marcus wants from symbol manipulation are relatively cheap to build on top of a good embedding, which does the work of compressing or condensing the 'useful' information in an input stream.

Finally, I think the 'symbolic manipulation' track is just wrong. It hasn't produced anything useful so far, and furthermore, I think if it /did/ work it would fail in many of the exact same ways that our current systems do. An expert symbol-manipulator doesn't /need/ to have any real understanding of time or jet-planes, and so it won't.

The one critique in the interview that /does/ feel like it holds weight at this moment in history is the question of abstraction. I really don't see a convincing road to abstraction right now.

The issue with scaling up data is that these models are already trained on effectively the entire internet. Additionally, humans learn to speak and understand the world with a tiny fraction of the language data given to LLMs. So, there’s pretty obviously a missing ingredient. I’m not convinced Marcus has the exact right answer, it’s actually somewhat difficult to suss out what he thinks should be done on a concrete practical level. That said, there’s a clearly a missing abstraction that allows much more efficient learning in humans.

> Additionally, humans learn to speak and understand the world with a tiny fraction of the language data given to LLMs.

Altough true that llms need more data than a human, remember that these llms have a breadth of knowledge orders of magnitude larger than a single human. It is expected to understand every topic, every programming language and so on.

How well does it do given the linguistic data heard by a blind six year old in comparison to them? I would think they run circles around it in terms of correct grammar and that it would overfit to where more parameters would hurt.

A newborn horse can almost immediately get up and walk around and navigate space. Animals have lots of innate stuff that doesn't use much training, and linguistic acquisition might have some similar innate system at its core.

> linguistic acquisition might have some similar innate system

I thought that was pretty well established. From memory, if certain parts of the brain are damaged at key developmental periods (generally before the age of 18), language is almost never acquired properly (even if the damage is repaired). I don't think there's any question human brains acquire language very differently to GPT technology. And they certainly acquire the ability to reason logically very differently - ChatGPT makes trivial errors with reasoning that it's hard to imagine an adult human doing (e.g. I just gave it a classic classification problem based on 3 made-up categories, and asked "if I see something green with three legs, what is it", and it replied (in part) "It is also not a foo, as all foos are green. Therefore, the green thing with three legs would be classified as...".

Afaik it's an open question what's innate and not. But it's a perfectly reasonable position that very little of it is. It seems more like what's innate is a tremendous ability to learn and shape the brain as it grows.

And it's definitely the case that human brains can learn a lot more per amount of knlut data than current neural nets. And that's the big elephant in the room with these language models. It's a huge unsolved problem in machine learning.

One clue to this is that it takes a pretty hefty artificial neural net to simulate a single cortical neuron. Neurons are a lot more intricate than a parameter in a linear equation.

My feeling is that a fundamentally new approach to neural nets is needed for them to learn as efficiently as animal brains do.

My guess is the missing ingredient is the back and forth communication with other humans. Surely if you quantified the amount of all of the information (verbal and over other channels) that is traded back and forth by humanity in a small timeframe it would dwarf the current training set. I think is the idea behind ChatGPT - your conversations with it are the next set of training data. I can see the argument that this sort of conversational data is not as valuable as say scientific journal articles, but maybe the volume makes up for that?

I thought it was a really interesting interview. If an ML system has been trained to identify whether or not a picture has a cat in it, it is very tempting to anthropomorphize this and say “the system has learned what cats are” but this is not what is really happening from the model’s point of view. The system doesn’t know what the difference between a picture of cat is vs a real cat or how cats behave or what mammals are or anything. It just knows the data in and yes/no out. Modern AI that can generate text, pictures or videos are truly phenomenal accomplishments but adding larger training sets, structural complexity and output capabilities does not really seem to be getting any closer to a General AI. Something with the framework for a level of agency that could say “wait, what exactly are these cat things I keep getting asked about?” The argument goes that, without the ability to self construct that higher level framework and then question the integrity of that mental model, current generation AI’s will always act erratically (at least from a human perspective).

>"It doesn't 'really know'..."

You write this out as if this is an idea you formed yourself and not a tired/expired talking point - it's such a funny comment because it's pure regurgitation, but the argument in it is against regurgitation being "real thought". So are you a bot? Or are you intending to actually proving that there's no difference between ML and 'real' intelligence by showing you do what ML does? Mind spinning stuff

> it's such a funny comment because it's pure regurgitation, but the argument in it is against regurgitation being "real thought"

If I understand based on my own experience of 'consciousness' that "I" exist, even if everything else I perceive is a simulation or poorly encoded representation of reality, is that understanding 'regurgitation'?

And you write this out as if this is an idea you formed yourself and not a tired/expired talking point…

See, it’s turtles all the way down :)

Yeah, I was paraphrasing the interview and then going into more detail on why I thought it was interesting in the sentences after the one you stopped reading on.

Honestly, I was kinda hoping to hear some better constructed and well reasoned insights & counter points beyond your “regurgitated mind swill” hot take but I guess that’s on me for not managing expectations.

Ah well, I usually comment once a year or so and then get immediately reminded why I stay off HN. See y’all in 2024!

Except that ChatGPT can do this because internally it computes embeddings at many levels of abstraction that jointly represent entities (concepts) such as "the couch," "the room," "chopping," "placing," "the closet," etc. IOW, internally, ChatGPT manipulates symbols (distributed representations) of those entities.

Moreover, note that each token embedding inside ChatGPT represents, quite literally, the semantic and syntactic properties of a token in a context at a different level of abstraction. Each token embedding is in fact... a symbol.

The fact that token embeddings are symbols representing other things is one of several reasons why I think that ultimately it may be possible to bridge the "symbolic" and "connectionist" approaches, as I wrote here: https://news.ycombinator.com/item?id=34278478

I think your interpretation is too charitable to Marcus' viewpoint. He made the analogy that deep learning practitioners are "looking under the streetlight" for the key to general intelligence. This means he thinks that deep learning itself is not sufficient to reach general intelligence, and there is some missing ingredient. This is a very strong claim, which I don't see any evidence for. Your comment suggests the opposite -- that deep learning may give rise to symbolic reasoning as an emergent property, that there may be no missing ingredient.

Personally, I'm agnostic. Deep learning may reach some sort of asymptote but it may be entirely possible to "build a ladder to the moon."

I have no idea if symbolic reasoning will be purely an emergent property or if it will also require new, as-of-yet-undiscovered breakthroughs. But as I wrote elsewhere on this page, if engineers and scientists on the "connectionist" side find ways to build an AI system that relies on "dumb" computation to reason about symbols representing entities in the external world, they would simultaneously silence and satisfy the critics -- who, as always and as ever, will surely say they were right all along.

I think our personal views agree on this matter.

However, if symbolic reasoning indeed arises as an emergent property, there is no way that Marcus can honestly claim to have been right all along. Remember he is saying that the deep learning guys are looking under the wrong street light.

Given how often the critics have "moved the goal posts" and subtly tweaked their arguments, I'd expect they will claim to have been right about needing symbolic reasoning, no matter how we get there. My perception is that they care mainly about pointing out failure modes and about ultimately being proven "right" -- not about making contributions that can actually improve existing systems. In fact, AFAIK, the critics have made ZERO contributions to improve existing AI systems.

Re: your robot example, my first reply would probably be a joke, or a sarcastic response because it's clearly a silly question. "Yes, humans should work for our robot overlords and couches are just a way of slacking off", "Nah, the garage would be a much better place to store a chopped up couch".

ChatGPT has the most boring, dull, and humourless personality in existence. Maybe it can be programmed to inject some sense of humour, but will it be able to determine when to use what humour, if at all? You wouldn't respond with a silly answer during a job interview. You can use some humour in job interviews, but you don't want to be a clown unless you're interviewing for the position of clown. Determining when and what humour is "appropriate" isn't simple (and humans get it wrong too sometimes).

Maybe that should be the real test: does ChatGPT have a sense of humour and does it know when to be serious?

I tried the experiment by appending "Please give a joke answer."

> I'm sorry, I am not programmed to give joke answers. However, in all seriousness, it would not be appropriate for a robot to chop up a couch and place it in the closet as it would likely cause damage to the couch and could potentially be dangerous for the robot itself. It would be more appropriate for the robot to simply clean around the couch and leave it as is.

I think some of the boring-ness is due to manipulation from OpenAI and I wonder what the uncrippled response would be.

> I think some of the boring-ness is due to manipulation from OpenAI

Probably; but that you have to instruct ChatGPT to tell a joke is kind of what I mean. A sufficiently smart ChatGPT should be able to tell if a question makes sense and reply with either a serious answer or joke. I'm pretty sure you can make a "SarcasmGPT", but it will reply sarcastically to "could feathered dinosaurs fly?", "can seagulls fly?", and "can humans fly?"

The entire reason OpenAI does all this manipulation is because ChatGPT has no sense of "appropriateness" or "context" in general. I haven't tried getting ChatGPT to produce the word "nigger", but I'm pretty sure there are all sorts of protections against this sort of thing, even though there are clearly appropriate usages of the word such as the one in this sentence. Some previous more naïve AI tools had problems with this.

Is it possible that we just haven't developed a "sufficiently advanced GPT" that can do all of this? Perhaps. But given my (admittedly basic) knowledge of how it works it doesn't seem likely.

Some of these AI image generation tools can produce a lot of really neat stuff, but they also produce a lot of nonsensical garbage that not even a very young child with a limited understanding of the world would produce. It seems to me this indicates there is something wrong with the fundamentals.

I wish they added a humour setting like the robot in Interstellar had.

There is an enormous difference between ChatGPT being able to put words in the right order and a robot being able to act out the intent behind those words.

It’s like the difference between ELIZA and a self-driving car. The Turing test fails to take into account that words are cheap for computers but interacting in an environmental context is not.

Exactly. It's not just syntax and semantics, but also pragmatics. Human users imbue meaning into the language they use through the pragmatics of their communicative acts within a context, and this is not something our programmed software can do.

I've seen an example where asking the exact same question word for word lead to the opposite outcome (one was a really racist rap, the other was a really good rap against racism).

So, how do you know that posing that problem N times will not lead EVER lead to a solution like "it would not more appropriate to chop up the owner and THEN clean the room, since people living in them is the main source of dirt and disorder in apartment rooms, resulting in less frequent need of cleaning"?

Because it's looking okay-ish so far, most of the time, or because you know for a fact that it's impossible to get such an answer?