Hacker News Reader

> As I see it, there is clearly no way to advance ChatGPT into anything more than it is today. Impressive as it is, the curtain is wide open for all to see, and the art can be viewed plainly as what it truly is: magic, and nothing more.

(I only RTFA after writing this comment, and I now see that the below is what they're doing)

I'm an outsider to this field. My unexpert thought was that perhaps this model could be used to identify the maths components of that question and then we (programmatically) feed that into a different system that gives the answer. That answer then could be provided as context when asking the real question, so the model has access to the mathematical facts.

E.g., I just put this question into the playground:

  "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result.  What is the result?"

  Identify the mathematical question in the preceding question and write it as an equation.

The answer I got back was:

  234241.24211 + 58352342.52544 = ?

That could then be fed into a different system that is designed to do mathematical calculations (or logic). In short, not by doing more of what has been done so far, but instead combining these models with different systems.

We take the result of that sum from a separate system, and redo the same question, providing the mathematical details as context. E.g., now asking:

  234241.24211 + 58352342.52544 = 58586583.76755

  Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result.  What is the result?

With the response:

  The result is 58586583.76755.

Thus getting mathematically accurate answers. Note that if I don't include this context with the calculation completed, I get back the answer:

  6,179,563.76754

You are asking ChatGPT to interface with a calculator. This is sort of what Wolfram proposes too: interface ChatGPT with a math language. It could work for simple problems. But extracting a mathematical description from a piece of text can require logic too, I suppose, which is where the difficulties start.

You would be trading the problem for another instance of the same problem.

When you ask ChatGPT to construct a mathematical question, it will do so the same way it does everything else: by semantic popularity.

And that is the problem we are trying to avoid. The semantically popular guess might be logically sound, but it might not. It's a gamble no matter when or where it is done.

--

All it takes is what I call a "semantic off-by-one error". That might look like our first problem:

> Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?

The problem is that a semantically close response is nothing like "234241.24211 + 58352342.52544". It's just going to be whatever arbitrary text that already exists in the training dataset is closest to the semantic phrasing of the question. That might be the correct number, is more likely to be an incorrect number, and is even likely to be a wordy response.

--

So if we follow your thought process to interrupt the guesswork, it would involve restructuring the prompt.

> Please restructure the following prompt into a mathematical statement: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?"

The output you are hoping for

>> 234241.24211 + 58352342.52544

Another completely valid and possible output:

>> 234241.24211 / 58352342.52544

Another:

>> 2424332.34434 + 535858932.5358

--

There is no way to guarantee the reformulated question is logically equivalent to its original. That's the problem, and the problem cannot be moved. With every step, ChatGPT is guessing. ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is.

The only place you can pause an interaction with ChatGPT to do some logic is instead.

> It's just going to be whatever arbitrary text that already exists in the training dataset is closest to the semantic phrasing of the question.

My understanding of machine learning in general is that this is not how it works, and rather it uses a neural network for a lot of what it does (which isn't merely picking the closest arbitrary text in its training set), though I don't know details about NLP specifically. I am aware that some activities involve ranking the similarity of particular texts, and picking the closest matches (that's how embeddings get used for providing context to prompts), but I didn't think that's how the underlying models operate when, say, asked the question I asked it.

Regarding the rest of your remarks, the aim isn't necessarily perfection. You listed some other possible outputs, but the fact is that it didn't offer these options, and instead it offered the correct one, and it seems that a system that uses ChatGPT + something else can provide an improvement over ChatGPT on its own (with the downside being that it requires more resources to provide those answers).

Is the other "completely valid and possible output" you provided actually likely? Even a human could possibly make the mistakes you list, but possible doesn't mean probable, and being possible doesn't take away from the results of such a hybrid system being an improvement over ChatGPT on its own.

I didn't think guarantees are the game here. Achieving something closer to what humans are (and even some humans struggle with logic and mathematics), and even exceeding it in some places, would be of immense value.

> ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is

Is this different to what humans do? There are accounts of the brain where it takes sensory input, updates an internal model, and makes predictions (guesses) about what inputs it expects to receive. Getting better at guessing might be the right game to play. i.e., it's not a criticism to say it's guessing, because that's what these models are trying to do. If it can guess reasonably well when a question is mathematical or logical, and can guess reasonably well about how to structure a mathematical or logical equation or statement based on that text, then that could be fed into another system to produce a more accurate answer.

> but the fact is that it didn't offer these options, and instead it offered the correct one

...because you gave it a "correct" prompt; or in other words, you gave it a prompt that ChatGPT can respond "successfully" to. But if you keep trying, you will surely find yourself with "failed" responses. And ChatGPT has no way of knowing the difference. That's my point.

> Is the other "completely valid and possible output" you provided actually likely?

We're splitting hairs here. I didn't go and test a bunch of prompts to find an example failure. You are welcome to do that if you want. It's not very hard, especially when you start adding variables. This is well-known behavior that ChatGPT itself warns its users about.

> > ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is

> Is this different to what humans do?

Yes!

We don't just spit out the first thing that comes to mind, unless that's the task we are intending to do - and anyone who has played music can tell you just how difficult that task is.

Most of the time, we conceptualize. We comprehend the meaning of language - sure, it takes some guesswork along the way, but that's not the main event - then we construct a new idea, then we express it.

We don't just want accurate answers. We want conceptualization. That's the feature we are all dreaming about with Natural Language Processors. If we can teach a computer to conceptualize language, then we can get it to interface with any arbitrary human knowledge!

But ChatGPT doesn't interface. It parrots; and that's an insult to the birds themselves - unlike ChatGPT, they have an intelligence of their own.

The only way ChatGPT could give you a valid response about any arbitrary concept is to write every thing from every angle about every concept ever, and feed it all to ChatGPT as training data. That's exactly the work we want to avoid!

Ok, I think this has moved on from the original point. I quoted you saying: "As I see it, there is clearly no way to advance ChatGPT into anything more than it is today. Impressive as it is, the curtain is wide open for all to see, and the art can be viewed plainly as what it truly is: magic, and nothing more."

I was disagreeing with this, and showing ways it can advance. Your reply focuses on the fact that it will still get things wrong. However, I never claimed that it wouldn't make mistakes. I agree it will, and frequently. I just wanted to show that it can do more, and valuably and recognisably more.

The next step up for ChatGPT isn't perfection of mathematics and logic. And with the examples we've been playing with, experimentations with prompts, and further training, when combined with other systems, I don't see why better couldn't be done, and I gave one such example. Sometimes the output it gives is what we're after, sometimes it isn't. Sometimes, we'd be able to feed its own input back to itself and ask it to evaluate if the previous output it gave is what we're after. E.g., I recall hearing about someone feeding the output to the Rust compiler, then feed error messages back, in an iterative loop until they get something that at least compiles. I don't know the best techniques yet, but I have no doubt that a variety of techniques combined with other systems could improve the output of ChatGPT, significantly.

Regarding fair prompts, here's another example of a prompt and its reply, trying at a more generic prompt:

  Provide a list of mathematical questions contained in the following text, phrased as equations: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? An answer was given by Suzie, who had recently had to count how many new potatoes she had in her pile, after putting the seven new ones in the pile of 15 she already had."
  
  1. 234241.24211 + 58352342.52544 = ___________ 
  2. 15 + 7 = ___________

Or maybe we can ask it to do something like a school assignment:

  Assuming the following text is given to students, create a list of mathematical equations that could serve as questions based on this text: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? An answer was given by Suzie, who had recently had to count how many new potatoes she had in her pile, after putting the seven new ones in the pile of 15 she already had."

  1. 234241.24211 + 58352342.52544 = __?
  2. 7 + 15 = __?

Doesn't matter if it fails sometimes, but it can do a lot better.

You might be right about your claims about conceptualising, and seemingly focusing on perfection as the next step. Perfection isn't the next step in this journey, because there's a lot more value to be added before trying to hit anything remotely close to that (if we even can!). (here, I'm referring to your comment: "I don't see any way to seed such a system to always produce logically correct results")

Side note: I think that humans (and probably many other animals) are conscious, and have a non-physical mind, while computers never will, or at least won't in virtue of us improving the physical characteristics of these systems. But that being said, I think there's a lot about the way human brains work that can be mimicked by machines, and I think that human brains do an awful lot of guessing in our day to day moving throughout the world. That doesn't strike me as different to ChatGPT. It guesses a lot of things, and so do we.

> when combined with other systems

That's my whole point. It can't. There is no room for the other system. Any other system will always be either too late in the process to work, or a complete replacement.

> Sometimes, we'd be able to feed its own input back to itself and ask it to evaluate if the previous output it gave is what we're after

And that doesn't accomplish what "evaluate" actually means. It only does the same thing it always does (find something semantically close to "evaluate the thing"), but with a new prompt. If you are lucky, the new prompt will be responded to with output you like, but only if you are lucky. If we keep doing this process, we are effectively guessing what password will match a hash. That's called "brute forcing", and the fact it doesn't work is why your passwords are secure.

> Doesn't matter if it fails sometimes

It fails every time. Sometimes it "fails up". Stumbling into success is still stumbling.

If I write a program that outputs random noise into a filter that hides all but the output I want, then eventually I will see that output. But crucially, the input is still random noise. What is the value of ChatGPT if it's no more than curated randomness?

My point is that there is a very clear difference between impressive and functional. ChatGPT is impressive, not functional. You can't make it functional. You can only make it better at impressing you.

Seems trivial to me to bolt on some critical thinking filters.

Right now ChatGPT acts like pure Amygdala. You poke it, and it spits out something that looks right based on what it was trained on. This is fine, it's just a demo, and this gives a really deep understanding of the direct limitations of these sorts of LLMs.

Ideally a more mature chat bot would just do what I do in situations where some critical thinking or research is called for. I do a quick fact check. I think up a few search terms for Google, Wikipedia, Wolfram Alpha, whatever, assume those results are more logically accurate, and present that data instead of the typical vomit my brain usually spits out. ChatGPT is actually really good at coming up with good search terms for popular search engines, and incorporating results into responses. It just doesn't have anything that reaches out to the web at the moment, which is probably so a certain subsection of the population doesn't have a total panic attack.

It seems like fact check functionality would take less than 100 lines of relatively straight-forward python. I bet you could even get ChatGPT to write it.

Here's what I suspect filters would look like:

    function critical_filter(response) => Maybe Text {
        if logic_parse_check(response) == LogicallySound then {
            return Some response
        } else {
            return Nothing
        }
    }

    do {
        static seed = 0;
        seed += 1;

        response = critical_filter(GPT::generate_response(prompt, seed))
    } while response != Nothing

We are effectively brute-forcing hashes here. Take as many shots in the light as you need to hit, but take no effort to aim.

The only solution is a replacement.

Thought the same thing — we need ways to figure out if what ChatGPT wrote is correct or not. What do you think about an extension like this for fact checking?

https://twitter.com/skippperD/status/1613000965929709569

This comment and many others speculate on the limits of ChatGPT based on assumptions about what ChatGPT does that are not quite accurate. In particular, ChatGPT does not simply output the “most semantically popular result”. That description applies only to the base model, before instruction tuning and RLHF.

As for the speculation itself, e.g., “as soon as you merge two subjects, you are right back to gambling semantics as logic”, it may be suggestive to consider that such models have learned to translate between English and French after having only trained on the two languages separately.

Thank you for this comment!

I haven't spent much time learning about how ChatGPT works, but:

'ChatGPT does not simply output the “most semantically popular result”. That description applies only to the base model, before instruction tuning and RLHF.'

was exactly the explanation I needed!

Much like ChatGPT, you seem to have comprehended the words I said, but not their meaning.

Nope.

If you were to consider carefully enough the language example that I gave, you would see that it already refutes your speculation.

Another way would be to apply your argument to humans, who have managed to be useful despite having been trained only on (a broader class of) semantics (i.e., the sum of all qualia) and the logical imperfections that this entails (cf. your comments).

Happy new year!

I would wager that it will eventually be able to do math and logic, and may already be able to with the right prompting. It can follow instructions quite well, and multi step problems can be solved by breaking the problem into small digestible steps, and having it write out each step, just the same as humans do. If it can't do this yet, it's likely just that it doesn't have enough parameters yet. But I don't see why this couldn't be achieved with fine tuning.

But ultimately I think it's just too computationally heavy to do math, remember obscure facts, or track rapidly changing data, with an LLM, and it's far more efficient to pass it off to a piece of specialized software, just the same as humans do.

An infinite amount of "fine tuning" would solve any problem, yes...

...but what we just described isn't a language processor anymore: it's just a database. There is no need to use a neural network if you already have a known response to every possible input!

The entire point of ChatGPT in the first place is to generate a response that doesn't exist in the training data. If all we needed was to look up the answer we wrote down earlier, we already have SQL.

What we are talking about here is being able to provide a logically correct response without just pulling it out of a lookup table. Either you can do that, or you can't, and ChatGPT can't.

I get what you're saying, and it's a database if you're talking about facts, but breaking down mathematics and being able to perform logic isn't a database in the true sense, it's still able to generate novel responses based on a set of rules.

My point here is even if it only "gets" the semantics, it has the ability to perform logic. It's just not very efficient. And, I'd say this isn't that far off from what is happening in our brains.

Do we really "get" logic, or do we rely on heuristics? Why do we know that 12 * 2 is equal to 24? Either because we remember our multiplication tables (look up from a table) or because we break it into smaller steps until we're left with pieces of the problem that we inherently know (including 12 * 2 means 12 + 12 which means take 12 and increment the number +1 for 12x times, or 10 + 10 = 20 and 2 + 2 = 4 so 12 + 12 = 24).

I don't see why that couldn't reasonably scaled up to advanced calculus.

I think the point that you're hitting on is that LLMs aren't "full brains" that have all of the components that human brains have, and that's true. But LLMs appear to essentially have the ability to simulate (or replicate depending on how far you want to go) executive function. As others have hit on, from that, you can either command other specialized components to be able to do specific tasks (like recall facts, perform logic, etc) or you can have it break down the problem into steps. Then take the output from that processing and form a coherent response.

The structure here is flexible enough that I'm struggling to find the upper limit, and if all research in the development of LLMs stopped tomorrow, and everyone focused on building with LLMs, I still don't think that limit will be found.

if by "do" math and logic, you mean spit out what seems most reasonable then yes.

ChatGPT doesn't "know" anything, its just a prediction engine. if you ask it what 2+2 is, it will tell you 4, but it didnt actually do math. it only said 4 because everyone else said it.

there is no generalizable reasoning going on there.

See https://news.ycombinator.com/item?id=34340079, I would venture to say that this isn't any different to the reason why you know that 2+2 is 4.

Yes, and this schematic popularity reflects very clearly in mathematical problems: in maths, the schematic probability of plus and minus operation are equal, thus we should see very often ChatGPT first writes a quite convincing reasoning sentence but then produces a nonsensical equation and thus ends up all wrong.

This is the best description of the state of ChatGPT that I’ve read thus far.

…Unless this whole comment was generated by ChatGPT. (I strongly dislike that I’m starting to second guess whether comments are written by humans or not)

That's because everything written by ChatGPT is a transformation of other stuff that was written by humans.

If we trained it on gibberish and nonsense, we would get that, and no one would care. The tricky part is that we don't have much gibberish or nonsense to train with: because of that pesky property of human expression we call logic.