Hacker News Reader

I don't understand how you can write a whole piece on Transformers failing to do arithmetics without mentioning the tokenizer used in GPT, which makes it more or less impossible for the model.

The reason is imo probably more along the lines of that the model can't see the actual numbers you're giving it.

When you provide GPT3 with the number "10011111110100011100011011110010", it actually sees [3064, 26259, 1157, 8784, 18005, 1157, 18005, 8784, 1157, 3064, 940] [see 0].

The number isn't even split up into even parts. Even if it knew the Peano axioms, it wouldn't have a chance because it can't 'see' the actual numbers that you're feeding it. Hence it can only estimate these operations based on combination pairs it has seen in training data, and it seems pretty obvious why this can't be a robust basis for doing arithmetic.

0. https://platform.openai.com/tokenizer

It also completely ignores the developments around GPT tool use. If you’re evaluating the AI field as a whole, why are you restricting your analysis to the raw output of GPTs? Arguably the most impressive display of intelligence exhibited by these models is their ability to use tools like calculators to augment their own abilities.

It doesn’t matter if the model can’t do three digit multiplication without access to a calculator. Most people can’t mentally multiply large numbers either.

Most people can do 32-bit binary addition using a pen and paper. And the point of using addition as an illustrative example is that the rules of addition are extremely simple, yet a binary addition problem can easily be scaled to N bits of entropy. This means it's very easy to show that the system fails to apply simple logic rules in such cases.

It didn't fail logic, it failed to read your string. You didn't even notice that the terms being added in the 'show working' version don't match the given strings. And you are elsewhere talking about adding spaces to make the digits 'tokens'... you have no idea what you are talking about at all.

You can input the space-separated strings into OpenAI's tokenizer: https://platform.openai.com/tokenizer

You'll see that it does properly tokenize each binary digital as a separate token.

But again, the tokenization doesn't matter. I could say to you, please add these two binary numbers:

      101 0011 11001
    + 00 11 10 100011

And you could do it.

If your goal is to simulate a human doing addition with pen and paper it only seems fair to give the GPT read/write access to a text file. Outputting the answer directly from the model is much more akin to mental math.

ChatGPT does have read write access to a text file. It's allowed to show its work as part of its output, which it can write tokens to, and then read those tokens back as part of generating the next tokens.

Why does the tokenization matter? Also, ChatGPT doesn't "see" those numbers, it just sees smaller binary strings, and a human could add that series of binary numbers with knowledge of how to carry the result of the most significant bit.

Also, just to put this criticism to rest, I redid the addition example using space-separated bits and verified the tokenization was one token per bit (but again, it shouldn't matter).

    Let's add the two binary numbers provided, limiting the work on each bit to a single line:

      1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0
    + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1
    --------------------------------------------------------------
      1 0 1 1 0 1 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 0 0 1 1 1 1 1 0 1

The answer is still wrong. (It also didn't use a single-line per bit, but that's ok).

I'd argue extremely few humans could do this in their head, which is essentially the bar you are holding GPT too.

Most humans can do it with a pen and paper once told the rules. But you are giving them tools (and purpose designed methods) to bridge the gaps in humans own tokenizer. Who knows what the equivalent of pen and paper would be for an LLM.

> I'd argue extremely few humans could do this in their head, which is essentially the bar you are holding GPT too.

The prompt was "Now add 1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0 and 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1. Limit work on each bit to a single line." So the model could use its own output to show its work. Again, the rules are really simple and a human could do this trivially.

All I'm saying, is that, to me, it looks like ChatGPT isn't capable for understanding logical steps. There are many, many examples online of people creating complicated logical puzzles that ChatGPT "solves." My hypothesis is that ChatGPT isn't actually "solving" these and ChatGPT would fail on any follow up examples that had sufficient entropy. To me, this is a significant difference. You might create a simple model of a problem and be amazed that ChatGPT seems to work, only to find that it fails with more complicated problems encountered in a real-world environment.

That doesn't mean ChatGPT is useless, but it would be limited to automation purposes (operating on known examples) and could never be a general thinking agent.

The reason humans can do it is because you give them a pen a paper so they can do it one step at a time. You are asking chatGPT to do the whole thing in one step.

Give it pairs one at a time and ask it to output the binary value and carry value. You'll see it can do it just as easily as a human doing it step by step with pen and paper.

As far as I know, GPT does the work one token at a time. It produces a single token at each iteration. So if you give it "abc" and want "def" as the result, you first give it "abc" and it returns "d", then you give it "abcd" and it returns "e", then you give it "abcde" and it returns "f", then you give it "abcdef" and it returns the stop token.

So just by outputting its work, it can provide its own context to memoize what the answer for each bit is.

On the adding part - have you looked at how it tokenizes that string?

https://platform.openai.com/tokenizer

    10011111110100011100011011110010
    00111010100101000100000101111011

is tokenized as:

    [100][1111][11][101][0001][11][0001][101][11][100][10]
    [00][11][101][01][001][01][0001][0000][01][01][11][101][1]

It might be more trainable if the the bits where space delimited so that only the tokens [ 1] and [ 0] were in use.

As it is, it doesn't really understand the contents of the tokens and manipulating [405, 1157, 8784, 486, 8298, 486, 18005, 2388, 486, 486, 1157, 8784, 16] is not easily trained - especially if there is a different bit sequence. Note also the misalignments of the tokens for bit length that can make it more difficult to work with.

I disagree that tokenization is important to consider. But just to test it out, I tried adding spaces between each bit and verified that each bit is tokenized separately:

    Let's add the two binary numbers provided, limiting the work on each bit to a single line:

      1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0
    + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1
    --------------------------------------------------------------
      1 0 1 1 0 1 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 0 0 1 1 1 1 1 0 1

    So, the result of the addition is the binary number 10110110101001010111111001111101.

The answer is still wrong. Think about it like this: even when tokenized into 1-5 digit binary chunks, there are still just a few tokens to consider. If ChatGPT could actually apply the rules of addition to a general, high-entropy example, it would be trivial to use these tokens to derive the correct answer. A human could easily do it.

Perhaps also consider that you (the human) are using 2-dimensional visual processing to help you solve the problem by collating your work. You also almost certainly work /backwards/ from right to left. The problem is simplified by scratch space and the human sensorium... (It would be interesting to see how multimodal models handle the problem, actually, given a picture of the addition problem.)

The problem given to the LLM is more like:

1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0 + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1 = ?

Try writing the answer entirely from left-to-right without losing your place... It'll be error prone for humans, to say the least. It's why we invented our particular by-hand addition algorithms in the first place: these algorithms are co-adapted to the human sensorium.

Luckily, we can ask any LLM worth its salt to write the single line of python code needed to solve the problem and it'll do just fine...

edit: Thinking about this more - and trying to explicitly check the intermediate steps - I'm guessing that the LLM is getting lost in the inputs. The attention mechanism is going to need to find the correct pair of digits at each step, and combine them with some info about the carry variable. I'm guessing this is hard to do - there's a positional embedding used to attach some location info to each token, but in a long run of homogenous text, I would guess that it's hard to find a particular digit and end up with some off-by-one errors in the lookup. (and this is very similar to how we would expect humans to fail at the single-line version of the problem.)

I wouldn't expect a human to fail at the single line version. It's pretty easy to do. Write a list, at each position n in the list, compute the sum of the nth least-significant bit and the carry from the line above. You can easily get the nth least significant bit by counting from the right side of each number. The end result is the list printed backwards.

Consider the amount of work that you need to do to reverse a word in GPT.

https://twitter.com/npew/status/1525900868305833984?lang=en#

Note that part of the trick that they did to get it to work better is putting spaces around each letter to enforce a particular tokenization.

Trying to do the same with mathematical objects rather than language objects would be even harder because GPT doesn't "know" math.

You lost the thread at 'write.' Try to do it in your head; we pretty much have to resort to additional tools to succeed at this task.

Here's a LLM-generated python solution, which works just fine:

  def add_binary(a, b):
    """Adds two binary numbers given as strings.

    Args:
      a: The first binary number.
      b: The second binary number.

    Returns:
      The sum of the two binary numbers.
    """

    # Convert the binary strings to numbers.
    a = int(a, 2)
    b = int(b, 2)
  
    # Add the numbers.
    sum = a + b
  
    # Convert the sum back to a binary string.
    return bin(sum)[2:]

  add_binary('10011111110100011100011011110010', '00111010100101000100000101111011')

I feel like we're missing the forest for the trees here. I'm not really asking it to add two numbers. I'm asking it to apply a simple set of logical rules on two symbols with an operation '+'.

Of course it can generate code to add two numbers, because the code to add two numbers exists in its training data. What transformers seem to be good at is generalizing over the training data. Where they seem to perform poorly is data outside of the training set. Binary numbers are interesting because you can add a bit of entropy just by increasing the size of your binary number, so it's very easy to generate examples outside of the training data. You also see this, albeit less directly, when trying to ask it to generate code that wasn't part of its training data. E.g. https://twitter.com/cHHillee/status/1635790330854526981

Arithmetic is a known-bad example because of how the tokenization works. Similar to calling a human unintelligent because we can't recognize bat calls.

Additionally 'can't do x, and is therefore a lookup table' is a very poor logical leap. Are humans who can't add just lookup tables as well?

I would say that if you can't teach a human to add, then they might be a lookup table.

Also, the tokenization is fine, as shown in other examples. Imagine we didn't call it arithmetic and instead used symbols a and b, s.t. a + a = a, a + b = b, and b + b = a w/ carry b. Binary addition really is just very basic logical rules on symbols.

> Why is that a requirement?

Some sort of self-awareness is required for an AI system interacting with the real world physically - i.e. via controlling a robot or a vehicle - safely and to react appropriately for situations it has not been taught for.

> But the model fails to apply even basic logical rules as entropy increases, indicating that GPT models have a fundamental limit of applying logic to high entropy problems.

IIRC, the problem with current GPT AI models is tokenization. Basically, words are long and somewhat visually distinct with clear separation, whereas numbers are far harder to "convert" into tokens the model can use.

In the end I believe that this problem will eventually be solved by applying enough compute power, as will the problem with "self awareness" - run a GPT model in a continuous loop that allows self training.

The real and actually very hard problem IMO is adversarial training and inputs. In humans, the limits of a human brain in the speed of interaction and the number of other actors one brain can interact with act as a natural hard performance ceiling as well as a quantity barrier for too much adversarial input (you can only get fed bullshit 24 person-hours a day), and yet we're seeing adversarial inputs (aka propaganda) be very effective to the tune that 40% of Americans believe that the 2020 election was stolen or manipulated [1].

An AI system that can interact with hundreds of thousands of people simultaneously? Microsoft proved how fast that can go sideways with Tay 2016, when 4chan and other trolls managed to turn the bot into a raging Nazi in not even a day worth of effort [2]. Now imagine you have an AI running airspace travel control and someone convinces it that a plane got hijacked by terrorists or to lead planes onto a collision course...

[1] https://www.newsweek.com/40-americans-think-2020-election-st...

[2] https://en.wikipedia.org/wiki/Tay_(chatbot)

the linked blog is yet another instance of making false inferences from the false assumption of the model working at a letter/symbol level. it cannot see letters. it cannot see the 1 or 0 in 11010 either. (note that a request to simply reverse a word would illustrate this usually, but some glue code has been added to do it)

> > Self-Aware

> Why is that a requirement?

And what does self-aware even mean? A lot of animals pass the mirror test.

> > Self-Aware

> Why is that a requirement?

Not to mention that this cannot be proven at all, even for humans.