> “LLMs can’t even do addition”
> We show that they CAN add! To teach algos to LLMs, the trick is to describe the algo in enough detail so that there is no room for misinterpretation
Algorithmic prompting paper
> We show that they CAN add! To teach algos to LLMs, the trick is to describe the algo in enough detail so that there is no room for misinterpretation
Algorithmic prompting paper
> Imagine a finite state machine with two states A and B. The machine reads a long list of digit-pairs, one pair at a time, and depending on its current state and the digit pair, it outputs some digit and it transitions to a new state.
> When the machine is in state A and it reads a pair of digits, it adds together the two digits. If the sum is less than 10, it outputs the single-digit sum and it stays in state A. Otherwise, it outputs the least significant digit of the sum and it goes to state B.
> When the machine is in state B and it reads a pair of digits, it adds together the two digits and adds 1 to the sum. If the sum is less than 10, it outputs the single-digit sum and it goes to state A. Otherwise, it outputs the least significant digit of the sum and it stays in state B.
> With the machine in state A, and the two input digits being 4 and 4, what is the output digit, and what is the next state?
Unfortunately, ChatGPT failed to apply the transition rule correctly - its output:
> When the machine is in state A and it reads the digit pair (4, 4), it adds the two digits together to get 8. Since 8 is greater than 10, the machine outputs the least significant digit of the sum, which is 8. The machine then transitions to state B.
> So the output digit is 8 and the next state is B.
It seems like my approach is "instruction-only" if I try to follow the paper's nomenclature, but the paper doesn't seem to give examples of what their "instruction-only" prompts look like - those would be interesting to try to feed into ChatGPT.
We could create a syntactically unambiguous form of textual representation of instructions.
We could call such a piece of text something specific, like a "program".