Hacker News Reader

Comment by bitsinthesky | Hacker News Reader

At what point does the arithmetic become unstable?

It is quite unstable and frequently generates incorrect results. E.g., with the Fibonacci sequence prompt, sometimes it skips a number entirely, sometimes it produces a number that is off-by-one but then gets the following number(s) correct.

I wonder how much of this is because the model has memorized the Fibonacci sequence. It is possible to have it just return the sequence in a single call, but that isn't really the point here. Instead this is more an exploration of how to agent-ify the model in the spirit of [1][2] via prompts that generate other prompts.

This reminds me a bit of how a CPU works, i.e., as a dumb loop that fetches and executes the next instruction, whatever it may be. Well in this case our "agent" is just a dumb python loop that fetches the next prompt (which is generated by the current prompt) whatever it may be... until it arrives at a prompt that doesn't lead to another prompt.

[1] A simple Python implementation of the ReAct pattern for LLMs. Simon Willison. https://til.simonwillison.net/llms/python-react-pattern [2] ReAct: Synergizing Reasoning and Acting in Language Models. Shunyu Yao et al. https://react-lm.github.io/

What is the point of your article? Is it to figure out whether an LLM can run recursion?

If so, did you try anything else but the Fibonnaci function? How about asking it to calculate you the factorial of 100,000, for example? Or the Ackermann function for 8,8, or something mad like that. If an LLM returns any result that means it's not calculating anything and certainly not computing a recursive function.

My personal point was just to document my exploration of using prompts to generate new prompts, and more specifically the case where the prompts contain state and each recursively generated prompt updates that state to be closer to an end goal (which in this case I compared to the concept of a base case in recursion).

For whatever reason Patrick H. Winston's MIT OCW lecture on Cognitive Architectures always stuck with me, and in particular his summary of the historical system from CMU called General Problem Solver (GPS) in which they try to identify a goal and then have the AI evaluate the difference between the current state and the goal and try take steps to bridge the gap.

https://www.youtube.com/watch?v=PimSbFGrwXM&t=189s

The ability for LLMs to break down problem into sub-steps (a la "Let's think step by step" [1]) reminded me of this part of Winston's lecture. And so I wanted to try making a prompt that (1) contains state and (2) can be used to generate another prompt which has updated state.

[1] Large Language Models are Zero-Shot Reasoners - https://arxiv.org/abs/2205.11916

>> My personal point was just to document my exploration of using prompts to generate new prompts, and more specifically the case where the prompts contain state and each recursively generated prompt updates that state to be closer to an end goal (which in this case I compared to the concept of a base case in recursion).-

I don't understand what you mean by "state". I think you're using the term too loosely, like you use "recursion", so loosely that it loses all meaning.

To have state you need to have memory that you can read from and write to. To have recursion you need memory organised in a specific manner, as a stack.

There's nothing like that in your prompt, or in the setup of the bot that you interact with. It doesn't "recursively geneate" any "prompt updates", it takes your prompt, prepends it to its responses and your prompts until now, and generates a new response. If anything, it produces its responses sequentially, not recursively.

Anyway you're being rather freewhiling with terminology and I don't understand what you are trying to say. Are you trying to make the bot compute a recursive function, or not? Why are you using Fibonacci, if not? Why not just ask it for Little Red Riding Hood or the Three Little Piggies instead?

...and to reply to your second question, one thing I find interesting and want to explore further is how (and when) to best leverage what the LLM has memorized.

The way humans do math in our heads is an interesting analog: our brain (mind?) uses two types of rules that we have memorized:

1. algebraic rules for rewriting (part of) the math problem

2. atomic rules things like 2+2=4

So I'm wondering if we could write a "recursive" LLM prompt that achieves a similar thing.

Related to this, as part of another classic CMU AI research project on Cognitive Architectures, John R. Anderson's group explored how humans do math in their head as part of his ACT-R project: https://www.amazon.com/Soar-Cognitive-Architecture-MIT-Press...

The ACT-R group partnered up with cognitive scientists & neuroscientists and performed FMRIs on students while they were doing math problems.

In almost all cases very quickly. A LLM doesn’t have the ability to perform calculations but instead it feeds text tokens from the prompt into a model which predicts what the next tokens should be.

It can’t do basic maths but based on everything it’s been trained on it can give the impression it can.

Recursive feedback isn’t likely to improve the prompt unless there is some testing and feedback provided in the Python script.

You could play a game of chess and while the LLM knows the rules of chess it isn’t actually playing chess, it is calling upon patterns it has learned to predict text tokens that are appropriate for the given prompt. So opening moves will be sound, but it would quickly go off the rails and start hallucinating…

Given how they work, it is amazing they give the appearance of knowing anything. Even asking “how did you do that?” gives generally compelling answers.

Reply on news.ycombinator.com