Hacking Around ChatGPT’s Character Limits with the Code Interpreter | Hacker News Reader

Hacking Around ChatGPT’s Character Limits with the Code Interpreter

knowsuchagency.notion.site

The character limit in GPT is a fundamental limit of the software architecture, not an artificial limitation used as an upsell or rate limit. Uploading a file give the plugin access to the file and GPT access to the plugin's output, so you might be able to get more informative answers using it, but it's not really fair to say that it "bypasses" the character limit in any meaningful way.

Consider a similar "bypass": you can upload the file to the web and the web browser plugin can read the page. You can do this with the whole git repo if it's public!

The token limit is 100% an artificial limitation. When ChatGPT first released last november I took the opportunity to try pasting 3k-line codebases into it to get it to walk me through them and it worked perfectly fine, putting that same code in the OpenAI tokenizer tells me it's ~33k tokens, way above the limits today. The reason they do this is because every token takes up ~1mb of video memory and that adds up real quick. If you had infinite video memory there would be no "fundamental limit" to how long a LLM can output.

OpenAI then has two limits on inputs. The first artificial one ensures that people don't get overzealous inputing too much, otherwise they'll hit the second hard limit of how much vram their cards have. To the LLM itself there is no difference between characters from the chatbot and human, the only hard limiter is the total number of tokens. I tried this out by inputing a 4k-token string into ChatGPT as many times as I could and it failed on the 20th input, meaning that the hard limit is >80k tokens. Converting this to vram gives us >80gb which is the exact amount of ram the Nvidia a100 card has.

> When ChatGPT first released last november I took the opportunity to try pasting 3k-line codebases into it to get it to walk me through them and it worked perfectly fine.

A common technique to work around the limitations in context length is to simply pull the most recent context that fits into the length. It can be difficult to notice that this happens because oftentimes the full context isn't actually necessary. However, specific details from the context are actually lost. For example, if you ask the model to list the filenames back in the same order, and the context was truncated, it would start from the first non-truncated file and the others would be dropped.

> If you had infinite video memory there would be no "fundamental limit" to how long a LLM can output.

Well, you've certainly got me there. One of the big limits with the transformer architecture today is that the memory usage grows quadratically with context length due to the attention mechanism. This is why there's so much interest in alternatives like RWKV <https://news.ycombinator.com/item?id=36038868>, and why scaling them is hard <https://news.ycombinator.com/item?id=35948742>.

FlashAttention has memory linear in sequence length. https://github.com/HazyResearch/flash-attention

> The token limit is 100% an artificial limitation.

My understanding is the token limit is an immutable property of the neural network once it has been trained, so definitely is not an artificial limitation - unless you’re suggesting OpenAI trained the NN with a higher token count then released it with a limit on the input to only allow smaller ones? Which I guess is plausible but I’m not sure why they’d do it, as they’d still be “executing” the same NN for every input so wouldn’t save any compute.

I think you misunderstood the token limit. LLMs don't block your buffer, they simply take the final n tokens of all of the input you've shared. Plenty of users have seen this.

It will still function, but anything you previously shared and referenced above it will lose context on. And if you ask it about that earlier content, it will do its best to hallucinate a reasonable answer of what might have been in your buffer before the cutoff.

Separately you may have found a physical hard limit with a bug that crashes the system, but that's not what's meant by a token limit in LLMs. It's a limitation of the architecture itself of any LLM.

  When ChatGPT first released last november I took the opportunity to try pasting 3k-line codebases into it to get it to walk me through them and it worked perfectly fine

Are these private codebases, or open source ones? If public, would you mind sharing a link to the ChatGPT session(s)?

The token limit depends on the way the training data was encoded, usually its using some sort of sin/cos function which has the problem where longer inputs than it was trained on cause the accuracy to plummet. Its also very likely that it just took the last part of your input as context.

I have had very rotten luck using the web browser plug-in, it almost seems like it only works once per website and then it gets hung up and fails to read any further information.

I had thought:

1. the GPT-4 code interpreter was just regular GPT-4 with the ability to swap out its context by running python code on one or more files, and

2. whenever it uses python to process files, it displays the 'Finished working - Show work' button.

But, based on the session I just had, it seems I was wrong: https://chat.openai.com/share/b97a206a-15cd-456a-9691-41946b...

Here, ChatGPT does all of these things:

- gives incomplete information about the files referenced in the JSON file

- runs some python code to get the correct list

- answers detailed questions about the code *without displaying the 'Finished working' button, suggesting it answered the series of questions solely from information in its context

This new breed of (ethical) hackers are what gives me hope.

Back in the day we had lod, mod, thc, mitnick we had the types of folks that didnt just give in. You know, ka-tet one from many.

Thanks to those people we have, at least a pretence of, checks and balances.

The battle of the intellects is on.

Those who know, know.

Reply on news.ycombinator.com