Hacker News Reader

It's actually supper annoying.

I keep asking it to write out entire pieces of code, instead of only things it changed, can be annoying to merge it otherwise. But nowadays it acknowledges my request "I will write complete code" and proceed to still comment out parts.

I think they probably finetuned that because it used to be the other way around and it was pretty bad UX to wait 1 min to apply a function change to a class or stuff like that.

Yeah sometimes you want it but sometimes you dont. I think they are enforcing it to save resources

Are you using the API, or just the ChatGPT UI? Do custom instructions/system messages help at all?

Saving resources like this seems okay(ish) for general-purpose ChatGPT, as they'd know there's a human who can interpret the abridged output. But for API-driven requests this would be pretty terrible, and makes it far harder to develop with.

no the UI, where I ask it to "Please write out entire code with skipping parts."

ChatGPT 3 or 4? Try the playground. I’ve had much better luck with system prompts.

Get it to write a diff patch?

Claude instant 100k does this all the time.

I have this too, but on information extraction tasks, if the input document is too long or complex it will skip sections saying something along the lines of

< !-- List of products. Truncated for brevity -- > (instead of the list)

< !-- Shipper details go here -- > (instead of the details)

Very annoying when you want to extract everything and is hard to fix. Maybe they had some examples solved by abbreviation in the fine-tuning set and it damaged the model. It doesn't make sense to train this behaviour into GPT-4.

Using GPT models like that is unreliable anyway.

I was talking to someone who was excited that he was able to use ChatGPT to convert a short tabular text listing of about 70 rows to a comma-separated list of the ids on each row, without having to e.g. import it into a spreadsheet or database. I pointed out that this couldn't be relied on, but he was skeptical. So I checked the output he pasted - there were only 69 ids. It had left out an id from the middle of the list, seemingly at random.

This boggles my mind. I know doing it by hand, or using a tool takes more time and effort, but if you’re going to automate it away _anyway_ you should be doing it in a reliable and retractable way.

I had a vim session saved that had a command history with a handful of regex patterns to format copy/pasted bank statement pdfs converted to docx (don’t ask) that would format the data into a CSV. Load the session, paste the ugly data, and cycle through the command history. It was ugly but reliable, and after the initial time sink of writing it once, it became quite efficient and wasn’t made more so (say a VimScript function) because it worked well enough.

Yeah, I know. The person in question is CTO of a well-funded Series B startup, and is highly technical. But he's very much a cowboy when it comes to coding practices etc. I think LLMs appeal to him because they hold out the promise of not having to deal with his own developers when he's trying to do something himself that he doesn't already know how to do.

The tendency to begin summarising is very annoying. I'd assumed it was because of limited attention span of human raters who rated summarised or shorter outputs more highly. And I'd assumed this had been there from the beginning.

I encountered it when doing some research into getting GPT-4 to reliably multiply n-digit numbers. Up to 8x8 multiplications it doesn't do this very much, but by 10x10 it is almost impossible to get it to stop doing it.

When the multiplications become even larger, it seems to be literally impossible to prevent.

This doesn't seem like an unintentional behavior? It seems like they don't want people querying their API for an indefinite amount of time.

What amount of time do you expect a query to be able to run for before timing out?

I lost countless hours debugging this, finally I came to the conclusion it must be unintentional because:

  1. It's undocumented. None of the regular rate limit responses are returned.
  2. You're charged for the full generation length. So if the output takes 10 minutes to generate, that's what you'll pay for (despite only getting half back).
  3. It defeats the point of the larger context limit models. Why offer a 32K model if it fails after ~6K tokens?
  4. The server response doesn't include any error codes or message, it simply terminates unexpectedly. Hit any of the actual rate limits, and you get told about it.

I'd expect to be able to generate output until the model reaches its context limit, or a stop sequence is detected, or I hit an actual documented rate limit.

We're paying for these requests in full. We should get the full response back!

You have fair concerns. Thanks for elaborating. Sorry you experienced all that trouble.

I wouldn't be surprised if this was an Azure related issue, given some of the similar madness I've experienced on Azure

Given that OpenAI is probably heavy on ML scientist types and lean on SRE types, I suspect there's just some nginx setting at the load balancer/edge servers that sets the timeout to 5 minutes, and it wasn't communicated to whoever was designing the API/writing the docs.

The inability to resolve issues like the mentioned 5-minute timeout with OpenAI support even after months should be a red flag to not build any products based on their services.

Can you give me a use case where generating a response for more than five minutes is important or even desirable?

It’s obviously a problem for the parent (https://news.ycombinator.com/user?id=jw1224), but the issue is not primarily whether such a limit is reasonable or not, it’s that it’s undocumented and seemingly unintentional (relative to the limits they do document), and most importantly it’s that the lack of response from OpenAI support indicates that you can’t count on them providing any reliable and timely level of support. This is just unacceptable for a service you’d want to build a business on.

There’s no real alternatives for a product that would need their service to begin with.

That still means it’s a no-go to build a business on such a product, in my book.

I agree 100%. It’s irresponsible and frankly a stupid idea for anyone to bet on building a business that relies on ChatGPTs API (or really any private API in 2023). We’ve seen time and time again (and more and more recently) companies weaponizing API dependence and fees to drive sucker-ons out of business.

If you want something to build a business around, build with an OSS model. Your competitive moat is going to be trash, but that’s going to be true for anyone building an AI startup that’s backed by any LLM technology. It’s a fool’s gold rush right now no matter how you slice it.

I've seen that "one, two, skip a few" behaviour converting structured data from Russian to English, and in the occasional programming task. Gotta really emphasise you want the whole document, complete, translated.

But we're fighting an uphill battle against OpenAI's own training, and I really wish we weren't.

I have to wonder if it's coincidence that this behaviour is less computationally intensive.

Yeah I’ve noticed similar, you can even explicitly say, do not truncate your output, or do not take any shortcuts. Make sure you haven’t missed anything ensure there are no placeholders etc…. And it ignores all of them.

That looks like a timeout configured on the web servebr

Yes, it seems their servers are terminating responses early. Unfortunately it's undocumented, was introduced unexpectedly one day, and prevents full use of larger context-window models.

It also charges in full for the entire response, which seems to continue running in the background. Definitely a bug at their end.

Seems like a means to avoid DoS?

I think it's just some networking misconfiguration they haven't noticed. I can't see them intentionally trying to prevent DoS this way, that's what API key rate limits are for (it would be a very expensive and slow DoS otherwise!).

>that's what API key rate limits are for

You don't need an API key to make a connection to the API.

Since it's 5 minutes I'm wondering if it's just "that's how the web works by default". I remember a while ago I was doing a long-running calculation on an POST request and was trying to debug why the connection always broke after 5 minutes, and found the hard way that there's a 5 minute timeout.

Very likely. Typical serverless hosts like Vercel times out at 5 minutes.

I was 'arguing' with it just yesterday about this with some coding work!

I am newer to some things I am currently working on so it's been like a free tutor/mentor. Really nice at times.

But lately, it will go into an explanation with code examples and part way through the code examples it adds things like '# The rest is very similar. Not including for brevity'

Not exact wording of what it says but it is essentially saying that. It is incredibly annoying because I have started telling it not to do that, to include the full code of the example, and it STILL DOES THIS.

No, I'm not getting it to code for me. The examples aren't even things I would use in my code but for teaching me certain concepts along with them walking through them with me which, in the past, worked really well and never had this repetitive issue.

GPT-4 is definitely operating differently.

Oh yeah, I had to write a simple but long Prometheus query.

> [Correctly lists the first fifteen Elements]

> # ... continue for each bucket

> This would be for individual buckets, and you'd have to add more lines for each additional bucket in your range.Given that you have a wide range of buckets, writing this manually could be quite tedious. This is why pre-computing these values with recording rules in Prometheus might be a more scalable solution, but the above should give you an idea of how you could do this in a more manual fashion.

Arguing with the AI who of the two of us has to do the tedious work.

I can't replicate this.

I tested your prompt verbatim in the OpenAI playground. Model set to gpt-4-0314, maximum length set to 6485 (nothing special about this number, I just dragged the slider to something high-ish), all other settings left at their default.

I cancelled the request when it reached "one thousand one hundred twenty-one" (which took over five minutes), but up until that point it did not take any shortcuts. I haven't meticulously checked every number, but it printed exactly 1,121 lines, and a cmd+F for "..." doesn't return anything.

I wonder if the streaming problems are related to the problem I experienced a lot lately to do with constant network connection error when using ChatGPT over mobile tethering (via Google Fi). The mobile connection itself is very stable and fast, but for a while every single ChatGPT response stops streaming halfway through with network connection error.

Did you take in to account that the default GPT4 model changed in June? That would explain the response difference. You can still use the original model released in march with the GPT-4-0314 identifier.

I first noticed the counting shortcut back in April. It still persists with both the previous and current models.

Wow, was this with the regular model (gpt-N) or a purported unchanging datestamped model like gpt-N-0314/0613?

This was using gpt-4-0314, fixed and datestamped. The counting shortcut still affects newer models too, but the old version undoubtedly changed behaviour without warning.

That's honestly beyond the pale. Thanks for the heads-up.

I can get 8k + tokens response from OpenAI API without problems (with gpt-3.5-turbo-16k) but I encounter errors from Azure OpenAI when using the same prompt and parameters. Obviously it sucks because I can't get long response from gpt-4-32k on Azure

The issue you linked to is related to an SDK, not the API itself. Have you tried just directly streaming the response back without an SDK? For example we use fetch directly without any SDK for OpenAI in production with no issue

I've tried making manual requests without an SDK, exactly the same thing happens. Even when you don't use streaming (wait for the full response), it still occurs.

Whilst the bug is API-specific, there's no other way to report this, and that issue is the only place I've found anyone discussing this.

I contacted OpenAI support about it, waited several weeks, and just got a generic useless autoreply back.

That looks like a library issue than an API issue no?

Definitely an API fault, but that library issue is the only place anyone's discussing it so far. OpenAI support were no help, I waited several weeks and they just sent back an autoreply linking to their docs.

How does chat-gpt actually get this right? This would appear to require some degree of reasoning, but as far as i understand its output is purely probabilistic, based on existing corpus of text.

You answered your own question.

Can you elaborate?

Your question:

  > How does chat-gpt actually get this right?

Your answer:

  > its output is purely probabilistic, based on existing corpus of text

Because GPT was trained on existing text, some of which included numbers and counting, it's learnt the natural ordering of most common/everyday numbers.

For larger or more complex numbers, it's learnt the patterns behind how they're constructed linguistically, which allows it to output a written count in sequence.

This same pattern recognition doesn't work anywhere near as well for actual numerals (e.g. "47600", instead of "forty seven thousand six hundred"), as the tokenizer tends to break long numerals apart (e.g. into ["476", "00"]).

In text, we don't often count in series, and it seems likely that we often choose a non-counting sequence: like 'I chose options 1, 2, 7' or 'my code was 0 1 2 5', whatever.

Unless training included line-level skips, rather than just next-word skips (like word2vec) or concept-level associations? At the line level, or paragraph level, ordered numerical sequences are obviously very common in formal texts or in code.

I've seen sentence based training, I suppose for code (which it seems GPT4 excells at) line-level training would be essential.

Anyone recommend a mid-level read on this covering different modes of training and such; I'm happy with a bit of code and undergrad level maths. Thanks.

> Unless training included line-level skips

Yes, of course — GPT-4 was trained on all common character sequences, including linebreaks and other invisible characters.

You can see how it works here: https://platform.openai.com/tokenizer

Nonetheless it doesn't need to have seen examples of line-level counting before. The "concept-level associations" you mentioned are an emergent property of the model, it forms its own concept-level associations as a result of being trained on such a massive dataset. It's what enables it to output original content which has never been seen before.

Maybe we count in series a lot more than you think.

https://www.youtube.com/watch?v=WO2X3oZEJOA

Aren't your sentences probabilistically based on the existing corpus of text within your mind?

No? If that's how sentence forming worked it would be impossible to count backwards, or count at all for that matter.

Me: Count backwards from ten

GPT-4: 10, 9, 8, 7, 6, 5, 4, 3, 2, 1

Me: Count backwards from a million to 999990

GPT-4: 1,000,000, 999,999, 999,998, 999,997, 999,996, 999,995, 999,994, 999,993, 999,992, 999,991, 999,990

I don't know about that.

I don't know how that connects in any way to what I and the commenter preceding me wrote.

You said that sentences aren't formed by probability from an existing corpus, as that method can't count backwards. GPT-4, a system based on such a method, can count backwards, thus disproving your claim.

I’m pretty sure the person you’re responding to is saying that counting is a formal operation with a formal definition, and that when GPT “counts” it isn’t doing that formal operation. The fact that it can semi-reliably emit ordered sequences isn’t counterevidence, for the same reason that memorizing the order of 0 to 100 isn’t evidence of an understanding of the inductive structure of counting.

How long a sequence of numbers (or how many sequences of different kinds of numbers) would it take to convince someone that it does do a formal operation?

Kids don't do a 'formal operation' when they count from 1 to 10, they just recite a list they've memorized. If they can do it from 1 to 1,000,000 you can be fairly sure it is no longer a memorized list.

> thus disproving your claim

Is there an alternative conclusion which you could reach here?

Why? Numbers have directionality relative to the species.

Not completely. If you trained me on a thousand bad arguments for why the earth is flat and one really good debunking, I predict I would reason myself to the correct conclusion. I predict ChatGTP would not.

That seems like a compelling argument at first, but the churches are full of intelligent people who will fail the same test because of childhood indoctrination. They have been conditioned not to care about what's objectively true, while the LLM simply doesn't possess a concept of objective truth in the first place.

I suspect it will be easier for ML researchers to fix that problem in their software models than it would be for psychologists to fix it in humans.

Objective truth doesn’t exist. Any system which defines truth can only point to objects within that systems language space.

We don't live in a "language space." We live in a material world where objective truth -- the kind that remains after you stop believing in it -- indisputably exists.

Utter bollocks. See how easy it was to refute your argument within that systems language space?