Hacker News Reader

Comment by PeterisP | Hacker News Reader

It's a big pain when trying to build things on top of the GPT-4 API. We had some experiments that were reliably, reproducibly achieving a goal, and then one day it suddenly stops working properly; then the student managed a different prompt that worked (again, reproducibly, with proper clean restarts from fresh context), and within a few days it broke.

I understand that there is a desire to tweak the model and improve it, and that's likely the way to go for the "consumer" chat application; however, both for science and business, there is a dire need to have an API that allows you to pin a specific version and always query the same old model instead of the latest/greatest one. Do we need to ask the LLM vendors to provide a "Long Term Support" release for their model API?

View on HN

As the founder of NLP Cloud (https://nlpcloud.com) I can only guess how costly it must be for OpenAI to maintain several versions of GPT-4 in parallel. I think that the main reason why they don't provide you with a way to pin a specific model version is because of the huge GPU costs involved. There might also be this "alignment" thing that makes them delete a model because they realize that it has specific capacities that they don't want people to use anymore.

On NLP Cloud we're doing our best to make sure that once a model is released it is "pinned" so our users can be sure that they won't face any regression in the future. But again it costs money so profitability can be challenging.

same for me - also the api itself is very unstable sometimes the same prompt finish’s within a minute, sometimes our client timesout after 10 minutes and sometimes the api sends a 502 bad gateway after 5-10 minutes. the very same request then runs fine within a few minutes after a delay of 5 minutes. the results vary very much, even with a temperature of 0.1

requests that needs responses with over ~2k tokens almost always fails, the 8k cannot be used

I try to use the api for classification of tickets, which i thought the model would be a good choice to use for

They have such a huge load, I’m not at all surprised

In the API you can ask for a specific version. Were you doing that?

They almost certainly were. But the API only offers two default choices of GPT-4 (unless one has been anointed with exalted 32k access):

1. gpt-4-default which has been progressively nerfed with continuous no notification zero changelog zero transparency updates .

2. gpt-4-0314 which is a frozen checkpoint from shortly after public launch and is still great but not quite as good as the very original, or as flexible as the fridged based model. Fine. However it’s currently due to “no longer be supported” i.e. retired on June 14th.

It’s kind of a challenge to commit to building on murky quicksand foundations of an API product that changes drastically (but ineffably for the worse) without warning like the default accessible version does, and soon it looks like there won’t be a stable un-lobotomized alternative.

The latest GPT 3.5 model has actually been getting better at creative writing tasks on a regular basis, which is actually bad for certain tasks due to the token limit. Whereas before GPT 3.5 could write a short story and finish it up nicely in a single response, now days it is more descriptive (good!) and thus runs out of tokens before concluding (bad!)

Reply on news.ycombinator.com