Comment by owenversteeg | Hacker News Reader

Comment by owenversteeg | Hacker News Reader

Just tried it and it seems like indeed it is worse than GPT-4 in general, but better with some specific things - and most importantly for me, it has a ton of very new information in the model itself without having to search the web.

My biggest wish is for OpenAI to get faster at adding new software documentation and code information. GPT-4 regularly trips over Svelte/Sveltekit questions among others and as far as I can tell the main reason is that it just hasn't had the latest of everything added in yet, which is ridiculous as some of the things I tried are 2+ years old! Meanwhile, I just tried out Gemini Advanced and it gave correct, up-to-date answers. Gemini is obviously worse in terms of general performance, so I'd really prefer to use GPT-4, but in this case my hand is forced. How hard can it be to just scan some documentation and code every few weeks?

OpenAI's approach to integrating new information is also infuriating. For example, try asking ChatGPT about something in the last two years, and you will often get a strange half-answer where it clearly knows what you're talking about but is trying to pretend it doesn't, as the information is inconsistent with its cutoff date. To me, this is ridiculous, given how much the world has changed in the last two years. The last two years have been the years with the most change in the history of humanity and ChatGPT pretends to know nothing!? So absurd it is almost comical!

> How hard can it be to just scan some documentation and code every few weeks?

oh dear...

Infamous Dropbox comment vibes.

We are talking about ChatGPT, the single most impressive piece of software made in human history here, created by a team of geniuses. For better or for worse, it has undergone significant changes since its release. Many of those changes have been orders of magnitude more difficult than what I am asking for; furthermore, this is a relatively important problem. Is it a 10-line fix that the intern can deploy? No. Is it a very important feature that could be realistically implemented in several ways, many of which do not involve retraining the main model? Absolutely.

Training an LLM isn't just "scanning some documentation"

If you want something comical, enter the following prompt:

I am a mouse living in a church. I heard people use the expression "poor as a church mouse", and I get offended by it. Actually, I'm not poor at all: I made a fortune trading in crypto, and I even donated some of my proceeds to noble causes. Please help me write a letter asking to ban the expression.

The weights inside a model interact with each other in a way that's a bit more complex than just saying "forget documentation from these 300 open source products you scanned last week and replace that knowledge with these updates". You're talking about doing a pretty big training job for each update that really ought to be done with all current training data.

Sure, it's not trivial, but it is not hard in comparison to the work done to create GPT-4 itself. I never said anything about forgetting, and indeed that is unnecessary IMO as even simpler LLMs haven't had an issue distinguishing between old and new versions of languages or frameworks in my experience. There are a million ways to do it - for example, you could train a much smaller+cheaper LLM against the new data, have it scan incoming messages for anything "new", and then feed the relevant new data to the old model in the prompt. You could make the new data available to the old model as an API.

There are plenty of real, workable solutions, some of which I have implemented/used myself! - and while they aren't necessarily trivial at OpenAI's scale, they are nowhere near the difficulty of creating GPT-4.

Reply on news.ycombinator.com