Hacker News Reader

Comment by theptip | Hacker News Reader

Very interesting, I'd like to see more concrete citations on this. Last I heard the training set for ChatGPT was static from ~ mid-late 2022. E.g. https://openai.com/blog/chatgpt/.

Is this something that Bing is doing differently with their version perhaps?

View on HN

I think the statement is that the LLM is given access to internet search, and therefore has a more recent functional memory than its training data.

Imagine freezing the 'language' part of the model but continuing to update the knowledge database. Approaches like RETRO make this very explicit.

I don’t think that parses with the current architecture of GPT. There is no “knowledge database”, just parameter weights.

See the Toolformer paper for an extension of the system to call external APIs, or the LaMDA paper for another approach to fact checking (they have a second layer atop the language model that spots “fact type” utterances, makes queries to verify them, and replaces utterances if they need to be corrected).

It’s plausible that Bing is adding a separate LaMDA style fact check layer, but retraining the whole model seems less likely? (Expensive to do continually). Not an expert though.

While ChatGPT is limited to 2022, Bing feeds in up to date search results.

Ben Thompson (of Stratechery) asked Bing if he (Ben) thought there was a recession and it paraphrased an article Ben had published the day before.

(From Ben’s subsequent interview with Sam Altman and Kevin Scott):

> I was very impressed at the recency, how it captures stuff. For example, I asked it, “Does Ben Thompson think there’s a recession?” and it actually parsed my Article on Monday and said, “No, he just thinks tech’s actually being divorced from the broader economy,” and listed a number of reasons.

Have you noticed how search results have evolved?

The Search box.

The Search box with predictive text-like search suggestions.

Results lists

Results lists with adverts.

Results lists with adverts and links to cited sources on the right backing up the Results List.

Results lists with adverts and links to cited sources on the right backing up the Results List and also showing additional search terms and questions in the Results List.

I'm surprised its taken them this long to come up with this...

It’s also really hard to get Google to say bigoted things.

Back in the day, all you had to do was type in “Most Muslims are” and autosuggest would give you plenty of bigotry.

It wasnt just Muslim bigotry, it was also anti-Semitic as well.

https://www.theguardian.com/technology/2016/dec/05/google-al...

However the so called free british press have perhaps outed their subconscious bias with their reporting and coverage!!!

https://www.telegraph.co.uk/technology/google/6967071/Google...

This is already documented. https://en.wikipedia.org/wiki/Missing_white_woman_syndrome

That’s relatively easy to fix, since autocomplete was probably working on just the most frequent queries and/or phrases. You could manually clean up the dataset.

Reply on news.ycombinator.com