Hacker News Reader

Comment by supafastcoder | Hacker News Reader

You’re right, but I think it’s fundamentally impossible to do with the current state of technology. Imagine the word “democracy”, would you be able to tell me where you’ve first learned the definition of it and be able to provide a direct citation of it? What about the thousands of other instances where you’ve seen that word defined? (In OpenAI’s case probably hundreds of millions)

Current LLMs work in a similar way, the information is synthesized by correlation, there is no way that it can directly relate where it learned it’s output from. The only viable way would be to do a reverse lookup of the output to find out if there’s a similar worded content on the internet. (Or what Perplexity does, wrap an LLM around search results, but you lose a lot of flexibility in this case)

View on HN

Citing basic definitions wasn't what I was talking about and it's not required for humans either for the reason you just gave, it's common knowledge. The problem the NYTimes seems to have is that ChatGPT can exactly reproduce large chunks of their articles. I'm saying that removing this ability would be bad. Finding a way for the model to identify when a large chunk of text is quoted verbatim and then clearly cite its source would be good. The output that is not a direct quote is not at issue here.

Yes, but an LLM can’t distinguish out of the box what needs to be cited or not. So you need another processing layer (and probably a search engine layer on top of that) to do that.

An LLM cannot do a lot of things out of the box, and OpenAI has more than a few processing layers already. They certainly already modify output based on verboten topics. Personally I feel any decent LLM should be able to attribute its sources even when not a direct quote, because that raises the bar that much higher on its trustworthiness and general value.

> LLM can’t distinguish out of the box what needs to be cited or not

LLM don’t know that they are supposed to answer questions out of the box either. This is what reinforcement learning from human feedback is good for.

> you need another processing layer

Certainly. For reliability and peace of mind I would implement a traditionally coded plagiarism search on the output. (Traditionally coded as in no AI magic required in that layer) If there is a match you evaluate how significant it is, that most likely is best done with machine learning. For example if the match is a short and simple statement you can ignore it. If it is a longer match, or a very specific statement you need to do something. What you do depends on your resources and risk appetite. Simply supressing the output is an option, it is cheap and very reliable. Rephrasing until it no longer matches is an other, takes a bit more compute and you can end up skirting too close to plagiarism. Rewriting the output to only quote a short amount and provide citation is even more costly, but can be done too.

You can do all of these post processing steps at generation time. And you can also use it to enhance your training dataset. That way the next version of the model will more likely to do the right thing right away, because it learned the correct patterns to quote things.

> Imagine the word “democracy”, would you be able to tell me where you’ve first learned the definition of it and be able to provide a direct citation of it?

Worth noting that we mostly learn words by seeing them used, rather than by being given an explicit definition. Most of my vocabulary I learnt not from a dictionary and I expect the same is true of almost everyone. As such, there is no well defined point at which "the definition" is learnt, both because there's no such thing as the one true definition and because the meaning is gradually being changed and refined by each person as they see the word used more.

Not only that, as Orwell says in Politics and the English Language [0]

> The words democracy, socialism, freedom, patriotic, realistic, justice, have each of them several different meanings which cannot be reconciled with one another. In the case of a word like democracy, not only is there no agreed definition, but the attempt to make one is resisted from all sides. It is almost universally felt that when we call a country democratic we are praising it: consequently the defenders of every kind of régime claim that it is a democracy, and fear that they might have to stop using that word if it were tied down to any one meaning. Words of this kind are often used in a consciously dishonest way.

[0] https://www.orwellfoundation.com/the-orwell-foundation/orwel...

> it’s fundamentally impossible to do with the current state of technology

Maybe.

> Imagine the word “democracy”, would you be able to tell me where you’ve first learned the definition of it

Sure, it was elementary school history lessons when we were learning about the greeks.

> be able to provide a direct citation of it

Naturally. I would just look it up in one of the many dictionaries. Copy the definition and tell you which dictionary it was copied from.

> What about the thousands of other instances where you’ve seen that word defined?

What about them? When you are providing sources you are not required to provide all the sources you ever read, nor is it desireable.

> The only viable way would be to do a reverse lookup of the output to find out if there’s a similar worded content on the internet

They don’t have to do that. It is enough if they do a plagiarism search within their own training dataset.

You're effectively saying that current LLMs cannot be taken seriously as some expert, they are just some kind of weird text remixing engines. I would agree. But if the AI wants to be relied on, then it should be, at minimum, capable of taking a word like "democracy" and compare its own definition with the Wikipedia definition, and verify whether it's used correctly in its output.

Yep, that’s what I meant with the reverse lookup strategy.

Reply on news.ycombinator.com