Hacker News Reader

Comment by jamesdwilson | Hacker News Reader

If it is never pulling from a source, then why is it able to provide citations?

check the chat bot on you.com:

https://you.com/search?q=list%20top%205%20best%20selling%20c...

One useful way to think of language models is that they are statistical completion engines. It attempts to create a completion to the prompt, and then evaluates the likelihood, in a statistical sense, that the completion would follow the prompt, based on the patterns in the training data.

A citation in legalese is very common. A citation that is similar or identical to actual citations, in similar contexts, is therefore an excellent candidate for the completion. A fake citation that looks like a real citation is also a rather good candidate, and will sometimes squeak past the "is this real or fake?" metric used to evaluate generated potential responses.

This may seem like "pulling from a source" but there is no token, semantic information, or even any information in the model about where and when the citation was encountered. There is no identifiable structure or object (so far as anyone can tell anyway) in the model that is a token related to and containing the citation. It just learns to create fake citations so convincingly, that most of the time they're actually real citations.

This explains some of the particular errors that I've seen when poking and prodding it on complex legal questions and in trying to get it to brief cases.

ChatGPT can provide correct citations because somewhere deep in its weights it does lossily encode real texts and citations to real texts. That makes real citations in some cases be its most confident guess for what is supposed to come next in the sentence. But when there isn't a real text it is confident about giving a citation about, but it still feels like a citation should be next in the output, it will happily invent realistic looking citations to texts that have never existed and it has never seen in any sources. On the outside, as readers, it's hard to tell when this occurs without getting an outside confirmation. I'm assuming though that to some degree it is itself aware that a linked citation doesn't refer to anything

A few days ago on HN there was a short story of five paragraphs that started badly & finished OK, and I wondered if some operations research tricks could be applied. One is forward-backward-forward planning, this produces better schedules, and if applied would create a better opening, and perhaps ending (i.e. model is run three times).

In the case of citations, you really need a language model & a fact model. The language model then passes over to the fact model, then back to the language model. This means double(+) training.

I suppose the fact model could include things like Wolfram (also discussed on HN).

Asking "Can you cite some legal precedence for lemon law cases?" gives an answer containing

"In California, for example, the California Supreme Court in the case of Lemon v. Kurtzman (1941) held that a vehicle which did not meet the manufacturer's express warranty was a "lemon" and the manufacturer was liable for damages."

I dont think that case exist, there is a first amendment case Lemon v. Kurtzman, 403 U.S. 602 (1971) though.

I can't find any reference to Kurtzman or 1941 in any of the references. I think the answer is that the AI generating the text, and the code supplying the references are distinct and do not interact.

The example you give isn't necessarily a valid one. You're asking for a specific piece of knowable, measurable data -- one that has a single right answer and many wrong answers. Legal questions may have conflicting answers, they may have answers that are correct in one venue but not in another, etc., I have't yet seen any examples of an AI drawing the distinctions necessary for those situations.

thanks very much for this insightful response. i haven't heard this perspective before but it makes sense.

ChatGPT has a lot of trouble understanding jurisdiction and what constitutes controlling precedent over what in which jurisdictions. As in it has no conception of it at all and gets it really badly mixed up. It doesn't understand the hierarchy of any court system so there are some questions that it will just always get wrong.

> If it is never pulling from a source, then why is it able to provide citations?

If you have the training set, and models that summarize text and/or assess similarity, and some basic search engine style tools to reduce or prioritize the problem space, it seems intuitively possible to synthesize probably-credible citations from a draft response without the response being drawn from citations the way a human author would.

Kind of a variant of how plagiarism detection works.

You.com is hugged to death right now, but from what I can see it's a different kind of chatbot. It looks closer to Google's featured snippets than it is to ChatGPT.

That kind of chatbot has different limitations that would make it unsuitable to be an unsupervised legal advice generator.

Reply on news.ycombinator.com