Hacker News Reader

Comment by herbst | Hacker News Reader

Ive used ChatGPT to make short factual videos for YouTube and honestly it's a bit worrying with supposed 'facts'

I would not suggest anyone to use ChatGPT outputs for actual knowledge at this point.

You can use it for that, but you have to test its claims. It's specially helpful with questions that would take a lot of research to find the answer, but which are easier to test. That could be a good thing overall. If people get used to always testing everything anyone tells them, people will be a lot better informed.

I think LLM checking is going to be the most important research direction this year. Generative models are worthless without verification. It relates to deciding truth and dealing with fake information, synthetic text and spam. Google hasn't been able to solve it in the last decade, but some people have an idea:

> "Discovering Latent Knowledge in Language Models Without Supervision" Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way.

https://arxiv.org/abs/2212.03827

In other words the model already tries to predict the truth because it is useful in next token prediction, but we need to find a way to detect the 'truth alignment' in its activations.

I asked it how to do something with an AWS cli tool. ChatGPT invented a new parameter that looked like a programmer would come up with it and it would do exactly what I was looking for (I assumed many people had my problem before).

aws cloudfront update-distribution --id <distribution-id> --distribution-config <new-config> --no-reset-origin-access-identity

Took me a while to figure out that the parameter --no-reset-origin-access-identity was not only not working. But it did never exist on any version of the cli tool.

Did you not think to check the docs? (Srs question)

At least in my experience, the aws docs are difficult to understand at best, and horrifically outdated and contradictory at worst.

ChatGPT was my first choice just because I want to know if it is any good at helping me with my work.

But for this problem (CF-Distribution lost OAC settings when updating the root file), all the google fu in the world did not help me. It turned out that I had to update my aws-cli and my problem went away. Apparently no one else on the internet had that problem, so only my gut could help me figure it out.

Same here. ChatGPT invented a feature of our API and wrote an example script. User then complained to us that it's not working.

I asked it to create a COBOL program to connect to AWS and create an S3 Bucket...

https://news.ycombinator.com/item?id=33991767

ChatGPT works best with analytic prompts that respond with translations and not synthesis.

Meaning, if all the facts exist in the prompt then the likelihood of synthesizing fiction is diminished.

There are a number of ways to use the principle of analytic augmentation to add most or if not all of the facts required for a truthful response, ranging from simple “prompt engineering” to evaluating code to document embedding in latent space.

For example, if you use prompt engineering to k-shot a task to turn math word problems into executable JavaScript, meaning LLMs are only translators and the computations are done by a software interpreter, then the results are much more likely to be truthful.

Sampling from a number of variations on a prompt can lead to a more accurate outcome if say 1:10 times the translation attempt has a different answer.

I think getting in to the habit of fact-checking everything LLMs come up with when used to generate is probably an excellent idea. That being said, I haven't really seen any confabulation when using it to summarize a supplied text.

I've not used ChatGPT yet (almost feel like a luddite), what happens when you tell it that it's wrong? Will it get it right after a retry or two?

Sometimes by accident it will, but most of the time it keeps making things up with no regard to factual and logical correctness. It has no notion of truth.

Reply on news.ycombinator.com