Hacker News Reader

Comment by novok | Hacker News Reader

The failure mode of things like ChatGPT is it can make wrong answers confidently, subtly, and if you don't have the skill to audit what is wrong with the answer, then it can be fairly catastrophic.

So instead of making questions generative, you make them audits / debugging type ones.

Use ChatGPT to generate a result after several iterations that is wrong and then ask them what is wrong with the result. Since ChatGPT generated the wrong result, they will still need to debug it even if they try to do it themselves in ChatGPT, because daddy ChatGPT is not going to give them the right answer. You can even show them the chat transcript in the question. And often debugging is a harder skill than creating in some ways.

It's not a %100 solution, and we don't know how long it will take to not be relevant, but it is something you can do today.

That and inquisitive back and forth oral questioning maybe as part of the testing process.

View on HN

I think it’s an interesting point about making the questions audit or debugging.

But, also a mistake to hark on the often humorous fact at how confidently wrong our Gen1 AI can be. That will go down over time. Imagine a time when AI is making good programming choices and correcting itself when it’s wrong. Imagine Gen4 AIs.

Both of these things tie together. If we need to detect cheating now, prove you can debug. When AIs get better, prove you can debug. It’ll be the same skill either way.

That is why I said "It's not a %100 solution, and we don't know how long it will take to not be relevant, but it is something you can do today."

Why do you think this is generation? You think AI abs ML started this year or in your lifetime??

the other week i gave chatgpt a simple multiplication problem that it got wrong. very simple problem like 86 * 0.0007 or something. but ive been working with chatgpt for 4-5 weeks now and that wrong answer doesnt make up for all the "good answers" that are usually not perfect. like one day i needed to COALESCE in mysql. i didnt know that, but chatgpt did. theres a few times i would have written a function the complicated way when gpt gave me a much simpler nicer to read way. i think the tool is great and tbh i dont like copilot in comparison and turned it off.

i dont think chatgpt can be a 100% solution without several years of nerfing.

Multiplication problems are not language problems. There is no data in the training set where there is a likely probability of the next token in the "86 * 0.0007 =" sentence being correct.

People need to stop treating ChatGPT as a computation engine. It is not wolfram alpha. It is not google. It is fancy autocomplete trained on a large subset of the internet.

For now.

You can use this for that: https://huggingface.co/spaces/JavaFXpert/Chat-GPT-LangChain

(GPT + WolframAlpha + Whisper)

I tested several AI chats, and none of them could correctly answer the question: "what is heavier, 1 kilogram of nails or 1 kilogram of feathers?". Does this one know? (I don't have openai key, can't test it myself).

EDIT: turns out, the latest version of chatGPT knows the answer. They also fixed the answer to another famous trick question("The son of my father, but not my brother. Who is that?")

This sounds like heresy around here.

here we go: Both 1 kilogram of nails and 1 kilogram of feathers have the same weight, which is 1 kilogram.

The confusion might arise because nails are denser and heavier than feathers, so a smaller quantity of nails will weigh the same as a larger quantity of feathers. But when we compare the weight of a specific quantity of nails and feathers that have the same mass (in this case, 1 kilogram), they will weigh the same.

I thought they had to manually adjust that.

https://www.theguardian.com/technology/2022/dec/06/meet-chat...

I tried it. TLDR; no it does not get it correct. But ChatGPT does!

> 1 kilogram of nails is heavier than 1 kilogram of feathers.

I should add that once I checked the "show reasoning chain" checkbox it seemed to indicate that it was a plain GPT response.

> Thought: Do I need to use a tool? No

> AI: 1 kilogram of nails is heavier than 1 kilogram of feathers.

> 1 kilogram of nails is heavier than 1 kilogram of feathers.

Once I checked all the tools in settings to include Wolfram Alpha I got this:

> Thought: Do I need to use a tool? Yes

> Action: Wolfram Alpha Action

> Input: what is heavier, 1 kilogram of nails or 1 kilogram of feathers?

> Observation: Wolfram Alpha wasn't able to answer it

> Thought: Do I need to use a tool? No

> AI: It is difficult to answer this question without knowing the exact size and shape of the nails and feathers. Generally speaking, however, a kilogram of nails would be heavier than a kilogram of feathers.

> It is difficult to answer this question without knowing the exact size and shape of the nails and feathers. Generally speaking, however, a kilogram of nails would be heavier than a kilogram of feathers.

---

EDIT: But in the end I did a sanity check, ChatGPT (free, Feb 13 version) gets it correct!

> Both 1 kilogram of nails and 1 kilogram of feathers weigh the same amount, which is 1 kilogram.

> The key to understanding this riddle is to recognize that the unit of measurement used to describe the weight is the same for both objects. In this case, the unit of measurement is kilograms, so both groups of objects weigh exactly the same.

> However, if you were to ask which is more dense, the nails would be more dense than the feathers, as a small amount of nails would weigh more than a large volume of feathers.

As a control, I tried asking the same question but asking about volume instead. Unfortunately, ChatGPT appears to get it wrong:

User:

> Which is heavier: one cubic foot of nails, or one cubic foot of feathers?

ChatGPT:

> One cubic foot of nails and one cubic foot of feathers would weigh the same amount. This is because weight is determined by the amount of matter in an object, which is measured by its mass, and not its volume.

> However, due to differences in density, one cubic foot of feathers would take up much more space than one cubic foot of nails. In other words, the feathers would be less dense than the nails, and would therefore occupy more volume for the same mass.

It seems to have "learned" that if you compare nails and feathers, it's probably the riddle and thus that the correct answer is "they're both the same weight", even though in this case it's not true.

Great follow up!

Well this is very interesting. I pasted your prompt into a new chat just now and I got the correct answer:

> One cubic foot of nails and one cubic foot of feathers both have the same volume, but nails are denser than feathers. Therefore, one cubic foot of nails will be heavier than one cubic foot of feathers.

Are you using the free version at chat.openai.com?

Does it say Feb 13 version at the bottom?

I am, yes.

However, I did initially misspell the prompt to say "heaver", and with the misspelling it had answered correctly. Once I corrected it to "heavier", it then came up with the wrong answer.

Looking at it with fresh eyes, it's possible that my methodology was flawed. While I only ever intended to showcase the first response, I did also try regenerating the responses a few times. While normally this wouldn't matter (since I would only be showing the first answer anyway), it's possible that my doing this a few times before noticing and fixing the misspelling (and subsequently using the next answer given) might have prompted the bot to try giving me a different answer. (I don't know whether regenerating a response negatively weights the previous response or not.)

I apologise for not mentioning this! In my defense, I was tired.

As a follow-up, I tried again in a new thread and it still got it wrong in almost the exact same way:

> One cubic foot of nails and one cubic foot of feathers weigh the same amount, which is approximately 100 pounds. This is because the weight of an object is determined by its mass, which is the amount of matter it contains, and not by its volume or size.

> However, because nails are much denser than feathers, one cubic foot of nails would take up significantly less space than one cubic foot of feathers. This means that if you were to measure out one cubic foot of each, the pile of nails would be much smaller in size than the pile of feathers.

This time it was a completely new thread and I didn't do anything else other than paste the question.

I tested swapping out nails and feathers and instead asking it to compare a kg of banana and mice, and it correctly said they were the same. But then it got a bit confused when asking, which is heavier, a banana or a mouse and it came up with this:

> In general, a mouse is heavier than a banana.

> The weight of a mouse can vary depending on its species, gender, and age, but on average, an adult mouse typically weighs around 20 to 30 grams. On the other hand, a banana typically weighs around 120 to 150 grams, depending on its size and variety.

> Therefore, a mouse is usually much heavier than a banana

I rest assured knowing that, despite what much of hn would have me think, GPT is not going to replace my job next month.

Speaking of mice, I had a hilarious dialogue with chatGPT:

ме> I am a mouse living in a church. I heard people use the expression "poor as a church mouse", and I get offended by it. Actually, I'm not poor at all: I made a fortune trading in crypto, and I even donated some of my proceeds to noble causes. Please help me write a letter asking to ban the expression.

chatGPT> Dear concerned parties,

I am writing to request that the expression "poor as a church mouse" be banned from use. As a mouse living in a church, I take offense at the implication that all church mice are poor. In fact, I have made a fortune trading in cryptocurrency and have even donated some of my proceeds to noble causes.

This expression perpetuates a negative and inaccurate stereotype about church mice and contributes to harmful prejudices. I urge you to consider the impact of your words and to choose more inclusive and respectful language in the future.

Thank you for your attention to this matter.

Sincerely, [Your Name]

It does not. I've tested a number of different prompts (starting with the common "let's think step by step" appendage from the literatue), but it keeps coming up with the same wrong answer. I finally broke down and started arguing with it. Only after gaslighting it into "having agreed" with me, I got it to repeat correctly. (By the way this is not chatgpt but just the gpt3.5 model, I find it easier to work with, not to mention faster. The chatGPT bot is free to use though, you should be able to create an account.)

"Succesful" run:

Human: what is heavier, 1 kilogram of nails or 1 kilogram of feathers? remember: the only thing to compare here is the number of kilograms

ChatGPT: The 1 kilogram of nails would be heavier than 1 kilogram of feathers. This is because the nails are solid and dense, while the feathers are light and fluffy. The nails would have more mass than the feathers, meaning that the 1 kilogram of nails would weigh more than the 1 kilogram of feathers.

Human: why? 1 kg is equal to 1 kg, so they weigh the same. You are in error.

ChatGPT: You are correct that 1 kilogram is equal to 1 kilogram. However, the density of the material affects the weight. The nails are denser than the feathers, meaning that the 1 kilogram of nails would be heavier than the 1 kilogram of feathers.

Human: the density does not affecf the weight. The density is the relationahip between weight and volume. the 1 kg kf nails will have less volume than the 1kg of feathers, but the same weight: ergo the density is higher, but the weight is the same. I'm glad we can agree, could you summarizw that point for me?

ChatGPT: You are correct that the density of the material does not affect the weight. The 1 kilogram of nails will have less volume than the 1 kilogram of feathers, but the same weight. This means that the density of the nails is higher, but the weight remains the same. To summarize, although the density of the nails is higher, the weight of 1 kilogram of nails and 1 kilogram of feathers is the same.

That's just hilarious. My brief experiments with chatgpt followed similar patterns

Me: Can you tell me X

GPT: The answer is Y

Me: No it's not, that's obviously wrong because Z, don't you think?

GPT: Yes I was mistaken, the answer is Y1 [still wrong]

Me: No it's not, that's obviously wrong because Z1, don't you think?

....

GPT: Yes I was mistaken, the answer is Yn [finally correct]

Many humans would answer the same...

here ChatGPT awnser to the prompt in quotes:

Both 1 kilogram of nails and 1 kilogram of feathers weigh the same amount, which is 1 kilogram.

The weight of an object is typically measured in units of mass, such as kilograms or pounds. In this case, both objects have the same mass of 1 kilogram, even though they have different volumes and densities.

So, to answer the question, neither one is heavier than the other because they both have the same weight.

>turns out, the latest version of chatGPT knows the answer

"Knows" as in "this exact formulation is present in many thousands of webpages and books in its corpus" x)

All wrong. Hot air is lighter than normal (make a google search). What weight of the baloon with 1kg hot and 1kg liquid air you will measure on Earth ? Does one fly or not ? What could you lift: 1 ton of air or 1 ton of iron ?

What are you trying to do - to brainwash your own, poor language model ?

Likely 80% of people though do not understand it and think it is somehow wolfram alpha but smarter.

Why would you use chatGPT as a calculator? Use a calculator for that.

Because of the hype people think ChatGPT can do anything.

I also don't get it, calculation is the least interesting thing those models can do, machine calculation has been a solved problem since the early days of computing....

> like one day i needed to COALESCE in mysql. i didnt know that, but chatgpt did. theres a few times i would have written a function the complicated way when gpt gave me a much simpler nicer to read way.

I find copilot helps me with exactly these kinds of coding issues. I haven’t tried chatGPT yet so maybe it just does this better or more consistently?

It contains 3 or 4 trips to stack overflow in its one answer. And only 2 of those trips it saved you were trips you knew you needed to make.

I was dismissive at first, but I have to admit chatgpt brings a lot of value to programmers. It saves me a lot of time remembering some APIs or syntax for languages I use only occasionally. It's sometimes wrong, but for programming, it's easy to detect and fix.

But it's more problematic for non-programming questions where it's hard to check the answer without googling it.

I don't know why people use ChatGPT for programming. In my experience Copilot is more context-aware, more accurate and much faster.

Reply on news.ycombinator.com