Hacker News Reader

Comment by VBprogrammer | Hacker News Reader

I dunno, I recently learned about how the Toyota Synergy Drive system works in detail. Despite being the type of person who'd be interested in that kind of thing I thought it was not generally well understood, I've only come across one website which described it in detail.

I asked ChatGPT about it and it game me a fairly surface level answer. I then asked it for more details and it went into detail far beyond what the average person would be able to tell you. I thought that was quite impressive for a fairly obscure bit of mechanical engineering.

View on HN

That's exactly the wrong sort of thing to turn to ChatGPT for. In probing it for more detail, you're giving it increasingly helpful hints on how to answer your questions in a way that makes you satisfied, regardless of accuracy. That's what it's optimized to do. And because you're working from a humble and curious understanding of the topic yourself, it can and will make up all sorts of things that make you believe its insightful because you want to believe it.

It's the same technique a stage psychic uses to demonstrate that they knew all about your late uncle and his peculiar love of ducks.

To OP’s point what’s with the gate keeping on “right” or “wrong” ways to use it?

It’s a tool. It has flaws. It can be useful and if it is, have at it.

If there was a hammer that, held one way, drove nails perfectly in one blow, and held another, made it look like the nail was driven but actually broke it right at the junction so that the work pieces weren't fastened... I'd say that the second way was the wrong way to use that hammer.

Not only wrong, but dangerous, because nails are often used to fasten structural elements of houses, and incorrect but hard-to-detect flaws like this could result in collapse.

Similarly, if ChatGPT gives you an answer high in truthiness but low in accuracy, it could negatively impact you, whether loss in credibility if you repeat nonsense in front of someone knowledgable, or even worse if you use the incorrect knowledge to try to solve a real world problem.

It's a tool whose flaw is that sometimes it secretly does the exact opposite of what you're trying to do with it in a way that's only detectable if you didn't already need it.

It's like Babelfish, but for monoglots.

The issue is that approach maximizes its flaws. It’s like shooing yourself in the foot with a nail gun, that’s objectively the wrong thing to do.

GP stated that they asked ChatGPT about a topic they already were familiar with, from other sources. And that the answer made sense.

ChatGPT is optimized to seem to make sense, it’s like talking to a world class con artist you won’t notice it’s talking nonsense. Thus is subtly dangerous in ways that aren’t obvious in the beginning.

Sure it seems great at writing boilerplate code, unfortunately it’s really good and writing the kind of bugs that you don’t notice. This is maddening in a completely novel way, like working with an actively malicious autocomplete.

You can verify it’s accuracy. The metaphor of “con artist” is a poor one. It has no adversarial interest. A con artist does. Furthermore, con stands for confidence. A con artist manipulates confidence. Gpt and other language models are “what it says on the tin.”

Not understanding the fallibility of LLM is not the fault of the technology. It’s the user’s fault.

Not at all like a con artist. More like a tool that can be dangerous in the hands of the uneducated. That is a very large set of human tools, from nuclear reaction to pharmaceuticals to the automobile. Perhaps we just need “drivers ed” for large language models.

The problem is needing to verify accuracy makes it a really bad search engine replacement. A good con artist is generally truthful, they just know how to seem truthful even when they lie which exactly the issue with trusting ChatGPT etc. You can’t tell if the model is telling bullshit unless you already know the truth.

This doesn’t make them useless in general, just a poor fit if you can’t easily fact check the results.

Good point. I don’t think they will be a good search engine replacement, but maybe a powerful supplement for some users. And honestly with SEO and other issues even search engines can be manipulated in ways that an average user might have difficulty detecting.

Easy, you use a second source and double check.

In google you would go to another search result.

For ChatGPT you actually need to verify via Google or actually try it when it is code.

What’s the point of using ChatGPT when searching if you still need to verify the information through another search?

Conversely I asked it some simple questions about some songs' harmonic structures, and music theory, and it got things quite wrong.

There's already an example of it in the article where it incorrectly stated the tritone of F was Bb.

If you're able to validate the results though (so they logically fit together, or with that newfound vocabulary and pseudounderstanding can you search for that information elsewhere), is that really such a bad thing? I've been doing exactly that to quickly ramp up on some new numerical algorithms I previously didn't know existed.

That's a really big if.

Why? It's the same standard you need to apply to anything else.

In my experience from asking it questions about things that I know well, the deeper you get into the weeds with it, the more likely it is to invent some kind of extremely plausible-sounding but incorrect answer. How confident are you that what it told you is actually correct?

That is the one thing I think about when people say "I used it to learn X and it was very good" but when people are learning they are not in a position to to judge if what they learned is correct.

A lot of other stuff is just regurgitation of a wiki article.

Still it is very interesting and definitely a step forward.

And then we must remember that wiki articles themselves can also be quite wrong.

That's what I like to use it for. To come up with plausible answers for questions that people don't have correct answers for yet, and then probe the reasons behind it. It can be very creative and insightful in such uses.

Yep, this is how I've been using it too. The responses I get are often good starting points for further investigation. Often it comes up with better terms to find the answer that I am seeking than I had originally.

> It can be very creative and insightful in such uses

I once worked with a very smart woman who at that time worked in marketing for the airside retail stores at a large international airport.

If anyone from outside her immediate sphere asked her what she did, she would typically say "I tell lies for a living".

So it's a junior dev that reads lots of blogs?

This sounds like a lot of interviews I've conducted over the years.

> Despite being the type of person who'd be interested in that kind of thing I thought it was not generally well understood

Its an extremely well understood piece of technology and has been for decades outside of Toyota. Here is a 2005 paper that accurately models the fundamental behavior of the Prius HSD [1]. There are dozens of pages that explain it in various detail, including ones with live simulators of the power split device. [2] There are tons of forum threads of amateur Prius owners chatting about the internals.

I don't say this as a dunk, but to point out that a person's perception of chatGPT's capabilities are heavily influenced by their familiarity with the domain.

[1] https://www.engr.uvic.ca/~mech459/Pub_References/01500979.pd...

[2] http://eahart.com/prius/psd/

I think you and I have different definitions of not generally well understood. I would be impressed if more than 1 in 1000 people could describe it to the level of "it's a planetary gearbox with 2 electric motors used as inputs and outputs to control the effective gear ratio".

> I would be impressed if more than 1 in 1000 people could describe it

Q: What use is it attempting to measure ChatGPT's capabilities on something vs the average person on the street, or vs the average user with access to google.com? What does that actually tell us?

Maybe I'm showing my age, but if one wanted to learn or discover something, wouldn't one first typically try and find a reliable source/expert in that field or subject?

ahh, I misunderstood what you meant. In that case can you explain to me why its meaningful that chatGPT can sometimes explain things for which there are tons of resources online to learn about them but that a lot of people aren't familiar with? I legitimately just don't understand why that is an interesting bar to clear, especially given that it can also unpredictably lie about some aspects of the thing its explaining.

It would perhaps be interesting if it were such a hard topic that only 1 in 1000 could be capable of understanding it, but I would hazard a guess that most people familiar with the prius who don't understand its drivetrain probably don't understand it because they don't care, not because its too complicated to understand.

who cares? that's literally trivial

Out of curiosity, I'd have the following questions:

1. Would you be able to notice if it made something up, given that it's going beyond the detail that you know?

2. Did it have any details that aren't covered in: https://en.wikipedia.org/wiki/Hybrid_Synergy_Drive

what exactly do you think its going to make up? "The Hybrid Synergy Drive runs on apples and fairy dust"? I doubt that OP wants to build one of the engines, they just want a concise explanation.

I've seen gpt hallucinate all kinds of things. But very rarely for something like explaining a process or system.

It writes things that sound reasonable in context. So it will likely give something that sounds like a Popular Mechanics article about how some kind of machine works, but it may not really be this particular machine that you are asking about, or any actual machine in existence. That's the tricky thing about it, if it actually said "fairy dust" you know you can stop reading, but what it actually writes typically needs some actual understanding of the subject to refute.

Because it’s really trained off datasets you’d have access too via search engine, if you couldn’t find the info yourself, it’s likely just making a lot of the answers up, convincingly

Well it's essentially a (good) search engine then, isn't it? Somebody somewhere wrote that description, and chatgpt understood your query and fetched it.

A good search engine gives you references you can read, because they exist.

Reply on news.ycombinator.com