Before you believe the ChatGPT hype, ask it something you know well

At the end of the day a chatbot is a chatbot: it's producing plausible-sounding output from the input. GPT is just better at it than other chatbots. It still doesn't _know_ anything, or _have opinions_. It is like a person who's BSing: quick, talk about a subject, and sound like you know what it is! It's a farce to consider it as something other than that, which is also why I don't personally care about all the news about it (and agree with the OP: the hype is silly and tedious).

When ML models are trained on whether they give correct answers, rather than correct-sounding answers, I'll be interested in the bot they produce.

I mean, pose that question to some random dev with little exposure to second life and you'd probably get an even more anemic/generic response.

It's weird to me that people seem to expect current AI to be as good as an expert, especially at something as subtle as the intersection of society and technology. The hype around chatGPT is not that it's an expert in all things, it's that it's a novice in all things, and better than that at some.

I have done this about mechanical engineering and manufacturing things that I know well. With some things, its answers are about as good as a decent overview article that you might find with google after sorting through a few spammy ones. With other things, it can make remarkably dumb mistakes, but as has been pointed out here (and elsewhere), it's possible to give it some corrections and ask it to do further reasoning.

For instance, I asked it to list common defects in injection molding. It did a pretty good job of listing and describing the things that I'm usually looking out for. On the other hand, I asked it about how much a solid tire is typically stretched when placed on a rim (e.g. for an electric scooter). It came up with a typical percentage range, and then I tried to get it to reason out how the cross-sectional dimensions would change based on that stretching. It made several really basic errors that I managed to get it to correct, but in the end, I didn't really trust its ability and or "knowledge" in this area.

I have found it to be pretty relaxing to use as a recipe source, since it will just give you a recipe and the basic instructions without the fake history and 17 ads. It's pretty good at taking some vague requests about having certain ingredients and wanting something in a particular direction, and giving a reasonable result. Maybe if I was an expert on cooking particular types of food, I would find its answers comical or pedestrian.

My favorite was asking it "how can I solve a rubik's cube blindfolded". That's something I've been learning on and off for some time, and there's enough tutorials on the internet on how to do it.

ChatGPT's answer (summarized)[1]: "Memorize the initial state, and track the complete state of the cube in your head as you solve a cube normally".

If you aren't a cuber, the answer sounds plausible enough (It does work for chess after all), but it's humanly impossible with a Rubik's Cube. The only way you can memorize a random state of the cube _and_ operate on it, is to keep your changes to a minimal so your memorized state gets incrementally solved - you solve a few pieces at a time.

[1]: https://twitter.com/captn3m0/status/1598301110955311107

> The answer (see above) reminds me of that famous story about the amazing harpsichord-playing robot from the 19th century... The robot was the toast of Europe, giving command performances to royalty across the Continent. Until someone finally noticed the midget on a keyboard, hidden beneath the robot's dress.

Wait, how is ChatGPT like a famous robot secretly operated by a human?

Anyway, a lot of the recommendations it makes, he dismisses as being unfeasible or ineffective, but that doesn't mean ChatGPT is wrong. For one thing, it's based on a fallacious appeal to authority: I'm an expert, I disagree, so it's wrong. Or, in some cases: We tried a version of what ChatGPT suggests, and it didn't work, therefore it's bad advice. That's not necessarily the case, since there are plenty of reasons something might have failed in the past, like execution or timing.

Anyway, I know ChatGPT should not be trusted (it makes no claim to trustworthiness), I just don't like this reasoning.

ChatGPT is merely a statistical aggregate of what people says. So when you ask a question you get the common misconceptions and platitudes about the topic.

Systems like wikipedia and stackoverflow have (imperfect) social mechanisms to filter out useless or false information. ChatGPT have none of that. No miracle.

(for the record, the editorialized title that will likely get changed is right now: "Before you believe the ChatGPT hype, ask it something you know well")

Even better: ask it something you just made up.

For GPT, there's no real difference and you shouldn't expect any.

I'm not sure this is a good article, but I did have a similar sobering experience with ChatGPT. I asked it about a character I've been playing in a fighting game[1] and not only was its answer generally wrong - it seemed to have confused the character with some other character from another game. None of the move names describe things Baiken can do in Strive. I tried with Giovanna (another Strive character, but one who appears in no other games) and the answer was similarly extremely wrong. Pretty interesting!

[1] https://twitter.com/stsDrex/status/1611904716023418884

But I don't need chatgpt to teach me what I know. I need it to teach new things easier than by reading books or digging into the endless amount of pages from Google.

Those are things that I don't need to have a deep and accurate understanding. Usually, they are topics that I barely know how to start searching about them. So, for taking me from zero to enough so fast and intuitively, Chatgpt is really impressive to me.

If you know a lot about something that is only one degree removed from a Wikipedia stub article, ChatGPT gets really silly really fast.

I tried to have a conversation about S1000D governance, and it was making up whole new bodies of NATO governance in about two replies.

The naesayers need to realise this is only GPT4

Imagine what GPT6 is going to look like

What an absurd take. The author asked it something extremely theoretical, with no actual objective correct answers, it gave reasonable answers, but the author disagrees, so it's no good?

What?