Hacker News Reader

Comment by PaulHoule | Hacker News Reader

ChatGPT's superpower is hijacking the human desire for meaning into causing people to believe it is much better than it really is.

https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach

has dialogues between Achilles and the Tortoise where they work out the limitations of various fantastic A.I. systems based on mathematical logic. Roughly, neural networks don't repeal the results of Gödel, Turing and Tarski.

The short of it is that ChatGPT is good at charming people, better than some people are at charming people, but when it comes to getting accurate answers some of the things which it tries to do are logically impossible, so it is not like it gets it right 70% of the time now and they'll get that up to 100% but rather people will be puzzled about how it hit a plateau and we'll hear a few years later about what a disappointment it was just like Amazon's Alexa.

View on HN

I had a weird effect with it: I played around with it for about an hour and was pretty impressed - I can't really wrap my head around how it can do some of the things it does. Then I started to see the patterns, and got repeatedly disappointed by having to adjust and refine my prompts. Then I got downright nonsense back from it and gave up. Either I made it to some kind of uncanny valley, and it would get better if I invest more time, or most people who get amazed don't make it that far.

At this point I'm mainly concerned about the unimaginable heap of garbage it will release on the world - good enough to impress decision makers at first glance, annoying to deal with if you actually have to work with it.

I feel like it’s state can be corrupted and it ends up in some weird rut. Often by telling it to try again I can unwedge it but sometimes I have to reset the entire state and start the convo over again. Each time is slightly different. These seem like bugs that optimization heuristics can overcome (like maybe replaying the same convo repeatedly and choosing the highest information next prompt to present? Who knows. It’s basically magic and I’m waving coconuts hoping the planes will land)

Pretty much! It probably sounds lame, but the main reason I enjoy programming is that I can, if I take enough time, reason about everything that's going on and use that knowledge to build, fix and improve systems people get some value out of. For me, the magical part is the possibility to understand the magic.

I find it even hard to reason about models I have full access to, downright impossible if it's some blackbox on someone else's severs.

Maybe I'm outdated, I suppose time will tell.

I’ve been using it to teach me programming languages. If it plateaus here I’m delighted by its uniqueness and utility.

You seem to think people are hinting this is general ai. I don’t think that’s what’s amazing people.

I’ll bet you Siri and Alexa would be 100x more useful with a language model like this behind it. Part of their uselessness is the inability to generalize or reason, but relying instead in coded prompt and replies. Is that lost on the Alexa team (or what’s left)? I’m sure not. So, hey, I guess Alexa won’t plateau yet either eh?

There will come a time that it’s harder and harder to distinguish what’s AI and what isn’t. At a certain point, will it matter? Is the utility of these tools their ability to manifest human sentience? I don’t know why that’s useful to me - I already did that creating my daughter. What I want is a tool that enhances and extends and supplements my own mind and abilities to make me more effective in what I do.

And ChatGPT actually already does that. I can ask it questions about programming problems I’m having and it’ll largely give me a right answer - at least as often as a person would, and more reliably than I get on stack overflow. I’ll still use stack overflow, but I’ll filter my questions through assistants like this.

"100x more useful" is pure hyperbole. 2x maybe, but 100x is a symptom of exactly what's wrong about people's thinking about ChatGPT.

I would point to "There is no royal road to geometry"

https://en.wikiquote.org/wiki/Euclid

high-social status people have often been put off by quantitative reasoning because you spend years learning to do it, spend time gathering facts, thinking hard about things and often end up with tentative statements. High-social status people can skip all of that and get deference anyway the same way ChatGPT does.

Have you ever used Alexa? 2x more useful than alexa is as useful as my doorstop. It can’t understand most prompts, and defaults to reading a snippet from Wikipedia on some random subject then starts playing a pop song from Malaysia at volume 10 while ordering 60,000 incandescent black light bulbs with 2 hour shipping.

Even if this ends up in a plateau, that doesn't discount the fact that it's a huge step forward from what we were capable of building 5 years ago. If the next generation of language models make us believe they're superintelligent but are actually at the level of an average college student, that would still be an amazing achievement.

I have some level of concern that we have the same superpower.

ChatGPT's superpower is hijacking the human desire for meaning into causing people to believe it is much better than it really is.

It was a low bar to clear, given human celebrity culture.

We live in interesting times, and they're about to get a lot more interesting.

That wasn't Hofstadter's point. Hofstadter didn't try to "disprove" the possibility of having AI -- he only set out to show that certain approaches couldn't work. GPT's approach isn't one of those he argued against. You seem to be referring instead to Penrose and Lucas, but their arguments have convinced virtually no one.

My point is that you can relax constraints such as Gödel's incompleteness theorems if you are going to give it credit when it gets wrong answers.

It might be able to write a TV Show like Sliders but it won't be able to do real physics, for instance.

Reply on news.ycombinator.com