Hacker News Reader

Comment by _huayra_ | Hacker News Reader

I can't tell what is worse now: the sycophantic ChatGPT hype guys/gals who write articles "it's coming for all of our jerbs!", or articles like this one that deliberately misuse ChatGPT and then say "it's overhyped".

They're both missing the point.

Yes, ChatGPT can be tricked, confidently give wrong answers, but it is still ludicrously useful.

It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field. Does this "person" make mistakes? Can't cite their sources? Yeah this definitely happens (especially the sources thing), but when you're trying to understand something new and complex and you can't get the "gist" of it, ChatGPT does surprisingly well.

I've had it debug broken configs on my server and router (and explain to me why they were broken), help me practice a foreign language I've slowly been forgetting (hint: "I would like to practice $language, so let's have a conversation in $language where you only use very simple words." -> ChatGPT will obey), and help me understand how to use obscure software libraries that don't have much documentation online (e.g. Boost Yap, useful but with a dearth of blog / reddit posts about it).

Does it sometimes goof up? Yep, but it is such an incredibly useful tool nonetheless for the messy process of learning something new.

View on HN

I don't know that it's an "incredibly smart" anything; I think it'd be better described as incredibly well-read, with poor discretion.

But there's definitely uses for a tool that produces a lot of boilerplate content that a human can then proofread (ex: writing cover letters, filling out unstructured paperwork, etc.)

Edit: I personally underestimated the use-case, because the "more optimal" thing to do when auto-filling out paperwork would be to define a formal data-structure with annotations for each field, and then have a system read annotations and spit out autosuggestions. But this is a lot of overhead.

If you can have a system that takes in arbitrary text, parses it, and fills it out with pretty good guesses, then that can be much more viable than the previous approach, as it doesn't require an overhaul of, say, the way the US bureaucracy generates paperwork. It's all on the consumer-end, the producer doesn't need to cooperate.

> I don't know that it's an "incredibly smart" anything; I think it'd be better described as incredibly well-read, with poor discretion.

But likely much better read and with much better discretion that most of people writing content these days.

Which I think is part of the point that people are missing. ChatGPT’s output might not be as good as the best writing, but most of what people read is mediocre content with questionable accuracy. Being better than the majority of the writing being consumed is still a pretty big deal, even if it’s not better than all of the writing that’s being consumed. Especially since we can expect ChatGPT to get better.

The same is true for AI art. Could you hire a trained artist and work with them to do a better job? Probably, but most people don’t have the time and money to pursue such a project.

There are still trained artisans producing high quality physical goods that you can place an order with. But the masses have greatly benefited from the factories that have produced lower quality products that are much more widely available, and available at a moments notice. It’s not wrong to point out when the mass produced stuff might not have the same quality as the products produced. But if you only focus on this aspect you miss the reason why mass production is important.

> ChatGPT’s output might not be as good as the best writing, but most of what people read is mediocre content with questionable accuracy.

I agree with this, but I think we may be missing the point.

Why are people reading mediocre content with questionable accuracy?

The problem is not: how to quickly and cheaply create content which is marginally less mediocre and marginally more accurate than what we currently have.

> Why are people reading mediocre content with questionable accuracy?

For most people that consume content, accurate information is not a priority. Being entertained, belonging to a group, or satisfying some immediate need or want matter more for most people, most of the time.

It's doubtful that people who consume content will change their wants to prize accuracy (most of the time). Partly, this is because accurate, precise, and/or qualified statements are often boring and hard to follow.

Consider also incentives that drive content creators: 1) the internet means many people can find a platform for many messages (democratization of speech is supposed to be a virtue of the internet) 2) the "freemium" advertising model means that most messages can be financially rewarded, regardless of accuracy

Engagement and financial rewards are much easier to get with lies, drama and controversy than with accuracy, precision, and qualified statements. In my experience, most people cannot find something worth striving for in the modern world, and settle on whatever comes easiest.

> The problem is not: how to quickly and cheaply create content which is marginally less mediocre and marginally more accurate than what we currently have

I'll grant that this is not a problem you're trying to solve. However, this sounds like exactly the arms race that content creators are engaged in until it stops paying.

>> The problem is not: how to quickly and cheaply create content which is marginally less mediocre and marginally more accurate than what we currently have

> this sounds like exactly the arms race that content creators are engaged in until it stops paying

I don't think they're trying to achieve these things at all. Look at the broken incentives you raised in the first part of your response, that's what this is about, and all this is about.

Content creators mostly don't care about how mediocre their output is, and they mostly don't care about how accurate it is either, as long as the clicks keep coming.

We don't need (and can't expect) ChatGPT to "fix" this, the underlying problem is the broken incentives, not the actual quality - or lack of it - in the content.

I don't disagree, with one caveat. I'm still curious about your opinion on your own question:

> Why are people reading mediocre content with questionable accuracy?

How would you answer this?

I don't believe bad incentives explains it fully.

"Why are people reading mediocre content with questionable accuracy?"

Because it's literally everywhere you look and is very hard to filter due to huge amount of it everywhere. Particularly if you are trying to learn and grow on a certain subject, it's difficult to filter out what you don't know is 'mediocre' or 'questionable' at least not until you've already consumed it.

A lot of it is for entertainment, even when it’s infotainment. News reporting, Wikipedia, current affairs podcast, Tweets, Reddit, Hacker News comments, etc. The inaccuracies here aren’t that important, because the vast majority of this content is just being consumed for dopamine hits and not actually putting the information to use.

Some of it is out of necessity. I don’t think it’s controversial to say that many people use Stack Overflow, and though some of the answers there are very good, others aren’t. If someone needs help with a specific problem they’ll often post a question somewhere (Reddit, Stack Overflow, etc.), and the quality of the answer they get is a roll of the die - even whether they’ll get an answer at all.

Some of it is because blog spam has made traditional searches much more difficult. There’s another comment that says that there are comments better than Reddit and Hacker News, but if you go into a discussion about Google you’ll find that a lot of people have been doing things such as appending “reddit.com” to searches because otherwise they’ll drown in a mountain of blog spam.

I dunno, I recently learned about how the Toyota Synergy Drive system works in detail. Despite being the type of person who'd be interested in that kind of thing I thought it was not generally well understood, I've only come across one website which described it in detail.

I asked ChatGPT about it and it game me a fairly surface level answer. I then asked it for more details and it went into detail far beyond what the average person would be able to tell you. I thought that was quite impressive for a fairly obscure bit of mechanical engineering.

That's exactly the wrong sort of thing to turn to ChatGPT for. In probing it for more detail, you're giving it increasingly helpful hints on how to answer your questions in a way that makes you satisfied, regardless of accuracy. That's what it's optimized to do. And because you're working from a humble and curious understanding of the topic yourself, it can and will make up all sorts of things that make you believe its insightful because you want to believe it.

It's the same technique a stage psychic uses to demonstrate that they knew all about your late uncle and his peculiar love of ducks.

To OP’s point what’s with the gate keeping on “right” or “wrong” ways to use it?

It’s a tool. It has flaws. It can be useful and if it is, have at it.

If there was a hammer that, held one way, drove nails perfectly in one blow, and held another, made it look like the nail was driven but actually broke it right at the junction so that the work pieces weren't fastened... I'd say that the second way was the wrong way to use that hammer.

Not only wrong, but dangerous, because nails are often used to fasten structural elements of houses, and incorrect but hard-to-detect flaws like this could result in collapse.

Similarly, if ChatGPT gives you an answer high in truthiness but low in accuracy, it could negatively impact you, whether loss in credibility if you repeat nonsense in front of someone knowledgable, or even worse if you use the incorrect knowledge to try to solve a real world problem.

It's a tool whose flaw is that sometimes it secretly does the exact opposite of what you're trying to do with it in a way that's only detectable if you didn't already need it.

It's like Babelfish, but for monoglots.

The issue is that approach maximizes its flaws. It’s like shooing yourself in the foot with a nail gun, that’s objectively the wrong thing to do.

GP stated that they asked ChatGPT about a topic they already were familiar with, from other sources. And that the answer made sense.

ChatGPT is optimized to seem to make sense, it’s like talking to a world class con artist you won’t notice it’s talking nonsense. Thus is subtly dangerous in ways that aren’t obvious in the beginning.

Sure it seems great at writing boilerplate code, unfortunately it’s really good and writing the kind of bugs that you don’t notice. This is maddening in a completely novel way, like working with an actively malicious autocomplete.

You can verify it’s accuracy. The metaphor of “con artist” is a poor one. It has no adversarial interest. A con artist does. Furthermore, con stands for confidence. A con artist manipulates confidence. Gpt and other language models are “what it says on the tin.”

Not understanding the fallibility of LLM is not the fault of the technology. It’s the user’s fault.

Not at all like a con artist. More like a tool that can be dangerous in the hands of the uneducated. That is a very large set of human tools, from nuclear reaction to pharmaceuticals to the automobile. Perhaps we just need “drivers ed” for large language models.

The problem is needing to verify accuracy makes it a really bad search engine replacement. A good con artist is generally truthful, they just know how to seem truthful even when they lie which exactly the issue with trusting ChatGPT etc. You can’t tell if the model is telling bullshit unless you already know the truth.

This doesn’t make them useless in general, just a poor fit if you can’t easily fact check the results.

Good point. I don’t think they will be a good search engine replacement, but maybe a powerful supplement for some users. And honestly with SEO and other issues even search engines can be manipulated in ways that an average user might have difficulty detecting.

Easy, you use a second source and double check.

In google you would go to another search result.

For ChatGPT you actually need to verify via Google or actually try it when it is code.

What’s the point of using ChatGPT when searching if you still need to verify the information through another search?

Conversely I asked it some simple questions about some songs' harmonic structures, and music theory, and it got things quite wrong.

There's already an example of it in the article where it incorrectly stated the tritone of F was Bb.

If you're able to validate the results though (so they logically fit together, or with that newfound vocabulary and pseudounderstanding can you search for that information elsewhere), is that really such a bad thing? I've been doing exactly that to quickly ramp up on some new numerical algorithms I previously didn't know existed.

That's a really big if.

Why? It's the same standard you need to apply to anything else.

In my experience from asking it questions about things that I know well, the deeper you get into the weeds with it, the more likely it is to invent some kind of extremely plausible-sounding but incorrect answer. How confident are you that what it told you is actually correct?

That is the one thing I think about when people say "I used it to learn X and it was very good" but when people are learning they are not in a position to to judge if what they learned is correct.

A lot of other stuff is just regurgitation of a wiki article.

Still it is very interesting and definitely a step forward.

And then we must remember that wiki articles themselves can also be quite wrong.

That's what I like to use it for. To come up with plausible answers for questions that people don't have correct answers for yet, and then probe the reasons behind it. It can be very creative and insightful in such uses.

Yep, this is how I've been using it too. The responses I get are often good starting points for further investigation. Often it comes up with better terms to find the answer that I am seeking than I had originally.

> It can be very creative and insightful in such uses

I once worked with a very smart woman who at that time worked in marketing for the airside retail stores at a large international airport.

If anyone from outside her immediate sphere asked her what she did, she would typically say "I tell lies for a living".

So it's a junior dev that reads lots of blogs?

This sounds like a lot of interviews I've conducted over the years.

> Despite being the type of person who'd be interested in that kind of thing I thought it was not generally well understood

Its an extremely well understood piece of technology and has been for decades outside of Toyota. Here is a 2005 paper that accurately models the fundamental behavior of the Prius HSD [1]. There are dozens of pages that explain it in various detail, including ones with live simulators of the power split device. [2] There are tons of forum threads of amateur Prius owners chatting about the internals.

I don't say this as a dunk, but to point out that a person's perception of chatGPT's capabilities are heavily influenced by their familiarity with the domain.

[1] https://www.engr.uvic.ca/~mech459/Pub_References/01500979.pd...

[2] http://eahart.com/prius/psd/

I think you and I have different definitions of not generally well understood. I would be impressed if more than 1 in 1000 people could describe it to the level of "it's a planetary gearbox with 2 electric motors used as inputs and outputs to control the effective gear ratio".

> I would be impressed if more than 1 in 1000 people could describe it

Q: What use is it attempting to measure ChatGPT's capabilities on something vs the average person on the street, or vs the average user with access to google.com? What does that actually tell us?

Maybe I'm showing my age, but if one wanted to learn or discover something, wouldn't one first typically try and find a reliable source/expert in that field or subject?

ahh, I misunderstood what you meant. In that case can you explain to me why its meaningful that chatGPT can sometimes explain things for which there are tons of resources online to learn about them but that a lot of people aren't familiar with? I legitimately just don't understand why that is an interesting bar to clear, especially given that it can also unpredictably lie about some aspects of the thing its explaining.

It would perhaps be interesting if it were such a hard topic that only 1 in 1000 could be capable of understanding it, but I would hazard a guess that most people familiar with the prius who don't understand its drivetrain probably don't understand it because they don't care, not because its too complicated to understand.

who cares? that's literally trivial

Out of curiosity, I'd have the following questions:

1. Would you be able to notice if it made something up, given that it's going beyond the detail that you know?

2. Did it have any details that aren't covered in: https://en.wikipedia.org/wiki/Hybrid_Synergy_Drive

what exactly do you think its going to make up? "The Hybrid Synergy Drive runs on apples and fairy dust"? I doubt that OP wants to build one of the engines, they just want a concise explanation.

I've seen gpt hallucinate all kinds of things. But very rarely for something like explaining a process or system.

It writes things that sound reasonable in context. So it will likely give something that sounds like a Popular Mechanics article about how some kind of machine works, but it may not really be this particular machine that you are asking about, or any actual machine in existence. That's the tricky thing about it, if it actually said "fairy dust" you know you can stop reading, but what it actually writes typically needs some actual understanding of the subject to refute.

Because it’s really trained off datasets you’d have access too via search engine, if you couldn’t find the info yourself, it’s likely just making a lot of the answers up, convincingly

Well it's essentially a (good) search engine then, isn't it? Somebody somewhere wrote that description, and chatgpt understood your query and fetched it.

A good search engine gives you references you can read, because they exist.

If you can think of a single individual who can demonstrate knowledge on the fly with the same accuracy and breadth as ChatGPT, I think most of us would agree in calling them “incredibly smart.”

If ChatGPT was a person, you'd immediately realize something was very wrong with them. Some kind of idiot savant perhaps who'd memorized an encyclopedia but was confused by how it all fit together, or maybe someone hallucinating and delusional, only able to hold coherent thoughts together for brief stretches of time before babbling gibberish that made no sense.

My gosh. Enough with the anthropomorphizing.

Yeah, I think making a decent, nuanced analysis of ChatGPT and related systems (which the gp starts trying to do) requires one to avoid terms like smart/intelligent/etc. The term intelligence is naturally not fully defined and primarily used by humans to rank each other. So our human intuition about intelligence comes from the ranking we do of people. IE, we have the intuition that "if they can do X, they can easily do Y" about a given person and such an intuition clearly shouldn't carry over to even a very complex machine.

> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field.

No. You've fallen for exactly the con the article is describing.

ChatGPT knows nothing. It is very good at generating text that seems plausible if you know nothing about the subject matter, but is not reliable as an actual source of information.

If you happened to get information from it that turned out to be ok, you were lucky. Don't expect your luck to last.

I was surprised to find ChatGPT does a better job of translating than Google translate, at least for one difficult task I needed to get done.

I have been a Google Translate user for more than a decade. I use it a dozen or more times a day for multiple languages. It is a very useful tool.

I recently had to translate a passage that included this phrase: 六六大顺

Google Translate gave me: Liuliu Dashun (basically no translation, just a transliteration or romanization)

Bing Translator: Six Six Dashun (a little bit better because 六 does mean six in some contexts.

Baidu: Liu Liu (basically no translation)

Naver: Six or Six Dashun (partial translation, plus added an incorrect particle)

Only ChatGPT gave the correct answer: "Everything going smoothly"

There is a cultural context that is essential to the translation, which apparently only ChatGPT has. It turns out that the number 6 is a lucky number in Chinese. It has the meaning of being “auspicious”. In Chinese culture, there is an idiom called “六六大顺 (liù liù dà shùn)”, which means “Everything comes in a smooth way” or more naturally "Everything going smoothly".

BTW, after seeing the ChatGTP results, I found this contextual info using Google search, in a website http://www.ichineselearning.com

It is ironic that Google search "knows" (helps you find) the cultural context but Google Translate apparently does not leverage this info.

Seems far more likely to me that ChatGPT knows the meaning of specific idioms rather than knowing the cultural context of that number and using it in the translation.

Just to give additional context, the full passage I was trying to translate was a New Year's greeting as follows:

里祝愿在新的2023年风调雨顺，四季平安，五福临门，六六大顺，十全十美，万事如意，好运连绵，兔年吉祥

This passage has some nuance and wordplay, due to the stylistic device of starting each of the key phrases with a character that can represent a number: 四 (four), 五 (five), 六 (six), 十 (ten) and 万 (ten thousand). These characters also have meaning as words within each phrase. For example, 十 can convey perfection, in addition to the numerical quantity.

So here is Google's lumbering and broken translation attempt:

"Wishing you good luck in the new year of 2023, safe all year, Five Blessings, Liuliu Dashun, perfect, May all go well with you, good luck Auspicious Year of the Rabbit"

Baidu does a better job (including handling the problematic 六六大顺 phrase:

"I wish the new year 2023 a good year, Four seasons are safe, Five blessings are at the door, Everything goes smoothly, perfect, Everything goes well, Good luck, Auspicious Year of the Rabbit"

Neither of these is as good as the result from ChatGPT:

"I wish you a smooth wind and rain in the new year of 2023, peaceful seasons, five blessings at your doorstep, everything going smoothly, perfection in all things, everything going according to your wishes, continuous good luck, and a lucky Year of the Rabbit"

ChatGPT is not perfect. I think the phrase 四季平安 is better translated as "four seasons of peace" (this one came from the translator from Korean company Naver).

Yes, one would think. However, if it were a matter of specific idioms, the other translation programs would capture this (Google, Bing, Baidu or Naver). But, for some reason, none of the others did. I think perhaps it is not that common an idiom. Or perhaps ChatGPT was just lucky.

> It turns out that the number 6 is a lucky number in Chinese. It has the meaning of being “auspicious”.

isn't that number 8? (not number 6)

Cultures can have multiple lucky numbers!

Correct. Eight is by far the most common lucky number in Chinese, but six is also lucky. But I think it has to be repeated multiple times. Apparently the number 9 is also lucky.

I wonder, does it get the meaning of "10-4" when translating English to other languages?

It does.

GPT Chat: 10-4 is a phrase commonly used in radio communication that means "message received and understood." It is typically used to acknowledge a message or request.

Google does too https://en.wikipedia.org/wiki/Ten-code

I think that's unnecessarily pessimistic view. I have recently started using ChatGPT for mid-hard questions (when I can't get an answer immediately using Google), mostly "how do I" questions. It lies of dreams up fake answers about half of the time, but this still means that 50% of time I get my solution without digging through obscure manuals.

> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field.

I like to think about it like asking questions to a well read but dishonest student during an oral exam. They know some things, and try really hard to answer every question, even resort to making things up in hope of getting a passing grade.

So it's a coinflip whether it's giving you correct information or something completely made up? And now you're not digging through the obscure manuals to actually verify? Seems objectively harmful.

Actually, that is extremely beneficial.

For example, I now use GPT3 for every single code debugging problem that I have.

I input the bug/problamatic code into the service, ask it for an answer, and then I take the code and see if it fixes my problem I was having.

It doesn't work half the time, but thats fine. Then I just figure out the bug the normal way.

But the other half the time, it immediately solves my problem, and then I run my code and it works/ does what I want.

I sincerely hope you work on some random SaaS and not any software that actually matters, because this is how you get subtle dangerous bugs.

It seems to me the utility of the 'good' 50% depends entirely upon your ability to recognize it.

How do you know when to dig through those obscure manuals?

It seems to me that it'd be more useful when:

a) I don't care whether the text says things which are not true

Write me a poem about ...

b) I care, and will have to verify it's results, but that cost is worth it

Generate some test data ...

I think we're at a very dangerous intersection between an apparent decline in American's ability to detect nonsense and an automated way to create convincing nonsense.

How many hours you spent trying ChatGPT out? I spent at least high tens, maybe even hundreds. You're absolutely wrong. Yes, it hallucinates, yes its wrong about obscure topics - but calling having success with it luck is absolutely wrong. It's very consistently good. Especially about things like programming, physics, math - and now I'm using it as my teaching assistant for my pilot training, it's perfect (and I can very simply verify the answers are good with my FAA Pilot Handbook, don't worry).

> I can very simply verify the answers are good with my FAA Pilot Handbook

Thank you for agreeing with my point.

If you need to check the answers with your FAA Pilot Handbook, wouldn't it be simpler just to read the FAA Pilot Handbook? That handbook, unlike the text generated by ChatGPT, was written by a process that was aware of the semantic relationship between the text being written and the things in the world that the text was referring to. That is what makes the FAA Pilot Handbook a reliable source of information about flying. ChatGPT is not any such process. That's why it is not a reliable source of information--as you agree, since you need to check what it says about flying with the FAA Pilot Handbook.

> If you need to check the answers with your FAA Pilot Handbook, wouldn't it be simpler just to read the FAA Pilot Handbook?

No, absolutely not. It's much easier to verify a couple specific pieces of information that you're unsure of then go hunting through a large corpus of information trying to pick out the bits that are specific to what you want.

I've used ChatGPT across lots of different areas and I find it incredibly useful. I'm not blindly trusting what it spits out, but it's pretty simple to verify what it's saying. While I definitely do have concerns about the impacts of ChatGPT on a societal level, and what will happen when so much computer-generated content can flood the Internet, but, at a personal level, the complaint that ChatGPT "bullshits with confidence" is not really that much of a problem for what I use it for.

Edit: To give a specific, real-world example, there was a post recently about using ChatGPT to replace a SQL Analyst. Now, ChatGPT definitely will and did churn out wrong answers, but it was incredibly useful to use as a starting point for some complex queries. When it fails, it tended to fail in pretty obvious ways, and to the complaint that it can fail in more subtle ways that look correct, I've certainly dealt with tons of human-generated queries that had the same issues. Are those all useless?

I don't think I agreed with anything you said.

I work with all available material in many different ways (Anki cards, videos, different explanations of the same thing, etc), and ChatGPT is another way to learn and help me generate learning material. For example I have it ask me questions like a tutor would. Or I ask various questions when I'm unsure about the wider context - e.g. it provides much more about the underlying physics than the Pilot Handbook itself. If I don't understand, I can ask for clarification, or an explanation like I am 5.

Reading the Pilot Handbook is a big part of learning but being limited to it would be hard. I'm very happy about having Chatgpt available.

Perhaps you and the poster are approaching your evaluations from different points of view. I've found that if I set out to break ChatGPT I can very easily do it. If my goal is to look for mistakes or find a failure case it's almost trivial to do so.

At the same time if I'm looking for success I normally find it.

Essentially if you work cooperatively with the tool then you'll find it useful, if you are antagonistic towards it you can also have success in breaking it.

I asked it to write some simple code do do a task. It confidently told me to use a library and use some functions in it.

Couldn’t get it to work. Couldn’t get any examples out of google of it being sued that way. Eventually looked through the code of the library and found that while some functions worked int he way chatgpt was trying, the functions it had selected didn’t work, didn’t support those arguments, and never had.

> Especially about things like programming, physics, math

I routinely find ChatGPT giving me completely made-up APIs and incorrect explanations when it come to programming. And I haven't found it much better with math. Sorry, I don't buy it. Maybe it's good at training pilots, and if so great, but it's wrong enough for me that it's hard to trust in general.

> If you happened to get information from it that turned out to be ok, you were lucky.

This is just saying that sometimes ChatGPT is right and sometimes it's wrong, with an implicit stance on the relative frequencies.

In my own experience it would be more accurate to say you're unlucky if it gets it wrong—but then much of this hinges on having a good sense of what it's going to be good at / useful for.

(The other thing I see people miss all the time when assessing its correctness: it is somewhat "on demand"—it may include bits that are vague and only correct-ish, but if you point out an error or request refinement you can generally get it to clear/tighten things up. It's a bit like using a Google Maps-like zooming UI, but in addition to being variably abstract it's always approximating: the potential for error is ever present.)

> This is just saying that sometimes ChatGPT is right and sometimes it's wrong

No, it's saying that ChatGPT is not reliable. It's not reliable because getting things right or wrong is not part of its design at all. All it does is generate text that matches a pattern. There is nothing anywhere in it that filters or evaluates text based on any semantic connection with anything else.

> No, it's saying that ChatGPT is not reliable

That's another way of saying sometimes it is right and sometimes it is wrong...

> All it does is generate text that matches a pattern

This is about as meaningful as saying, "all things are is collections of atoms".

We are barely beginning to explore the implications of these pattern-extending structures: their most interesting features emerge from their structure, cannot be simply derived from the fact that they do pattern continuation in the same way you're not gonna derive the concept of friendship from analyzing chemical bond structure.

There's lots of real knowledge encoded in ChatGPT and it can recombine and regurgitate it in enough ways that I'd be okay saying that it knows stuff. The real problem is that it doesn't know its limits and will make shit up instead of saying "I don't know". How to reliably detect and enforce these limits probably requires a completely different approach to AI.

I think ultimately any truly intelligent entity needs to have agency - to be able to interact with the world and test truth (prediction correctness) for itself. Without this you're really just an expert system - a bunch of canned data/rules.

The strength of ChatGPT over a GOFAI expert system like Cyc is how broad it's knowledge base is, but the weaknesses are that is doesn't know whether this "knowledge" (being self-deduced from the training set) is true or not, nor does it have any reliable way to combine facts/sources, since those rules are also self-derived and unverified. I'm sure some of this could be mitigated by more human curation (which seems to be the path OpenAI is going down), but then it seems it is just an expert system (maybe this is all people are looking for anyway).

> If you happened to get information from it that turned out to be ok, you were lucky. Don't expect your luck to last.

For creating bash scripts and english to code it remains remarkably consistent. No need to be a master of multiple languages syntaxes now.

Of all the languages one could ask ChatGPT to generate, bash would be nearly the bottom of the list. The number of subtle footguns in bash are through the roof.

Yeah, ChatGPT is exploiting how bad humans (or at least Americans) are at detecting sociopathic liars and bullshit artists.

It is consistent with how awful our business management class is along with our politics and relationships.

So frustrating to see people complain that it provides wrong information. With this logic it can never know anything. It is not made for querying. It's made to complete the text you provide it, with some minor modifications to help it answer questions. It does know the most common patterns of characters in the internet which implicitly contain knowledge.

> With this logic it can never know anything.

That's correct: ChatGPT does not know anything. That's not what it's built to do.

> It does know the most common patterns of characters in the internet which implicitly contain knowledge.

No, it doesn't. It knows patterns of text in its training data, but it knows nothing about the semantics of the text--its relationship to other things in the world, which is what is involved with any text that contains knowledge. That is why ChatGPT is not reliable as a source of information.

I share your skepticism of LLM’s output but I don’t think it’s fair to say they know nothing about semantics. I think it’s still an open question as to what degree LLM’s encode a coherent world model. Also you can just ask chatgpt about objects and their relationships and it gets the answer right way more often than you’d expect by chance, so it has some understanding of the world. Not good enough for me to trust it though

The answer is clearly none. It literally knows nothing about the world.

Not that I see much evidence of what I'm about to assert in ourselves, but you should be able to correct ChatGPT's knowledge if it knows things and isn't just a fancy parrot.

I use ChatGPT not because it is so great at solving tricky technical question, but because Google has become so incredibly shitty at it. Google's response these days is ten YouTube videos and a random mix of web posts dating between 2005 and 2023 for a piece of software that I'm using the latest version of. I honestly don't even bother trying anymore.

ChatGPT is a mixed bag. Sometimes it nails it, but I've also seen it hallucinate complete nonsense solutions, for instance in Davinci Fusion using a node type that doesn't even exist in the program.

As for "incredibly intelligent philosopher", I don't even find it an interesting "person" to talk to. Its responses are pedestrian, pedantic and boring. A question like "if God is good then why evil in the world?" is met with a summary of the wikipedia article on the problem of evil. I can read a Wikipedia article, thank you.

All of this seems eerily similar to Wittgenstein's use theory of meaning:

https://en.wikipedia.org/wiki/Use_theory_of_meaning

From wikipedia: The Investigations deals largely with the difficulties of language and meaning. Wittgenstein viewed the tools of language as being fundamentally simple and he believed that philosophers had obscured this simplicity by misusing language and by asking meaningless questions. He attempted in the Investigations to make things clear: "Der Fliege den Ausweg aus dem Fliegenglas zeigen"—to show the fly the way out of the fly bottle.

"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger W. Dijkstra

> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well

This is the issue. It's more like the dumbest guy in the room confidently convincing the everyone else that they know what they're talking about.

Yup.

As a pattern matching engine it's extraordinarily good. Ask it about the implications of some obscure theory and it will usually correctly interpret your question and return some sentences summarising the current state of knowledge on that theory. Ask it to explain it in simpler terms and it will simplify the sentence structure, swap some words and usually retain meaning.

But then... ask it for some reading material on the same obscure theory and it'll use its knowledge of scientific papers on those theories and academic citation to return a well formatted list of papers with plausible titles and authors with the significant flaw that none of these papers exist. The dumbest guy in the room would struggle to make up plausible sounding scientific papers even if you gave them the reading list, but then the dumbest person in the room wouldn't be dumb enough to use the reading list to make up some nonexistent papers in the first place! (and ironically nor would the dumbest information retrieval algorithm in the room...)

I think the bigger point is that it’s like a guy in the room. ChatGPT shows that the industry is graduating from Turing test territory to IQ test territory, and a lot of people are really hung up on the fact it’s failing the latter.

We've entered the Dunning–Kruger effect phase of the AI revolution.

There's a significant epistemological problem: how do you know that what it's telling you is true?

For things that you yourself can verify easily, this isn't too much of a problem. It gives you some boost tips, you see whether the program works or not. But as you come to rely on it, and it matters whether the answers are correct in important ways, what validation/verification/QA steps can you apply to ensure that you're getting the correct answer?

I'm reminded of the FDIV bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug , where people found that their CPU gave mathematically incorrect answers for certain inputs.

I'm also thinking of a concept of "responsibility". If you ask a human for tips on, say, building a high power laser, or making explosives, they might decide whether they want to give you that information, and they might decide to give you some other information along with it ("wear goggles") for your own safety.

Who is liable if an AI gives someone advice, they act on that advice, and serious injury or property damage results? If the AI (or its operator) has no responsibility at all, is that disclaimer adequately communicated to users?

I use it like Wikipedia: a good jumping off point. Often times I will forget an exact thing I wish to find online so I will ask ChatGPT to list multiple things that could be what I just described. I then take this list to my search engine du jour.

Good point about Wikipedia. In just a few years the image people had of Wikipedia changed from an untrustworthy joke (The Office had a joke about Michael Scott trusting Wikipedia) to one of the first places people went to get information.

Before that, there was a lot of concern about blogs when they were the hot new thing. Here’s a New York Times article[1] from that period discussing the blogging controversy:

> There are two obvious differences between bloggers and the traditional press: unlike bloggers, professional journalists have a) editors and b) the need to maintain a professional reputation so that sources will continue to talk to them. I've been a journalist for more than a decade, and on two occasions I asked acquaintances whether I could print information that they had told me in social situations. Both times, they made clear that if I published they would never speak to me again. Without a reputation for trustworthiness, neither friendship nor journalism can be sustained over time.

The concerns weren’t incorrect - Wikipedia or Blogs are often unreliable. But the concerns overlooked how useful these things could be despite their problems.

As an aside, it's interesting to see comments here bring up the potential unreliability of ChatGPT - and then say that people should go read Wikipedia instead.

[1] https://www.nytimes.com/2004/12/19/magazine/your-blog-or-min...

> Good point about Wikipedia. In just a few years the image people had of Wikipedia changed from an untrustworthy joke (The Office had a joke about Michael Scott trusting Wikipedia) to one of the first places people went to get information.

> As an aside, it's interesting to see comments here bring up the potential unreliability of ChatGPT - and then say that people should go read Wikipedia instead.

On the other hand, Wikipedia's image changed largely due to the transparency behind its editorial processes- both the edit history and the discussions regarding editorial decisions are available in public. Are there pages where it's important to take the information presented with a grain of salt? Sure, but I have FAR more metadata available to help me discern how reliable the article may be.

And that's kinda the difference- having a way to figure out the source of the information. I was in high school when Wikipedia really took off, and although there were more than a few kids who got caught copying their essays from Wikipedia (and a few more who got chastised for the lesser sin of trying to cite it as a source), the vast majority of folks were perfectly capable of realizing that all you needed to do was scroll down to the list of citations and work from there.

With all that in mind, the contrast with ChatGPT should be pretty clear. You can ask it to explain a topic, and it'll probably do a decent job (or appear to do so) but the process by which it produced that output is extremely opaque. And (as has been noted in other threads) OpenAI hasn't been particularly forthcoming about how they influence its responses, which is pretty ironic considering their name.

> On the other hand, Wikipedia's image changed largely due to the transparency behind its editorial processes- both the edit history and the discussions regarding editorial decisions are available in public.

That was still true early on when people were making fun of the idea. Like with many things, the public perception changed when it got popular and the people who had been laughing at it realized they also had a use for it.

I also can't agree that the production of Wikipedia articles isn't opaque. In fact, I would say that the perception of reliability of Wikipedia (as noted in many of these comments) makes it more likely to lead people astray .

Do Wikipedia articles often list sources? Sure. Do most people not bother checking the sources, and simply rely on a third-party to accurately summarize them? Also true. Do the few who actually check the sources often find it impossible, because the source is simply an academic book that most people don't have available? True as well. Even if it was available to them, the source is often (though not always) give as the entire book, so tracking down the source for a particular claim means you're searching for a needle in a haystack (and that's for just one claim).

I say this as someone who's actually interested in the source for the claims in Wikipedia articles, and have spent a lot of time trying to track them down. Often to no avail (and when I do manage to track them down, it's not uncommon to find the summary on Wikipedia misleading).

For instance, it takes me all of 30 seconds to write:

> Lincoln was actually in favor of subdividing Texas into four different states, and had the support of Congress to do so. It probably would have happened if he had lived. Source: Harris, William C. (2011). Lincoln and the Border States: Preserving the Union. Lawrence, Kansas: University Press of Kansas.

The source exists, but I have never read the book. Almost no one that reads this comment has easy access to the book, and even if they did I doubt anyone would go through it to fact check me (and by the time anyone did, this conversation would have been long over).

Does ChatGPT sometimes give inaccurate answers? Certainly. But at the moment more of its users seem to be more aware of the potential issues with the platform than the users of Wikipedia, AskHistorians, The New York Times, etc. are.

For nearly as long as Wikipedia has existed, there have been many people who trust it and about as many people who deride those who trust it.

It is very useful but there's definitely a constant issue with quality-of-results, but I've found a pretty good workflow, involving jumping back and forth between search engine results and ChatGPT results.

Fundamentally, it's:

1) Ask ChatGPT a series of questions, going from the general down to the more specific. For example, 'Give me a list of options for setting up a headless Raspberry Pi as a Nginx server from scratch using only command line tools'. ChatGPT will produce a decent list of options.

2) Run that list of options in truncated form, just the keywords, through Google or other search engine that allows for a 'verbatim' search. Set to 50 or more results per page to get a decent spread of results at a glance. Get the more specific information for various options.

3) Run that more narrowed information back through ChatGPT to see what it produces. At this point you might be pretty close to the correct specific information you need, but generally I run it through search again just to be sure.

For example, this way I was able to quickly discover my headless RPi setup with the latest Raspberry Pi OS wasn't allowing me to ssh into it over wireless because the RPi engineers did away with default user/password settings for security reasons, so I had to manually include the 'userconf.txt' file in the boot directory on the SD card. Also it was very helpful with what needed to go into the wpa_supplicant.conf file.

Trying to find stuff like that out by search alone, in the past, would have probably taken at least 10X as long.

I wouldn't ever blindly trust what it suggests, however.

> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field. Does this "person" make mistakes? Can't cite their sources? Yeah this definitely happens (especially the sources thing), but when you're trying to understand something new and complex and you can't get the "gist" of it, ChatGPT does surprisingly well.

if you don't "understand something new", how would you know to trust what it says or not? the usefulness or not has to be based on something, not feelings.

That goes for pretty much any resource; there's a lot of botched or sub-optimal explanations of complex things out on the internet. But being able to have a conversation with someone who has a decent conceptual grasp on something is better than a perfect encyclopedic article alone, and ChatGPT is able to roughly do that for a lot of subjects.

As for how you know it's accurate, you don't really have to know it's accurate most of the time, because if you're using it to try to learn something new it's probably going to become apparent quickly enough. Nobody is reading something and then going "Welp, now I'm done learning."

As a software example, you can ask it about implementing the Fourier transform. If it says something wrong, you'll find out.

The question then becomes how often it is inaccurate and how badly. But in that regard, as long as you don't treat it as an all-knowing oracle, it's at least as useful as having a friend that knows a lot of stuff you want to learn.

> That goes for pretty much any resource; there's a lot of botched or sub-optimal explanations of complex things out on the internet. But being able to have a conversation with someone who has a decent conceptual grasp on something is better than a perfect encyclopedic article alone, and ChatGPT is able to roughly do that for a lot of subjects.

people keep saying this, but it isn't true. if you ignore sites like this one and reddit there are plenty of authoritative articles and explanations about things with provenance. and if they are wrong they'll update their information.

Clearly you are of the opinion that ChatGPT is useless, there are superior resources already available for any topic, and it's all just hype.

Well, then we don't need to argue this since the problem will elegantly solve itself if that's true.

I disagree though. For a lot of things it feels like I can get much better answers than Google, especially when it comes to somewhat conceptual questions.

(Also, I don't use news aggregator comments to learn things unless it's the only possible source. But if you think every blog post or YouTube video that got things wrong has a detailed errata, you'd be sorely wrong. It's so uncommon on YouTube that the practice is usually commended when noticed.)

> Clearly you are of the opinion that ChatGPT is useless, there are superior resources already available for any topic, and it's all just hype. Well, then we don't need to argue this since the problem will elegantly solve itself if that's true.

I disagree though. For a lot of things it feels like I can get much better answers than Google, especially when it comes to somewhat conceptual questions.

---

I never said ChatGPT is useless, lol. it's truly amazing how people can be so bad at reading comprehension yet praise AI bots in the same post.

Ironically, the direct quote you just copied and pasted actually shows that I never claimed you explicitly said that at all. So much for reading comprehension :P

But in all seriousness, I just simply didn't have much to go off of, and extrapolated based on context in the thread. You could always elaborate on what your opinions are.

By having a conversation with it. Ask probing questions. Verify later with wikipedia.

For instance I asked it about space time metrics, and I learned about the varying types and in what circumstances each are used. Ask it about their mathematical properties and it gives a succinct explanation contrasting the different approaches. Certainly you can see the value now of conversational style pedagogy.

Think of it like learning about things from mates at the bar. In the thick of it there may be bits of inaccuracy, but you'll certainly know more about a topic than before. Then you can verify bits of things later in papers or books, etc.

seems like a waste of time if you're going to have to verify with Wikipedia anyway. personally I'd prefer to read wikipedia first.

Yeah, it's fascinating to watch, right?

As others have pointed out — what did we expect of 'AI' created in our image? Aren't we all just bullshit artists of varying degree?

HN is the perfect place to learn how much BS you are - I DARE any subject matter expdrts on here to post on any topic and I guarantee another expert will step in and tell you why you're wrong, or at best, nitpick you apart.

Thank you. Tons of noise because people are mistaking a statistical tool for a sentient consciousness.

ChatGPT has no more interest in running a confidence game than it does in helping children learn critical reasoning. Which is to say none, zero, zilch in both cases. It has as much volition and desire as a calculator. It’s just a super impressive and useful calculator.

>"It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field."

LOL where do you get this notion? I haven't seen one chatGPT bot useful for law. They all have no idea what they are doing and are basically dangerous if you don't know better than the LLM. There is literally nothing chatGPT can do for me as an attorney that isn't already solved by using a template.

> Can't cite their sources?

Has there been research on neural nets who could cite their sources? In common cases, thousands or millions of items in the training set have contributed to the weights in the pathways that activate for a certain input. But maybe for some inputs, only a small number of items in the training set contributed strongly to the weights on the pathways the are decisive to the output? Could you design a data structure that would keep this information in the neural net?

I’d have thought the short lived Galatica from Meta would have demonstrated more capability on this need given it was explicitly positioned as being to help Researchers.

I wonder if this is a special case for Explainable Models [1] — once you select an answer, another model helps describe and source the output.

[1] https://en.m.wikipedia.org/wiki/Explainable_artificial_intel...

I would imagine that citing sources would look more like finding sources that have a minimum distance to the output in a latent space representation? I don’t think it’ll make sense to track how each connection weight changed over time due to various inputs.

perplexity.ai/ specifically uses web sources to construct natural language.

I believe it's going to be in Bing soon?

> perplexity.ai/ specifically uses web sources to construct natural language.

I am going to guess that even though it uses the GPT language model, it works fundamentally differently than ChatGPT.

My guess is, Perplexity AI is a traditional search engine, that first finds a list of links that match your questions. And then it feeds text from those links to the GPT language model to write a summary of the results.

Whereas ChatGPT produces text based only on the trained deep neural net.

Maybe I am guessing wrong how Perplexity AI works. Maybe it works in a different manner.

It still has the exact same issues. I asked it to tell me about “Microsoft layoffs before 2022” and it confidently stated that Microsoft had no layoffs before 2022, with one of its sources even talking about the previous layoffs in 2009.

> It is basically like having an incredibly smart engineer/scientists/philosopher/etc

It is a copy/paste machine that strings together words to make sentences that look like sentences it’s read.

So it’s like an incredibly smart engineer/scientist/philosopher that does not and fundamentally cannot understand any of those concepts (or any concepts at all.)

A parrot that can recite Macbeth’s monologue is not a s Shakespeare scholar.

I asked it to explain some very technical medical testing procedures to me "like I was five years old". It completely nailed it by truly reducing it to its essence and using appropriate metaphors that a child would understand (coloring books). Because of how niche the topic is, I'm fairly sure no such text exists in the training corpus.

I'm not sure whether this constitutes "understanding", since that term usually applies to people's cognitive states; however, if a student provided this answer on an exam, they would certainly convince the examiner and pass the test!

A well-adjusted person should be able to signal some degree of confidence about what they're saying ("I know that…", "I think that…", "Maaaaaaybe…", etc). That's an anspect that ChatGPT doesn't have.

For the api appending such language is a waste of tokens, therefore a waste of money. People should use it while knowing it hallucinates all the time.

In other words, everyone should assume that it's lying at all times. Nobody should be using a product that both lies to you and then lies to you about the fact that it's lying to you.

Stating a falsehood without intent to deceive is just incorrectness, not a "lie". Let's dial the hyperbole back a few notches.

If the people running the platform are advertising a product that produces truths while delivering falsehoods and pathetically refusing to label their falsehoods as falsehoods, that's a lie.

There is a bit right on the chat box above every conversation that says, verbatim, "may occasionally generate incorrect information".

Determining truth is not in the wheelhouse of a language learning model and it is not defective for not doing so in the same way a tablesaw is not defective for not driving nails. Adjust your expectations accordingly.

> "may occasionally generate incorrect information"

The points of this thread are 1) this is a dramatic understatement; ChatGPT's output is not just occasionally incorrect, but usually incorrect in some dimension, and 2) in the absence of any fine-grained confidence value for individual statements, you must pessimistically assume that all statements are incorrect until proven otherwise, which dramatically reduces the utility of the tool compared to how its fanboys portray it as the second coming of Christ.

The first point is a bald-faced assertion with only anecdotal evidence, the second is a reduction to the absurd. It is absurd because if you uncritically accept the words of anything you read online, without, say, validating them with common sense, your own experience and knowledge, and so forth, the problem is with the reader, not the words.

You and the author of this article are making this false dichotomy where there is no middle ground between "usually incorrect" (which is hyperbolic nonsense and trivially falsified by five minutes of using it), and "always correct" (which even your straw "fanboys" have not done to my knowledge), and then using this dichotomy to set up another one to pretend that the only way to act on information read from a computer screen is either uncritical acceptance or presume that it's bullshit.

Neither of these models are accurate and neither of them have any relation to how people in the real world generally interact with information online or with ChatGPT.

Furthermore, your insistence on "labeling falsehoods" is not something we can do accurately anyways, let alone in the context of a language model which has no concept of truth or falsehood in the first place! You are asking for something completely unreasonable and I can't tell if you're doing it out of ignorance or bad faith.

That would depend on a) how OpenAI are marketing it, and b) what disclaimers/etc. they put in the UI.

It's not lying about the fact that it lies, they have a disclaimer.

It's the user's responsibility to verify truth. This was trained on the internet and everyone knows not to believe everything you see on the internet. This isn't any different.

Was looking for the disclaimer comment, they do genuinely mention on their page it's not fit for consumption, although so do many services we often use.

And you don't see how this makes it useless as a paid product? Truth is the product. An AI chatbot that makes no pretensions to the truth might as well be a Markov chain for all the good it does me. Are people really so blinded by the song and dance of this mechanical monkey that they can't see through the ruse?

For many of us truth is not the product.

For a lot of people it's a faster way to google something you're familiar with, a more convenient stackoverflow.

When used right its hallucinations don't matter, and the user takes advantage of producing things quickly that would have been tedious to write manually. Like one-off shell scripts.

If your use case necessitates truth then I see why you don't think it's worth paying for.

However this is the only digital product I'm paying a subscription for. It's faster than googling and trudging between useless results, ads, and useless blog content, so it's worth every cent.

Well said. Exactly. So much hyperbole! So much click bait. It makes me dizzy.

I'm a front-end dev. I use it every day. I have CoPilot too. These things help me do my job. No question. They also help me learn better than just about any other tool.

The big one for me is that they have made me more confident taking on bigger tasks. The impact is not trivial at all.

What people also seem to forget (if they knew in the first place) is how bad GPT3 was compared to GPT4/ChatGPT. I subscribe to SubSimulatorGPT3 on Reddit and it generates tons of text that looks and feels like English, for example, but is devoid of any real meaning. Every sentence, every paragraph I read from GPT3 has me questioning whether or not I’ve had a stroke.

Now, compare that to your comment, and it’s INSANE how much the capability (and the conversation) has shifted.

GPT3 was what, three or four years ago? What will we be debating, what will the capabilities be, in another three or four years?

GPT4 hasn't been released yet. I think you're probably thinking of GPT3.5.

> help me practice a foreign language I've slowly been forgetting

So with the software stuff you can pretty easily verify the output: either the suggestion fixes the problem or it doesn't. But how can you trust it for something like this where you can't distinguish what's good and bad output? It could be leading you down the garden path with this language exercise. And it's not in the helpful thinking tool/bouncing ideas around category, either: there are rules of grammar that you don't know, and you don't know whether ChatGPT knows them either.

Well an AI that can't explain itself transparently makes it untrustworthy for more serious use and applications since ChatGPT it is being celebrated as a great sophist. There is nothing new in this, other than 'train more data on the internet'.

The hype in ChatGPT reminds me of the short-lived hype of Clubhouse; the end result is the same: The bigger players will come in to surpass it, just like they did with GPT-3.

I wonder. If the bigger players can make an AI good enough to be indistinguishable from human (but not quite), then wouldn't the AI slowly and incestuously train itself to become dumber, with nothing but AI-generated content left to "feed it?"

I can see tons of jobs disappearing even with current level. Web these days is filled with mediocre articles, and quite a few bureaucratic jobs can and will be replaced by some next iteration.

Instead of army of mediocre journalists filling web portals, you just need 1 person to proof read it per business, maybe not even that. I can see the same for software dev in less than decade with current pace, 1 senior going over generated code and doing minor tweaks. Which is probably road to hell long term since how will juniors become seniors, but cost savings can be real and massive, drop in quality small if at all and it may be good enough for many.

It has very good language skills, in my obscure and rather difficult native language which only cca 6 million people use it formulated perfect sentences consistently. I can easily imagine faking social, even love chats easily. This can help lonely or elderly people (with good speech recognition/generation, since language skill itself is there already). With image/video generation porn industry will be steamrolled, 24/7 streams of anything you can imagine, looking realistic enough, tailored just to you.

Or just help with learning foreign languages, heck few months ago a guy was posting here on HN just a smart chat that could be used for training and got standing ovation also from me for that, this has it and everything else.

I understand google's reaction, instead of browsing through paid entries it gives me answer directly, no need to click through search and then wikipedia and then parsing texts. At least in some cases. This could help google massively if they could somehow do it on their scale.

People trying to find errors in it are missing the point now - its not about academic perfection, but often good-enough almost for free. Also, this is not final product, only 4th alpha/beta/whatever, previous ones were much worse. Imagine 10 or 20 years down the road...

I agree. I regularly use ChatGPT to help reword paragraphs to be more succinct. I rarely ask it for actual information that it has to look up. At its core, it's an LLM which models how language is structured, and that's what it's good at.

It has an impressive amount of understanding about language. There was a meme a bit ago about "Everyone on my floor is coding" and the different reaction of a software developer and a doctor. So I asked for jokes about a doctor and a software developer about the different meanings for the words they used.

    Write five jokes about a doctor and a programmer about the different meanings of words they use.

To which GPT responded:

    1. Doctor: I'm sending you a prescription.
    Programmer: Great! I'll take a look at it and see if I can debug it.

    2. Doctor: I need you to take two aspirin.
    Programmer: Is that enough to solve the problem?

    3. Doctor: I'm going to give you some instructions.
    Programmer: Do I need to compile them or can I just run them?

    4. Doctor: I need you to take a break from work.
    Programmer: Does that mean I should pause the program or stop it completely?

    5. Doctor: I need you to stay hydrated.
    Programmer: Does that mean I should store more data or drink more water?

I will certainly admit that those jokes are awful, but the interesting part of it is that... it did it. I think that 3, 4 and 5 are the "best".

I don't think it understands those things with the ability to reason about them, but that demonstrates more than I would expect from a Chinese room.

    Explain the joke "Doctor: I need you to stay hydrated.  Programmer: Does that mean I should store more data or drink more water?"

returns back:

    The joke is a play on the terms "hydrate" and "store data". In programming, "storing data" is the process of saving information to a computer's memory, while "hydrating" means drinking water to stay hydrated. The joke is implying that the programmer is more comfortable with programming than with taking care of their own health, so they interpret the doctor's advice to stay hydrated as a suggestion to store more data.

What’s worse, too much hype, or the non-believers?

Well the OpenAI hype squad is racking up personal fortunes from their work in a finding deal that is pretty clearly over-valued. Especially versus what Deepmind sold for [1].

While perhaps both sides are not being honest, YC’s own sama is clearly profiting excessively over the hubub. He’s clearly proving to be a symbol of the faction of YC that rakes in money through arbitrage not prolonged value.

[1] https://techcrunch.com/2014/01/26/google-deepmind/

> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field.

I have yet to see a single instance of this with any query that requires even a modicum of expertise. I have asked it a lot of questions about my field. I have asked it to summarize and paraphrase and ELI5 all kinds of shit. Everything it produces is either inane or a regurgitation of the input just with verbs and prepositions changed.

I would like to see a nice example, input and output, of this. I really would.

First thing that came to mind:

> Explain how git works in very simple english

Git is a version control system for software development that helps track changes made to the source code over time. It allows multiple developers to collaborate and make changes to the same codebase without interfering with each other.

When you start a project in Git, you create a "repository" that stores all the files and their changes. You can then "commit" changes to the repository, which is like saving a snapshot of the code at a certain point in time. You can also "push" the changes to a remote repository on a server, so that other people can access and work on the code.

If someone else makes changes to the code, you can "pull" the updated version of the repository to your local machine and "merge" the changes with your own code. Git also allows you to create "branches" of the code, so that you can work on different features or fixes without affecting the main code.

By using Git, you can easily revert back to previous versions of the code, compare changes between different versions, and track who made which changes

"It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field."

I don't know how to tell you this without being a little rude, but if you think ChatGPT is incredibly smart or that it can tell you meaningful things about "any field," well, there is a difference in intelligence somewhere in the equation between you and ChatGPT, but its not the one you think.

ChatGPT/GPT fails a lot when you probe its knowledge, but I find it works really well when you give it an explanation of the whole domain of the problem, then something to do with it.

For example can give examples of some programming language, or an API, and natural language explaining what you want to do with it (or code in another language), and it will translate this with remarkable accuracy, 0 shot. We've never had anything that's even close to this capability.

Its honestly astounding how coherent and dare, I say, eloquent it can be. From factual "Google"-type inquiries to jokes to whatever. I particularly enjoy the poetry it generates in defined formats like limericks, sonnets, etc. Yet to encounter any actual "bullshit" that it is frequently accused of (although I tend to ask closed-type questions or provide very specific parameters with which to constrain its output)

> "It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field. Does this "person" make mistakes?"

Does this person stubbornly insist on being right, even if they are wrong?

Likely, that's just how humans usually behave - and ChatGPT does the same.

At least that was my thinking when people complain about ChatGPT being a know-it-all bluffer.

> Does this person stubbornly insist on being right, even if they are wrong?

Actually ChatGPT will almost always change tune if you tell it that it is wrong. That's because it is a stochastic parrot and that's an "unlikely" event if you actually got the answer right. But we'll see if this gets updated/broken too.

I've seen ChatGPT change tune but I've also seen it squirm plenty. Overall I think it behaves uncannily human in this regard.

ChatGPT is pretty amenable to being corrected if it says something you know is crap. And then, in my experience, you usually get better results.

That is not something I can say for most humans.

Only if you correct a wrong answer. If you "correct" a right answer ("I don't think that's right") then it gives an even worse answer. So be careful when correcting.

Agree. ChatGPT provides great results while doing quite a bit of dreaming at the same time. The role of the human is to separate dreams from reality.

>but it is still ludicrously useful.

Wrote me a script to quickly convert SRT to VTT files as I edit and prep for uploading online. I haven't done anything akin to "coding" since I took a very basic class in college and messed around with HTML/CSS. Shaves several steps off of every video I do (I'm in a high volume world here).

> It is basically like having an incredibly smart engineer/scientists/philosopher/etc that can explain things quite well, but for pretty much every field.

No it can not explain anything. It's a (stochastic) parrot, nothing more.

> Yep, but it is such an incredibly useful tool

It is not. You are fooling yourself. You are building a castle on a swamp.

ChatGPT tells me that Disney World's Magic Kingdom was built on a swamp.

It's just a tool. You don't use a shovel to rake leaves, but that doesn't make a shovel any less useful.

My problem is the type of person that proudly proclaims that all jobs can, should, and are going to be outdated by the use of AI. I have talked to people that believe that, and think that AI need to replace humanity for some reason. I don't understand them.

AI is exciting, and it can be an useful tool. There are a lot of use cases for it, and in the end I think it will be a good thing. However, what's worse than the detractors are the people blindly hoping that AI will automate every single thing a human touches. It shouldn't be used in that way. It has to be a tool to assist humans, not take away our professions and reasons to get out of bed in the morning.

FWIW I see it the same and have had the same experience.

The tech is there now to be utilized as an assistant. However as the end user you better have enough knowledge to understand the solution or answer it spits at you to fact check it.

The biggest danger I see is utilizing the tool for solutions or answers by novices that don't understand enough to grasp the solution and confirm if in fact the answer it gives is correct or appropriate.

Simplistic example: A non techie manager trying to have it create sql queries to get the information they are asking for , is asking for trouble. However in the hands of a sql developer it could provide enough of the query build out to just require minimal tweaks.

> Does this "person" make mistakes? Can't cite their sources?

Making citations up is not a "mistake", it's a lie. I have absolutely no use for a tool which lies to me, especially not when troubleshooting computer systems.

I agree, its incredibly useful if you know how to use it. I used it to make sentences more succinct for emails to managers (no sensitive data of course), I used it to turn my dreams into short stories, gather ideas for research, etc.

People bashing on chatgpt just dont know how to use it. is there going to be an example in the world where it just plagiarizes bob dillan? sure ... so what? is there going to be instances where its not useful? duh.

but if you dont like it, that's fine. not every tool is for everyone. I find it very useful for certain things, and I imagine its only going to improve from here.

I had problem with the citing as well (have to ask separate questions, then google it).

It would be a huge improvement if it would be trained to provide references by default.

None of the coding questions it answers is closed to usable, and i need to spend more time correcting it than just write it on my own. So it is just not “incredibly smart”.

The latter group exists to counter the former. If folks weren’t making some of these over-the-top claims, then there wouldn’t be the claims to debunk.

In all the fields I've tried (software/music/epidemiology/geosciences/mathematics) ChatGPT can mostly explain things at the lay level, but even there often gets things wrong that without more expert guidance, you will not notice.

Experts in each of these areas were consistently better at giving a correct explanation.

ChatGPT is great when occasionally being wrong doesn't matter. It's still pretty good when you can't be wrong, but you're an expert. But in any "no mistakes, I need to talk to an expert" situation, ChatGPT is setting you up for failure.

That's the key part - the information is unreliable/untrustworthy. As long as that is not a problem, it's awesome.

The problem is that it presents wrong data with absolute confidence in a very convincing voice. (Hence people calling it a con artist). It'll become a huge problem as it makes inroads in mis/disinfo campaigns.

(The usual tech problem. Don't let large scale interests touch it, and you'll be OK)

“Sycophantic” to whom?

Sometimes I think ChatGPT was just some twisted, treacherous, indirect plot, to find a way to finally discredit Jordan Peters.

Reply on news.ycombinator.com