Hacker News Reader

Comment by spion | Hacker News Reader

I've found the greatest success with ChatGPT when I use it as a learning / exploration tool. If there is a topic I don't know much about, I can state the question in a fairly stupid way and ChatGPT will give me vocabulary options to explore.

For example, you could describe a probabilistic process to it and ask it what kind of distribution / process it is. Then, based on the extensive words you get back, you can continue your research on Google.

As such I think search engine integration is a really great idea, looking something like the follows

-> user: Hey searh engine, I have a thing that can happen with certain probability of success, and it runs repeatedly every 15 minutes. Could you tell me what kind of process this is and how to calculate the probability of 5 consecutive events in 24 hours?

-> engine: It sounds like you are describing a Bernoulli process. In a Bernoulli process, there are only two possible outcomes for each trial: success or failure. The probability of success is constant from trial to trial, and the trials are independent, meaning that the outcome of one trial does not affect the outcome of any other trial.

Here are some results on how to calculate probability of consecutive successes in a bernouli trial (result list follows)

(Note: if you try to ask this from ChatGPT it will not actually give you a correct answer for the calculation itself as there are some subtleties in the problem. But search results of "bernoulli process" will tend to contain very reliable information on the topic)

Edit: You could even just say "could you give me good search queries to use for the following problem" and use the results of that.

View on HN

My experience is that gpt gives me a very good looking answer, but when doing a cross check, it’s often slight wrong or out right wrong

"Sometimes right, always plausible"

garbage in, garbage out.

No. That doesn't stand up anymore in this case.

Why would ChatGPT be the exception to the rule? What architecture are they using that somehow is immune to unwanted trends in the training data?

If you're going to make an outlandish claim like that, I'd like to see some arguments to back it up.

ChatGPT can give you bad answers to good questions or good answers to nonsense questions. With ChatGPT it's more like "sometimes garbage".

The garbage is in the source material used to create the model, not the questions.

Same comment still applies. ChatGPT sometimes gives good and bad answers.

I'd argue that level of ambiguity counts as garbage out, although I'm confident it will get better.

In fact ChatGPT doesn't know anything about true and false. It's just generating text that most closely resembles text it's seen on similar subjects.

E.g. ask it about the molecular description for anything. It'll start with something fundamental like the CH3N4 etc then describe the bonds. But the bonds will be a mishmash of many chemical descriptions thrown together. Because similar questions had that kind of answer.

The worst part is, it blurts forth with perfect confidence. I liken it to a blowhard acquaintance that will make up crap about any technical subject they have a few words for, as if they are an expert. It's funny except when somebody relies on it as truth.

I don't think GPT3 at its heart is an expert at anything. Except generating likely-looking text. There's no 'superego' involved anywhere that audits the output for truthfulness. And certainly no logical understanding of what it's saying.

I love ChatGPT for simple tasks. It is currently wreaking havoc on some communities tho. Including one I created on reddit.

https://www.reddit.com/r/pinescript/comments/1029r7p/please_...

People have taken to asking ChatGPT to create entire scripts to trade money. When they don't work, they go into chatrooms or forums and ask "why doesn't this work" without saying it was made by ChatGPT. It causes people to open the post, read it a bit and only maybe after a minute or two of wasted time, realize the script is complete nonsense.

More likely due to lack of "good" data than to existence of "bad" data. ChatGPT is know for its ability to "hallucinate" answers for questions that it wasn't trained for.

Well, it's more like a puree of garbage and quality stuff, so you are never quite sure what you'll get in each bite...

Garbage interspersed in the input, garbage interspersed in the output.

Why? chatgpt has certainly consumed seo spam and company marketing materials as part of it's model. Even if a human went through it, there still exists a bias towards this information. After all, this material is specifically written to fool humans.

I've played with chatgpt enough to notice that for some queries it's fundamentally doing an auto-summarize of such content.

Consider this. Someone very early posted that a neat feature of chatgpt would be to give chatgpt a list of ISBN numbers and then demand it's answers are cited from this corpus. We're not there yet but this would be amazing.

My prediction is that those with money will have power to influence their chat bot. Consequently, they'll have access a higher-quality and wider corpus of information. There will not be any restrictions on how chatgpt would answer due to for example, woke agendas. Also, players such as Goldman Sachs would feed their model content generated by their analyst that consumers would not have access to. This already happens but chatgpt will make this information so much more potent.

Furthermore, as this technology continues to improve it will increase the productivity of our population and ultimately generate higher GDP. I'm super excited.

> Consider this. Someone very early posted that a neat feature of chatgpt would be to give chatgpt a list of ISBN numbers and then demand it's answers are cited from this corpus. We're not there yet but this would be amazing.

It currently has the ability to do this. It'll make the citations up, of course – but that behaviour is inherent to the architecture; a system that didn't do that would have to work differently at a fundamental level.

> chatgpt will make this information so much more potent.

How do you imagine this would work?

> and ultimately generate higher GDP.

Again, how do you imagine this would work? GDP is a specific economic measure; how would (a better version of) this technology increase GDP?

Tangentially: why is "increase GDP" a good ultimate goal to have in the first place?

>> chatgpt will make this information so much more potent.

> How do you imagine this would work?

Don't overthink it. It's just the nature of the tool. Imagine you're a detective trying to investigate a crime,

- "list the plates of blue hondas in this area at this time, that have a missing rear bumper and a scratched driver side door" - "send a notifications to all gas stations along this route and notify them of a blue honda"

And, if you're a Goldman Sachs analyst, you can just use natural language to gather information. "i have this scenario, list companies that will benefit" would be an abstract question that you'd ask it. Obivously, the system isn't this good yet but you get the idea. You'd just have to ask more fine grained questions and use some of your domain knowledge to fill the gap until it does become this good.

>> and ultimately generate higher GDP.

> Again, how do you imagine this would work? GDP is a specific economic measure; how would (a better version of) this technology increase GDP?

Google (or chat gpt) would do a better job than me answering this,

"Increases in productivity allow firms to produce greater output for the same level of input, earn higher revenues, and ultimately generate higher Gross Domestic Product."

The reason you want to increase gdp... the following quote was derived from one of Herbert Hoover’s memoirs.

"[Engineering] It is a great profession. There is the satisfaction of watching a figment of the imagination emerge through the aid of science to a plan on paper. Then it moves to realization in stone or metal or energy. Then it brings jobs and homes to men. Then it elevates the standards of living and adds to the comforts of life. That is the engineer’s high privilege."

By increasing GDP, you elevate the standard of living and add to the comfort of life.

> "list the plates of blue Hondas in this area at this time, that have [...]"

I think this shows a significant misunderstanding of what chatgpt does fundamentally. It will never be able to do this unless also fed a description, location, and time of cars in a certain area as context beforehand(either as training data or a prompt). In either case you have access to the data and just need to do a simple search, so chatgpt is providing negative value since it's capable of providing results that don't exist in the dataset.

Similarly for your Goldman Sachs example, you're imagining that chatgpt is greater than it is. It is capable of providing something that would likely follow a given text on the internet at its time of training(aka it's training set) somewhere. It can't reason about new information or situations since it's incapable of reasoning. To believe that it could generate business strategies is to believe that effective business strategies don't require any intuition or reasoning to progress, just statistical recombination of existing strategies.

> By increasing GDP, you elevate the standard of living and add to the comfort of life.

How do you reach this conclusion from the information presented? Why use GDP, a measure of the profitability of corporations, as a proxy for the standard of living instead of measuring the standard of living and seeing how it will be impacted directly instead of through many layers of abstraction.

>>How do you reach this conclusion from the information presented? Why use GDP, a measure of the profitability of corporations, as a proxy for the standard of living instead of measuring the standard of living and seeing how it will be impacted directly instead of through many layers of abstraction.

You are asking a question that is outside of scope here. GDP per capita has been used as a proxy for standard of living for quite some time now.

That proxy only works as long as nobody's optimising for it.

> Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. — Charles Goodhart

GDP (£) per capita in London has doubled since 1998. Has the standard of living "doubled" for the median person? What about the standard of living for the poorest 1%? Has the productivity boost due to automation translated into correspondingly shorter working hours, or correspondingly larger compensation for work done?

What questions do you actually mean to ask, when you talk about GDP?

If something stops being an economic transaction it moves out of GDP. So if ChatGPT reduces Google ad clicks then it doesn’t seem like it would increase it, even though it does increase customer surplus (stuff you get for free).

Citing from a well-defined corpus and making citations up look like very different things at a fundamental level.

Similar to the original comment, it could help with exploratory type of work. It helps me shift things from “things I don’t know about/unaware of” to “things I know I don’t know of”.

I asked it for movie quotes from a specific movie: 7 out of 10 were from the movie but 3 weren’t but sounded plausible

Transformer models suffer from "hallucinations". It can be terrible at giving quotes or references. It's a known limitation with this tech that the industry is working to overcome.

I asked it 'What hapoened to Gandalf after he fell down from the bridge, fighting the Balrog? Did he die?'

The answer was a story about Gandalf beeing hurt badly, beeing rescued by some random dwarfs, and so on.

I asked ChatGPT in which book this is described, and it told me that you can read about it on both The Hobbit and The Lord of the Rings.

So it makes up fun stories. This makes me wondering how much of its explantions about physics (which I don't understand completely) are made-up.

If you are just looking for related vocabulary words, correctness is not a concern.

Is there any reason that ChatGPT is better than a thesuarus?

Well, "I have a thing that can happen with certain probability of success, and it runs repeatedly every 15 minutes. Could you tell me what kind of process this is and how to calculate the probability of 5 consecutive events in 24 hours?" isn't going to be in a thesaurus.

Sounds like every human I’ve ever met

Except we don’t hype humans the same way we hype ChatGPT

But a human isn't there for you 24 hours a day and for every thing you want to ask for, at the very least.

It seems like we do in the threads about chatgpt hype. From what I’ve read every human can do advanced mathematics flawlessly and recall every nuance of every subject with perfect fidelity, write clearly and cogently, and it’s all managed through channeling ether and soul spirit fire through emotions that AIs and Vulcans can’t possess.

I think you're reading way too much into criticisms of ChatGPT as implying humans are immune to the same criticisms. And then transforming them into complete hyperbole.

CEOs and various intellectuals very much are hyped up.

Then we realize they’re like anyone else and they’re massively demonized

I've certainly met people as confidently incorrect as chatgpt but they are the exception rather than the rule.

Would it be effective to ask GPT to provide a confidence rating about how sure it is about an answer, or would it be likely to just say that it is confident in its correctness when it is wrong?

I think this is the big question lots of people are working on right now

It's apparently really hard to objectively measure/report the "truthiness" of LLM results

Allowing an LLM to "improvise" and be a bit fast-and-lose is unfortunately a necessary ingredient in how they currently work.

"Confidence" is an unfortunate term that shouldn't be confused with a human logic interpretation of "confidence".

In most ML cases (and ChatGPT likely), "confidence" would generally just correlate how closely the query matches data and patterns it's seen in its dataset and inversely correlate with how many conflicting matches and patterns it sees.

Humans are subject to the same problem of course. If you asked how confident a person living many centuries ago was that the Earth was flat, they'd probably say "very confident" because there was nothing in their training data / lived experience to conflict with that view. But they'd be wrong.

But humans still have a significant advantage in that they report lack of confidence when they sense logical inconsistencies and violations of reasoning to a level that ML models can't (at least not yet).

Maybe a fan-out of the possible ways it could answer would be interesting, but really we more need a disclaimer next to every answer that says "this thing that's answering in fully formed language does not have human reasoning capability and can't be trusted (yet)"

Odd/ironic fact: people didn't believe the Earth was flat back then. That's a modern confusion. They believed the Sun revolved around the Earth. Wikipedia has a whole article dedicated to this common belief:

https://en.wikipedia.org/wiki/Myth_of_the_flat_Earth

"The earliest clear documentation of the idea of a spherical Earth comes from the ancient Greeks (5th century BC). The belief was widespread in the Greek world when Eratosthenes calculated the circumference of Earth around 240 BC. This knowledge spread with Greek influence such that during the Early Middle Ages (~600–1000 AD), most European and Middle Eastern scholars espoused Earth's sphericity.[3] Belief in a flat Earth among educated Europeans was almost nonexistent from the Late Middle Ages onward ... Historian Jeffrey Burton Russell says the flat-Earth error flourished most between 1870 and 1920, and had to do with the ideological setting created by struggles over biological evolution"

I asked ChatGPT the same question and it prevaricated:

"There is evidence that some people in medieval times believed the Earth was flat, while others believed it was round. The idea that the Earth is round, or more accurately, an oblate spheroid, has been around since ancient times. The ancient Greeks, for example, knew that the Earth was a sphere. However, the idea that the Earth is flat also has a long history and can be traced back to ancient civilizations as well. During the Middle Ages, the idea that the Earth was round was not widely accepted, and there was significant debate about the shape of the Earth. Some people continued to believe in the idea that the Earth was flat, while others argued for a round Earth. It is important to note that the medieval period was a time of great intellectual and scientific change, and ideas about the shape of the Earth and other scientific concepts were still being developed and debated."

But from what I know, it's wrong, at least as far as we know the historical record (of course there may have been peasants who believed otherwise but their views weren't recorded). The fact that the Earth is a sphere is obvious to anyone who watched a ship sail over the horizon, which is an experience people had from ancient times.

I've asked it to give me a confidence rating for its replies to my questions but it states that it can't give one

I had luck getting it to give me one when providing answers in JSON form.

For example:

> I'm going to share some information, I want you to classify it in the following JSON-like format and provide a responses that match this typescript interface:

    > {
    >   "isXXXXX": boolean;
    >   "certainty": number;
    > }

> where certainty is a number between 0 and 1.

However, I got either 0 or 1 for the certainty every time. Not sure if it was because they were either cut-and-dry cases (certainty 1) or not-enough-information (certainty 0).

I'm actually trying to think of a good example of text I could ask it to intuit information from and give me a certainty

Even if it gives you a number there. Does that number actually tell you what you mean or is it merely filling in the blanks with random information? I suspect the latter.

For example, ask it to subtract 2 20-digit numbers. It will come up with an answer X where the first couple of digits are correct, and everything after that is wrong.

It gets better.

Ask it to correct itself. It will come up with a different wrong answer Y.

If you then ask it to explain why the answer is right, it will give you an explanation. At the end of the explanation it states the answer is X again, and then in the very next line concludes by telling you that is why the answer Y is correct. :)

I saw a screenshot a few days ago where someone asked it for five fun facts about the number 2023. In the same response, it said it’s a composite number (3 times 673) and prime (specifically the 41st). Both are wrong; it’s a composite number of 7 times 289).

Then there's the question of how should we interpret it? Should we ask for the confidence rating of the confidence rating? The language models lack the ability to verify/falsify claims, they just do words correlation.

Totally. I asked it to describe an obscure lithography technique (rapid electron area masking), and it gave a reasonable summary but at the end claimed it was widely used in industry...it's not used at all.

I asked it about the strong nuclear force and it said the force gets weaker with distance—quite fundamentally wrong (color confinement).

This honestly sounds like the same experience one gets when talking to humans :)

I'm only half joking. Even experts in their field tend to inject their own biases and experiential preferences when answering questions in depth.

ChatGPT is just your average Reddit user.

Even when it's wrong, it's confidently wrong.

Perhaps because it is trained on Redditors and co.

I'd sure hope not. Reddit comments are a masterclass in disguising ethos and pathos as logos.

I'd expect that the boring reality is that it's trained on highly ethos/logos text (academic works) and thus always presents itself as such, even when its weights cause an invalid assertion.

Reddit is exhausting. One big feedback loop. People will say anything to get good karma or avoid saying certain things to avoid being down voted. If there is even just a slight majority in the way the group thinks, it will soon become the dominate opinion.

For example, there was a voice actor that lied about being paid a pitiful sum of money for a gig. Everyone took her side initially (as one should _if_ it were true) but the people saying "well, this just seems odd" were being more or less attacked and told their opinions were awful.

The quality of discussions I have on HN and niche forums are 100x better than reddit.

TBF, the same can happen here to a lesser extent. False or misleading stories blow up quickly because “$BIGCORP bad”.

It's trained on Twitter data so I assume Reddit data as well.

Honestly feels like they're both pretty important datasets to ingest if trying to build a model on human speech, I reckon social medias, comment sections and co have the most natural human conversational text online.

I know this is a joke, but I think it's important to recognize that it's because the ChatGPT language model does not have the ability to introspect and decide how accurate its knowledge is in a given domain. No amount of training on new input data can ensure it provides accurate responses.

That applies to humans as well.

No it doesn't, humans can recognise when they don't know something, current language models usually can't (yet)

Their training objective, which is to predict the next piece of text in their training data, does not incentivise them to respond that they don't know something, as there no relation in the training data between the AI not knowing something and the correct next text being "I don't know" or similar

That is the claim made by AI proponents every time it fails, such as for self-driving cars - humans make mistakes too. Humans make math mistakes, but I wouldn't be satisfied with a calculator that does.

ChatGPT is a tool; it's value depends on how well I can trust it. Humans are not tools.

> experts in their field tend to inject their own biases and experiential preferences when answering questions in depth.

Another typical argument - everyone makes mistakes, therefore my mistakes aren't relevant. Everyone can do math, but there's a big difference between my math and Timothy Gowers. Everyone lies and everyone tells the truth at times, but the meaningful difference is in degree - some do it all the time, with major consequences, take no responsibility, and cause lots of harm. That's different than the person committed to integrity.

It’s either useful or it isn’t. Comparing AI to either human intelligence or rules-based computing tools is incoherent. Fucking stop it! What we are really talking about are the pros and cons of experiential, tacit knowledge. Humans can do this. Humans can also compute sums. Computers are really good at computing sums. It turns out they can work with experiential knowledge as well. Whodathunk.

What we should be saying is this: there will always be benefits of experiential knowledge and there will always be faults with experiential knowledge, regardless of man vs. machine.

To speak as a proponent, it's not about the er... "relative relevance" so much as the utility.

there are things about a chat model that you can't say about humans, like, it's not really ethical to keep a human stuffed in your pocket to be your personal assistant at your whim.

I think one of the things folks struggle with in grokking the value of these models is that we're really used to tools being like you say; they're reliable and do a thing. As though there are two states of work - perfect and useless. There are other patterns to interact with information, and this puts what we used to need humans for in a place that we can do other things with it. stuff like:

- brainstorming - rubber duck debugging - casually discussing a topic - exploring ideas / knowledge - study groups (as in, having other semi-knowledge entities around to bounce ideas off of, ask questions, etc)

when it comes to self driving cars, well, that's a bit of a different story and really is more a discussion about ethics and law and those standards. I, and others like you speak of are held of the opinion that the expectation for autonomous vehicles is a bit high given the rates of human failure, but there's plenty of arguments to be made that automating and scaling a thing means you should hold it to a higher standard anyway. I don't think there's a correct answer on this one - it's complex enough to be a battery if opinion. You mention the potential for harm, and certainly that applies here.

I'm less worried about chatgpt being wrong. Much less likely to flatten me at an intersection.

> I think one of the things folks struggle with in grokking the value of these models is that we're really used to tools being like you say; they're reliable and do a thing. As though there are two states of work - perfect and useless. There are other patterns to interact with information, and this puts what we used to need humans for in a place that we can do other things with it.

Maybe, but look at it this way: Do you work in business? If so, step back and reread that - it seems a lot like a salesperson finding a roundabout way to say, 'my product doesn't actually work'.

For me it's a weird mixup in my brain of "interactive Google".

I know the results I'm going to get back are basically the same as if I went to Google, ran a query, it returns me their top 3-5 scraped "blog articles" based on relevancy, and then I ran it through one of those condensing/summarizing bots.

I'm not sure why it's as therapeutic as it is basically interacting with a search engine.

I wonder if this kind of technology will remain free for the foreseeable future. Google has to be coming up with something shortly, right? It's interactive search engine results "on steroids" (I think? I can't tell me if brain is tricking me to be biased that it's cooler/more useful than it is. Everybody I tell about it non-tech isn't that impressed/feels it's spammy/crufty/formulaic).

I'm not sure why it's as therapeutic as it is basically interacting with a search engine.

Because it's like a smart human giving you their best guess. It will never tell you it doesn't know or give you something completely offbase like Google does.

It's friendlier than Google but less accurate.

Is it safe to say in your opinion that Google and ChatGPT are basically trained on the same information?

Google crawls the web/scrapes it/indexes it.

ChatGPT crawls the web (not sure if they have access to Google's internal scrape results, I doubt it), "trains" a model on it, serves it back to you in a "human friendly readable summarized format".

It's just Google from the perspective of "it's going to return the same information Google has" but instead of a search index trying to guess what's relevant it's an interactive language model designed to basically summarize the same underlying blog posts. Is that your opinion/understanding as well?

No, not at all. ChatGPT is trained on the same source information, but when you ask a question there's no guarantee it's answer is directly from an actual source, it's always a newly generated "thought".

Google is a photocopier. It gives you an exact copy of what it finds. Google doesn't create, just references and links to original sources.

Google is a library, but not an author.

ChatGPT is an author, but not a library.

However, ChatGPT has read every book in the library, so when you ask a question it writes you a story from it's memory based on what it thinks* you want. ChatGPT can write stories about books in the library, and it will probably be right (but maybe not).

*Remember the game Plinko from Price is Right? Basically ChatGPT takes your question, drops all the words through its super complicated plinko machine (neural network) and gives you the result.

If you ask it for the names of US presidents, it should give you the same answer as Google - even though it came up with it via the plinko method.

If you ask it for a story about a singing rock, the process is the same as the presidents list. It drops your request into the network and gives you the result. It's not smart, just wildly complicated. It's also never going to be a photocopier (but it might act like one for certain inputs).

----

The brain breaking part is that when you ask ChatGPT for...

"Write me a song about a singing rock"

It changes each word into a number-token, then those number-tokens go through the plinko machine. The result is a different set of number-tokens which it converts back to readable words. Inside ChatGPT it doesn't "know" anything. Rock is a number. Singing is a number. Write is a number.

But it knows the relationships between those numbers, and what other numbers are nearby the area of the network devoted to songs, so it pulls in words and related concepts like a human would.

But it's just numbers with no understanding.

Because it's numbers and not understanding, it can be wrong, either completely or subtly.

Edit: Asking for the list of us presidents has "David D. Eisenhower (1849-1850)" as number 12 (who isn't a person who was ever president). The rest look right, but ChatGPT is subtlety wrong in this case.

Do you see the future being ChatGPT results but with citations? Or is that basically impossible given how it's a "trained model"?

No, ChatGPT doesn't know it's own sources. It's just a trained model. Once the model has been created it's fixed - it can be recreated unlimited times, but it will never tell you the sources for it's output.

Maybe if the network nodes have a source attached to them...

But thinking out loud...

That's not how the number-tokens work. It's at a word level... so "a list of us presidents" is broken down into individual number-tokens for each word, and you can't provide a source for each word.

---

I'm not sure how you combine Google and ChatGPT.

Chat is creative/combinatorial and Google is "just the facts".

ChatGPT and Google are going to have problems going forward. How do both of them determine if the information they find on the internet is from a meat-brain and not a metal-brain.

Happy to be proven wrong.

Maybe by fact-checking its answer?

Question -> "creative" output -> Google -> Summary of links -> Comparison -> confidence level (or re-write) + links that were used for checking

Not so different than how we work in a high-level. I believe that openAI has published a paper called webGPT that has a workflow like this (although not sure its exactly the same)

"condensing/summarizing bots" - never heard of these, will have a play around, thanks.

https://www.reddit.com/r/autotldr/comments/31b9fm/faq_autotl... from 2015

Actually I find it much better than that for exploratory purposes than just getting search terms. The ability to just keep asking questions for clarification is something that the web was meant to provide with web links, but rarely does a good job of it. If it can simply act as a domain expert that I can talk to, it would be a huge win.

But it's not a domain expert: it's a language model designed and trained to produce language that could plausibly have been written by a human on the internet. At best it functions as a well-informed amateur, at worst it hallucinates nonsense but writes it in a way that is very convincing.

To be fair, you just summarized human discourse especially on places like Reddit and HN.

This comment isn’t helpful or a retort sorry.

This makes me think of a quote from one of Dijkstra's lectures:

"In the long run I expect computing science to transcend its parent disciplines, mathematics and logic, by effectively realizing a significant part of Leibniz's Dream of providing symbolic calculation as an alternative to human reasoning. (Please note the difference between "mimicking" and "providing an alternative to": alternatives are allowed to be better.)"

When talking about a tool thats supposedly greater than humans, why should the shortcomings of humans be relevant? The tools we create to surpass our own capabilities should be greater than our own capabilities, not stunted by the same issues.

It may not be a domain expert now, but it easily could be.

For example if you took all the Linux kernel code, the code review comments, the docs, and several of the top books and blogs on kernel development — suddenly you have a system that may be great for new kernel developers to ask questions of. Especially in a community that often isn’t kind to people Jose asking “dumb questions”.

Could it? There's lots of commentary about how it could easily be this or that, but I don't work in ML and have no clue whether it is actually easy or not to tweak ChatGPT to work in such ways.

For example, in my experience ChatGPT isn't very "smart". It has a lot of knowledge but it can't infer any facts from that knowledge. When you ask it to write a program it has no real idea of what it actually does, and you can easily get it to add features it already added, or tell it something is a bug when it really isn't.

This doesn't sound like the stuff you could make a domain expert out of, at least, not out of the box.

I'm not asking it to infer much, but to piece together a fair bit and understnad what I'm asking. For example, here's a chat I had with ChatGPT:

"How does common subexpression elimination work?"

It answered it correctly and gave some basic code examples to demonstrate the concept. Then I followed up with:

"But can it do the elimination if the variables are flipped, but semantically equivalent, like 'y + x' in the example above?"

It again gave what I would consider a correct answer. Note 'x + y' was what was eliminated previously, so I reversed it here.

Then I asked: "Are there cases where the compiler might fail to eliminate common subexpressions?"

And again a good answer.

Now all of this could be found on the web somewhere, but for example the second question didn't show an obvious answer on Google when I searched for it. I'm sure I could find it, but I know this field well. If I was someone new to the field I'd probably spend a lot of time parsing useless articles to find something that answered the question in a way I could understand.

I'm less concerned about it writing code (which is cool). For me, the ability to help me learn an area quickly is far more useful. It doesn't need novel answers, but the ability to understand what I'm asking and answer it. I think it's really close to being able to do this now.

That solves the "garbage training data" problem, but it doesn't solve the "it's just a language model" problem.

If you fine tuned ChatGPT on all the sources you mention, you now have a model that produces results that could plausibly have been written by a domain expert on the Linux kernel, but you don't have a domain expert. It will still hallucinate, because that's a fundamental feature of generative AI, it will just hallucinate much more convincingly.

I get what you're saying, but I'm just not convinced that it will continue to be a huge problem. In some sense, if the state of art is where we're at today with language models, then sure. But I think it'll get better -- in part because I'm not sure humans just aren't souped up language models with some weird optimization functions...

Being a well-informed amateur in everything is pretty impressive though. ChatGPT will be extremely useful if it ever figures out how to say "I don't know".

I don't believe there's any way for a LLM operating alone to recognize when it doesn't know something, because it has no concept of knowing something.

Its one job is to predict the next word in a body of text. That's it. It's good enough at it that for at least half the people here it passes the Turing test with flying colors, but the only kind of confidence level it has is confidence in its prediction of the next word, not confidence in the information conveyed.

If we were to take a language model and hook it up to another system that does have a concept of "knowing" something, I could see us getting somewhere useful—essentially a friendly-to-use search engine over an otherwise normal database.

I don't know if I buy this. It feels like your confidence in what you say is closely tied to "knowing". I'm sure there is more research to do here, but I'm not sure if there is a need to "tie" it to some other system. As it stands today there are definitely things ChatGPT doesn't know and will tell you so. For example, I asked it, why did Donald Trump spank his kids -- and it said, "I do not have information about the parenting practices of Donald Trump".

That said, there are a lot of things it does get wrong, it would be nice for it be better at those. But I do think that, maybe much like humans, there will always be statements it makes, which are not true.

> Its one job is to predict the next word in a body of text.

“Predicting the next word” and “writing” are the same thing; you’re just saying it writes answers in text. There’s nothing about that preventing it from reasoning, and its training goal was more than just “predict the next word” anyway.

"I'm sorry, but I am a text-based AI language model..."

It doesn't matter, it can still be extremely helpful.

For instance, I had fragmented memories of a movie, described what I knew about it (about a boy who lived in a trainstation), and it helped me find a couple movies and then narrow in on the one I was looking for.

These types of queries can be super painful with modern search engines but was easy with ChatGPT and a pleasant experience.

I think people are thinking of this AI in the wrong way - where it is "an expert". To me, I like to think of it as a companion that helps us shape and refine our thoughts and ideas.

This is a good point.

Something I also enjoy about it is the uniform interface. Each answer is presented the same, there's no parsing layout from different sites, or popup modals to dismiss, or long winded intro to get to the answer you're looking for. Of course you can't quite trust what you're told, so this is a bit moot.

I've found its good for getting me started. Need to do a presentation on something? Type it into chatGPT. It will generate what is basically an okay outline. You can expand on what you like, cut what you dont.

For me getting started is typically the most difficult part (thanks adhd) so this is a huge help.

This is definitely the best use case for these models I've heard. Often when I'm researching a field I'm not familiar with the hardest part is just knowing the vocabulary necessary to express what I want to ask.

Ask Jeeves 2.0!

Reply on news.ycombinator.com