Hacker News Reader

> ChatGPT does not ask for clarification, because it is not a knowledge base trying to find an answer. All it does is figure out what character is statistically most likely to come next, it has no heuristic to know that there is a task it hasn't fully completed.

It absolutely can, but it requires the right prompts. If you just say "give me a diagnosis" it won't, but if you prompt it with a "you are a doctor and I am a patient" kind of prompt, it will.

I'll agree that it tends to "close off" conversations without a lot of open endedness - I role-played through a session using myself as a patient with a chronic condition I suffer from and it correctly identified the problem, but I had to push it a little to get it to try to narrow down the problem (instead of "here's a bunch of things it could be" and ending the convo there).

ChatGPT's heuristic may lead it to emit a sequence of words that suggest to English speakers that they should provide clarification, but ChatGPT is congenitally incapable of determining that answering a question requires more information that was provided in the prompt and that clarification is needed. Rephrased, ChatGPT can ask for clarification, but only as a side effect of satisfying it's heuristic.

Calling it a "side effect" is really downplaying things. All of its behavior is just satisfying its heuristic.

Just did this chat on 3.5.

[Me] Boxes that are black with stars have gold inside. Black with circles means silver. Blue with squares means copper and blue with triangles means tin.

Your job is to determine the contents of the box. If you need more information, ask for it.

Example:

User: I have a black box.

Assistant: Does it have stars or circles?

User: Stars.

Assistant: It has gold.

I have a blue box.

[ChatGPT] Does the blue box have squares or triangles?

[Me] Squares

[ChatGPT] According to the information you provided, a blue box with squares contains copper. Therefore, the contents of your blue box are copper.

Interesting example. I wonder if it is building a small world model (a la SHRDLU), or if it is answering based on the most statistically likely words, or what.

It's both - statistically the next word depends on the world model established by the previous words, so it's devoted some of its neurons to that.

"A computer can store and operate on data"

"That's really downplaying things. My computer can play cat videos"

Change your first line to “a computer is congenitally incapable of providing entertainment. It can provide entertainment, but only as a side effect of storing and operating on data” and it would be more applicable to this exchange.

You can ask it to complain if there is a lack of information in the query. You can ask it for what information might be missing. Everything about this is in the prompt and the questions you ask it.. Just asking "what is the diagnosis" leaves out a LOT of context, and what you expect.

> but ChatGPT is congenitally incapable of determining that answering a question requires more information that was provided in the prompt and that clarification is needed.

What's stopping the ChatGPT heuristic / model from determining that? That's within the capabilities of a statistical model.

I would say that ChatGPT like systems could probably be built that would understand when they need more context, but they would fail at effectively eliciting that context from their human users.

I would suggest a read of this ChatGPT transcript (from Twitter today):

https://twitter.com/teropa/status/1643641395754745859

My hunch is that this is exactly what he was expecting. There is a lot of hype around ChatGPT passing the medical exam and this exercise is a counter point to that.

GPT-4 passed the medical exam, not ChatGPT running GPT-3. There's a rather significant difference.

> GPT-4 passed the medical exam, not ChatGPT running GPT-3

With a subscription you can use GPT-4 with ChatGPT. ChatGPT is just the wrapper to the model.

That's not true, ChatGPT is a model. Quote from the ChatGPT announcement post: > ChatGPT is a sibling model to InstructGPT, which is trained to follow an instruction in a prompt and provide a detailed response.

I’m looking at the ChatGPT interface right now (paid account) and I have a “Model:” drop-down at the start of a new chat that says:

- Legacy (GPT-3)

- Default (GPT-3.5)

- GPT-4

The medical exam has very specific questions which define what you're expected to include in the answer. The question asked in this case was nowhere near that detailed, do I don't think they're comparable. To really evaluate something beyond the "random generic user" level, you need to be familiar with the tech as well.

The article really tells us more about the experience of someone with no chatgpt knowledge checking their own symptoms rather than its usability for emergency diagnosis.

any example that doesn't use the current sota isn't a very good counter point to be honest. 3 barely passed. 4 aced it. For all we know, GPT-4 erases most of his concerns (not saying it would).

I don't mean this in any sort of negative context - curating your online presence is a never-ending task - but I believe that the authors primary expectation was to drive traffic to their online accounts by publishing a post on the subject matter. A cursory peek at their accounts show that this was published across all of them with Twitter showing the farthest reach.

> Dr... is vice president of innovation at {company_name}, an innovation incubator for health tech.

"I'm not really sure what he expected here, ChatGPT ..."

Doctoring is quite similar to ITing if you are careful with your similes. In medicals you look for indications and contraindications. In IT you look for magic beans and sometimes look at logfiles.

I have no idea why we (IT) don't follow the medical way and have indications and contras (I do). They have been doing complicated diagnostics way longer than we have.

BTW: It doesn't worry about a char but about which word will come next.

ChatGPT has written several programs for me. One of them was decent after much negotiation. The rest needed some changes.

At one point I told it to change language (BASH to Python) for a script and it did. I spend 20 mins fining stuff and cracked on.

There was an old USENIX symposium where one of the talks was, "If computers had blood, they'd call us doctors." My thought at the time was, "If computers had blood, they'd call us serial killers."

> I have no idea why we (IT) don't follow the medical way and have indications and contras (I do).

Because we have logs. Ignoring them and focusing on the symptoms would be like doctors refusing to do examinations on their patients.

Doctors separate their sources of information into signs (objectively observed) and symptoms (subjectively reported). Both signs and symptoms can be indication or contradictions.

Tangentially I wonder if there's sources of diagnostic info in tech that are closer to symptoms than signs. You might separate out potentially unreliable info reported by the suspect system from more objective info other systems see.

He probably didn't have any expectations. Merely experimentation and observation and maybe a bit of word of warning after seeing the results for anyone thinking of using it in ways it shouldn't be.

> All it does is figure out what character is statistically most likely to come next

To be pedantic, it's a token - not just a character - right?

Yes

> I'm not really sure what he expected here, ChatGPT was not trained to be a doctor, it is far more general than that.

Except that O̶p̶e̶n̶AI.com and many others on the AI hype brigade were showing ChatGPT and GPT-4 to be passing "Medical Licensing Exams" with flying colours and it turns out to be a confident bullshitter offering dangerous untrustworthy medical advice when tested by a medical professional as I said before [0].

So they (O̶p̶e̶n̶AI.com and AI bros) set those high expectations from the start.

> As soon as we have agents that can do a proper feedback loop of querying a LLM consecutively until some heuristic is reached then the kind of AI doctors are looking for will emerge.

No.

A black-box neural network AI model that is unable to transparently explain and reason about its own decisions is fundamental to the trust element that medical professionals are looking for. It is more than just spitting out numbers from a computer.

This doctor is also the one still reviewing the so-called 'diagnosis' from ChatGPT which without him, ChatGPT alone would have certainly killed the patient. The human doctor also gave a transparent explanation about why ChatGPT's 'diagnosis' was incredibly wrong. ChatGPT was confident that it was still right.

[0] https://news.ycombinator.com/item?id=35311996

> This is the same reason ChatGPT cannot yet write programs by itself: in order to do so you'd need to specify the entire program up front (which is exactly what code is). As soon as we have agents that can do a proper feedback loop of querying a LLM consecutively until some heuristic is reached then the kind of AI doctors are looking for will emerge.

https://github.com/Torantulino/Auto-GPT

> ChatGPT does not ask for clarification

It depends on how you use it. Knowing how it works, you can explicitly check "What further questions for the patient would make the diagnosis more certain?"

ChatGPT is not magic - if you ask for diagnosis based on some info, that's exactly what you're going to get.