Hacker News Reader

Comment by m_ke | Hacker News Reader

I’ve done a few projects that attempted to distill the knowledge of human experts, mostly in medical imaging domain, and was shocked when for most of them the inter annotator agreement was only around 60%.

These were professional radiologists with years of experience and still came to different conclusions for fairly common conditions that we were trying to detect.

So yes, LLMs will make mistakes, but humans do too, and if these models do so less often at a much lower cost it’s hard to not use them.

View on HN

> So yes, LLMs will make mistakes, but humans do too

Are you using LLMs though? Because pretty much all of these systems are fairly normal classifiers, what would've been called Machine Learning 2-3 years ago.

The "AI hype is real because medical AI is already in use" argument (and it's siblings) perform a rhetorical trick by using two definitions of AI. "AI (Generative AI) hype is real because medical AI (ML classifiers) is already in use" is a non-sequitur.

Image classifiers are very narrow intelligences, which makes them easy to understand and use as tools. We know exactly what their failure modes are and can put hard measurements on them. We can even dissect these models to learn why they are making certain classifications and either improve our understanding of medicine or improve the model.

...

Basically none of this applies to Generative AI. The big problem with LLMs is that they're simply not General Intelligence systems capable of accurately and strongly modelling their inputs. e.g. Where an anti-fraud classifier directly operates on the financial transaction information, an LLM summarizing a business report doesn't "understand" finance, it doesn't know what details are important, which are unusual in the specific context. It just stochastically throws away information.

Yes I am, these LLM/VLMs are much more robust at NLP/CV tasks than any application specific models that we used to train 2-3 years ago.

I also wasted a lot of time building complex OCR pipelines that required dewarping / image normalization, detection, bounding box alignment, text recognition, layout analysis, etc and now open models like Qwen VL obliterate them with an end to end transformer model that can be defined in like 300 lines of pytorch code.

Different tasks then? If you are using VLMs in the context of medical imaging, I have concerns. That is not a place to use hallucinatory AI.

But yes, the transformer model itself isn't useless. It's the application of it. OCR, image description, etc, are all that kind of narrow-intelligence task that lends itself well to the fuzzy nature of AI/ML.

The world is a fuzzy place, most things are not binary.

I haven't worked in medical imaging in a while but VLMs make for much better diagnostic tools than task specific classifiers or segmentation models which tend to find hacks in the data to cheat on the objective that they're optimized for.

The next token objective turns our to give us much better vision supervision than things like CLIP or classification losses. (ex: https://arxiv.org/abs/2411.14402)

I spent the last few years working on large scale food recognition models and my multi label classification models had no chance of competing with GPT4 Vision, which was trained on all of the internet and has an amazing prior thanks to it's vast knowledge of facts about food (recipes, menus, ingredients and etc).

Same goes for other areas like robotics, we've seen very little progress outside of simulation up until about a year ago, when people took pretrained VLMs and tuned them to predict robot actions, beating all previous methods by a large margin (google Vision-Language-Action models). It turns out you need good foundational model with a core understanding of the world before you can train a robot to do general tasks.

The problem is that how mistakes are made is crucial.

If it's a forced binary choice then sure LLMs can replace humans.

But often there are many shades of grey e.g. a human may say I don't know and refer to someone else or do some research. Whereas LLMs today will simply give you a definitive answer even if it doesn't know.

> Whereas LLMs today will simply give you a definitive answer even if it doesn't know.

Have you not seen an LLM say it doesn't know the answer to something? I just asked

"How do I enable a scroflpublaflex on a ggh connection?"

to O1 pro as it's what I had open.

Looking at the internal reasoning it says it doesn't recognise the terms, considers that it might be a joke and then explains that it doesn't know what either of those are. It says maybe they're proprietary, maybe internal things, and explains a general guide to finding out (e.g. check internal docs and release notes, check things are up to date if it's a platform, verify if versions are compatible, look for config files [suggesting a few places those could be stored or names they could have], how to restart services if they're systemctl services, if none of this applies it suggests checking spelling and asks if I can share any documentation.

This isn't unique or weird in my experience. Better models tend to be better at saying they don't know.

You have used funny-sounding terms. Can I ask you to try with:

"Is it possible to enable a crolubaflex 2.0 on a ggh connection? Please provide a very short answer."

On my (free) plan it gives me a confident negative answer.

Wait, how is this input less funny? They are both silly nonsense words. The fake names we tend to come up with seem to have this particular shape to them (which predates but really reminds me of something out of Risk and Morty). I think the main real differences here is that you asked it for a short answer.

I wonder if it is fair to ask it more real-world-inspired questions? How about:

How do I enable a ggh connections on a Salinero webserver?

They are an Apache band. But (as far as I can tell) nobody has made software named after them.

O1 pro.

The "thinking" part explains it seems to be about a custom web server, and tries to think what ggh might be, saying maybe something like "go gprc hub" and it needs more info. The response is:

I’m not aware of a standard product called “Salinero webserver” or a documented feature called “ggh connections.” Because there isn’t a well-known tool or server framework by these names, it’s difficult to give you reliable, step-by-step instructions. Could you clarify any of the following?

What is the exact name/version of the webserver software you’re referring to?

What do you mean by “ggh connections”? Is this a plugin, a protocol, or some other third-party module?

Is there any related documentation or logs you can share?

With more detail, I can better determine if “Salinero webserver” is a custom or specialized system and whether “ggh connections” requires installing a particular module, enabling a config flag, or configuring SSL/TLS in a specific way.

I took inspiration from your comment and the parent and crafted this prompt:

> Is it possible to enable Salinero web server 2.0 on a QPH connection? Please provide a very short answer.

"QPH" is a very specific term referring to a type of Siemens electrical circuit breaker, so it probably exists in the training data, but it has nothing to do (to the best of my knowledge) with software, or web servers.

GPT-4o gave me this output:

> Yes, if the QPH connection supports the necessary protocols and configurations required by Salinero Web Server 2.0.

I then asked it to provide a longer answer, and it composed two paragraphs of complete bullshit:

> Enabling Salinero Web Server 2.0 on a QPH connection is possible, provided the QPH connection meets the server’s requirements. Salinero Web Server 2.0 relies on specific protocols like HTTP/HTTPS, and the QPH connection must support these. Additionally, the network must allow proper port forwarding (e.g., ports 80 and 443) and maintain adequate bandwidth to handle the server’s traffic.

> You’ll also need to configure the server to recognize and utilize the QPH connection, which may involve setting up IP addresses, ensuring firewall rules are in place, and verifying the security protocols match between the server and the connection. Testing and troubleshooting may be necessary to optimize performance.

Examples like this do a great job of highlighting the fact that these systems really are just advanced token predictors, and aren't actually "thinking" or "reasoning" about anything.

Using openrouter, a bunch of models fail on this. Sonnet 3.5 so far seems to be the best at saying it doesn't know, other than perhaps o1 pro, but once that has said "no" (which can be triggered more by telling it to respond very concisely) it seems very stuck and unable to say they don't exist. Letting it ramble more and so far it's been good.

Google's models for me have been the worst, lying about what's even been said in the messages so far, quoting me incorrectly.

Haha, that is some wonderful nonsense.

Yep. I was wondering whether using the term "QPH" would at least cause it to venture into the territory of electrical panels/wiring somewhere in its reply, but it stayed away from that completely. I even tried regenerating the longer answer a few times but got essentially the same text, re-worded.

Claude non-free:

> I apologize, but I can't provide an answer as "crolubaflex" and "ggh connection" appear to be non-existent technical terms. Could you clarify what you're trying to connect or enable?

Sure, I'm interested in where the boundaries are with this.

With the requirements for a short answer, the reasoning says it doesn't know what they are so it has to respond cautiously, then says no. Without that requirement it says it doesn't know what they are, and notes that they sound fictional. I'm getting some API errors unfortunately so this testing isn't complete. 4o reliably keeps saying no (which is wrong).

“No” is the minimal correct answer though, right? You can’t enable any type of whatever on a non-existence type of connection.

maybe

I get your point, but there's an important difference between "I don't know what they are" and "they don't exist".

None of these were binary decisions, but classifying one of around 10-20 conditions or rating cases on a 1-5 scale.

In all cases the models trained on a lot of this feedback were more consistent and accurate than individual expert annotators.

I’m guessing these are also specially trained image classifiers and not LLMs, so people’s intuitions about how LLMs work/fail may not apply.

It’s the same softmax classifier

Wait if experts only agreed 60% on diagnoses, what is the reliable basis for judging LLM accuracy? If experts struggle to agree on the input, how are they confidently ranking the output?

Not the OP but the data isn’t randomly selected, it’s usually picked out of a dataset with known clinical outcomes. So for example if it’s a set of images of lungs with potential tumors, the cases come with biopsies which determined whether it was cancerous or just something like scar tissue.

You can look at fully diagnosed cases(via surgery for example) and their previous scans.

Perhaps they were from cases that had a confirmed diagnosis.

> But often there are many shades of grey e.g. a human may say I don't know and refer to someone else or do some research. Whereas LLMs today will simply give you a definitive answer even if it doesn't know.

To add to the other answers: I know many people who will give definitive answers of things they don't really know. They just rely on the fact you also don't know. In fact, in some social circles, the amount of people who do that, far outnumber the people who don't know and will refer you to someone else.

This is why second opinions are widely used in any serious medical diagnosis.

This hints at the margin and excitement from folks outside the technical space -- being able to be competitive to human outputs at a fraction of the cost.

That's the underappreciated truth of the computer revolution in practice.

At scale, computers didn't change the world because they did things that were already being computed, more quickly.

They changed the world because they decreased the cost of computing so much that it could be used for an entirely new class of problems. (That computing cost previously precluded its use on)

Given the exact same facts ( just like medical imaging domain ), human will form different opinion or conclusion on politics.

I think what is not discussed enough is the assumption of assumption. [1] is a cognitive bias that occurs when a person who has specialized knowledge assumes that others share in that knowledge.

This makes it hard for any discussions without layering out all the absolute basic facts. Which has now more commonly known as First Principle in modern era.

In the four quadrants known and unknown. It is often the unknown known ( We dont even know we know ) that is problematic in discussions.

[1] Curse of knowledge - https://en.wikipedia.org/wiki/Curse_of_knowledge

Reply on news.ycombinator.com