Hacker News Reader

Comment by bamboozled | Hacker News Reader

I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical).

Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have to really know a subject before it’s properly useful. I find this with programming too. Yes it can generate code but I’ve introduced some decent bugs when over relying on AI.

The plumber I used laughed at my when I told him about there sand thing. He has 40 years experience…

View on HN

I nearly spit my drink out. This is my kind of humor, thanks for sharing.

I've had a decent experience (though not perfect) with identifying and understanding building codes using both Claude and GPT. But I had to be reasonably skeptical and very specific to get to where I needed to go. I would say it helped me figure out the right questions and which parts of the code applied to my scenario, more than it gave the "right" answer the first go round.

I'm a hobby woodworker - I've tried using gemini recently for an advice on how to make some tricky cuts.

If I'd follow any of the suggestions I'd probably be in ER. Even after me pointing out issues and asking it to improve - it'd come up with more and more sophistical ways of doing same fundamentally dangerous actions.

LLMs are AMAZING tools, but they are just that - tools. There's no actual intelligence there. And the confidence with which they spew dangerous BS is stunning.

> I've tried using gemini recently for an advice on how to make some tricky cuts.

C'mon, just use the CNC. Seriously though, what kind of cuts?

Give a single reproduceable example using ChatGPT thinking

I've observed some horrendous electrical device, such as "You should add a second bus bar to your breaker box." (This is not something you ever need to do.)

I mean... you do have to backfill around your drainage pipe, so it's not too far off. Frankly, if you Google the subject people misspeak about "backfilling pipes" too as if the target of the backfill is the pipe itself too not the trench. Garbage in, garbage out.

All the circumstances where ChatGPT has given me shoddy advice fall in three buckets:

1. The internet lacks information, so LLMs will invent answers

2. The internet disagrees, so LLMs sometimes pick some answer without being aware of the others

3. The internet is wrong, so LLMs spew the same nonsense

Knowledge from blue collar trades seems often to in those three buckets. For subjects in healthcare, on the other hand, there are rooms worth of peer reviewed research, textbooks, meta studies, and official sources.

I mean... you do have to backfill around your drainage pipe, so it's not too far off

Really? You think filling you pipe with sand is comparable to backfilling a trench ?

People don't talk about "backfilling trenches", they talk about "backfilling pipes" which is precisely the problem. They also will say things like "when you backfill a pipe, use sand or granite."

It pretty obvious why an LLM would be confused by this.

honestly I think these things cause a form of Gell Mann's Amnesia where when you use them for something you know already, the errors are obvious, but when you use them for something you don't understand already, the output is sufficiently plausible that you can't tell you're being misled.

this makes the tool only useful for things you already know! I mean, just in this thread there's an anecdote from a guy who used it to check a diagnosis, but did he press through other possibilities or ask different questions because the answer was already known?

It’s funny you should say that because I have been using it in the way you describe. I kind of know it could be wrong, but I’m kind of desperate for info so I consult Claude anyway. After stressing hard I realize it was probably wrong, find someone who knows what they’re and actually on about and course correct.

I'd frame it such that LLM advice is best when it's the type that can be quickly or easily confirmed. Like a pointer in the right (or wrong) direction. If it was false, then try again - quick iterations. Taking it at its "word" is the potentially harmful bit.

Usually something as simple as saying “now give me a devils advocate resoonse” will help and of course “verify your answer on the internet” will give you real sources that you can verify.

I have very mild cerebral palsy[1], the doctors were wrong about so many things with my diagnosis back in the mid to late 70s when I was born. My mom (a retired math teacher now with an MBA back then) had to go physically to different libraries out of town and colleges to do research. In 2025, she could have done the same research with ChatGPT and surfaced outside links that’s almost impossible via a web search.

Every web search on CP is inundated with slimy lawyers.

[1] it affects my left hand and slightly my left foot. Properly conditioned, I can run a decent 10 minute mile up to a 15K before the slight unbalance bothers me and I was a part time fitness instructor when I was younger.

The doctor said I was developmentally disabled - I graduated in the top of my class (south GA so take that as you will)

The great thing is the models are sufficiently different enough, that when multiple come to the same conclusion, there is a good chance that conclusion is bound by real data.

And I think this is the advice that should always be doled out when using them for anything mission critical, legal, etc.

All the models are pre-trained on the same one Internet.

"Bound by real data" meaning not hallucinations, which is by far the bigger issue when it comes to "be an expert that does x" that doesn't have a real capability to say "I don't know".

The chance of different models hallucinating the same plausible sounding but incorrect building codes, medical diagnoses, etc, would be incredible unlikely, due to arch differences, training approaches, etc.

So when two concur in that manner, unless they're leaning heavily on the same poisoned datasets, there's a healthy chance the result is correct based on a preponderance of known data.

Reply on news.ycombinator.com