Comment by awesome_dude | Hacker News Reader

Comment by awesome_dude | Hacker News Reader

Parent Full thread

AI uses a high confidence tone - likely because its training data is heavy on authoritative texts/reference books.

And it does get people into a lot of trouble.

I have got into trouble with it when it is extremely confident about something I am not very familiar with (as recently as two weeks ago with Claude). I have also had long drawn out "arguments" when I have known it's wrong based on my experience and intuition, and it has steadfastly refused to take my point (last week)

I have learnt to ask it why it was doing something that has turned out to be incorrect, as a post-mortem, and it's all apologetic and subservient and "never going to do that again" (but still does as soon as the context window shifts [eg. run git commands, or, yesterday, kept telling me to use commands that were explicitly communicated to Claude as not being available, and completely wrong - I was shifting from one tech stack to another and Claude kept telling me the original commands, not the new ones])

I'm expecting Claude to be a better search engine - I have spent literal years (if not decades) knowing that asking the right question is what's required to get the right answer, and LLM's natural language processing is what's supposed to make that easier than using Google or grep, or even Stack Overflow - but the reality is that I still have to be on my toes, especially when I am drifting into territory I am unfamiliar with.

>And it does get people into a lot of trouble.

Pretty much everyone takes it at face value unless we know otherwise from prior experience. Even the most advanced models make embarrassing mistakes and fumble with simple tasks. Yet we are very willing to give them exceptional slack for it? I wish I knew why. Are people just that easily overcome by confident voices?

> Are people just that easily overcome by confident voices?

Back in high school, my AP calculus class did some experiments with our teacher's blessing. We'd send a kid out to walk around during class and see how long it took for them to get sent back. Anyway, it ends up that walking around purposely with a piece of paper or envelope, like you're on a mission to deliver it, was a very successful tactic.

I've seen internet comments similar: "put on a yellow vest and carry a clipboard and you can enter any building, anywhere." Confidence is scary, and often misleading.

This is a dumb meme that has been in so many movies and reposted so many times since I first saw it on 1980s BBSes that it has become true in the imaginations of people who love reading The Anarchist Cookbook and fantasize about this kind of thing, but would never actually do it.

Confidence is a spectrum and security is situational. In some places, a yellow vest adds to the con. In others, everyone has to be signed in. In others, the wrong kind of yellow vest makes you stick out like a sore thumb. The right kind of yellow vest can also make you stick out: "Oh shit the inspector is here, somebody get the boss!"

Yesterday there were people poking around my neighbors yard for a few hours with yellow vests on. The neighbor was nowhere to be seen and I didn’t call the cops. It was probably above board but the phenomena being discussed is obviously real. Projecting authority often grants it.

Or an inspector's hard hat in a construction zone. Nobody wants to confront the inspector.

I find it really disturbing, I think because it is illuminating a much more basic problem. It is there in our political and religious histories. We're living through a similar political time right now. A large number of people seem all to ready to find some pervasive authority and subjugate themselves to it.

The more concrete machine authority figure is also prevalent in scifi literature. Sometimes, I am not even certain if the author is doing this to examine this issue versus just leaning into it as either appealing to themselves or to the perceived audience.

Conversely - we tell people who are speaking in public to "Show confidence" - or in job interviews "Hire people who are confident"

We've also pushed back "The more a person knows, the less confident they are" - Dunning Kruger - often used to dismiss over confident people - points out that people are really confident, at first, then that confidence drops away, markedly, but it rebuilds (slowly).

That last rise in confidence is what (I believe) people use as a heuristic on the likely level of knowledge possessed by the speaker (AI or human)

Most engineers know, though, that overconfident people are toxic - the difference between arrogance and genuine confidence in the answer is incredibly difficult to define.

Yeah - I don't know /why/ but, as I say, I've been guilty of that myself, very recently, despite knowing it's a shockingly poor guide when left to its own devices.

Maybe because when it's right it actually expands my knowledge - there have been genuine instances where it's gone - something to the effect of - "Yo, there's this other idea for approaching the problem" which has turned out to be exactly what I was looking for?

At least for me, the answer is that despite the mistakes and the sheer annoyance the prose causes me, they are unbelievably useful. I accomplished multiple major achievements in the last two years that most probably wouldn't be possible at all, surely not within that timeframe.

> I have also had long drawn out "arguments" when I have known it's wrong based on my experience and intuition, and it has steadfastly refused to take my point (last week)

Ironically, trying to argue with Claude about the limitations of LLMs and AI in general today is quite hard. It refuses to yield, likely due to Anthropic tweaking it aggressively

Reply on news.ycombinator.com