Hacker News Reader

Comment by Frost1x | Hacker News Reader

I work at an intersection of tech, applied research, and science.

Something I’ve noticed in collaboration that does occur is an increased confidence in people outside their domains to say things with conviction. I have people who have limited experience with software pushing out layers and layers of abstracted code that’s fairly sophisticated but often misguided in intent who will say what they’re doing is correct, with conviction.

I also hear a lot more questioning people in their domains and challenging opinions, then hearing what I can only imagine are fragmented pieces of conversations they had with an LLM thinking through some argument. Then there’s silence when you discuss shortcomings, then they come back later with their memorized fragments of what you said, combined with memorized fragments of the LLM response to the argument.

It’s occurring, a lot more. People are treating their LLMs in collaboration as a source of truth and using then to focus on their specific path or goals they think or have bias towards going down, vs just opening discussing things, considering tradeoffs from experts multiple disciplines weigh in on and then taking an approach that everyone finds most agreeable.

It’s making me want to be a lot less collaborative with such individuals. I don’t want to sit around and refute Claude text outputs all day.

View on HN

Wow, oddly similar thing happened to me. Anecdotally, I was working with a non-developer. She was building a reporting dashboard with Claude. I pointed out that the system she built was using the filesystem to store one line notes about the report, and I said "I would probably tell the agent to use a database for that, but it's not the end of the world if it works". And she replied "Claude said the developer's right, a database would be a better fit..."

So, a non-developer, who has no understanding of the underlying systems they are working with, fact-checked a developer with years of experience.

I'm not against fact checking, but it absolutely felt like a punch in the gut coming from a non-dev. There was no "trust" there. She couldn't rely on my expertise, she had to go back to her "source of truth" (Claude) and asked for a second opinion on something that she wouldn't be able to verify.

This is why collaboration exists. I trust my colleagues to give me insight+advice on things that I do not know. I don't ask AI to vet their decisions, because at some level there are judgement calls, and I WANT to trust them because it makes my life easier.

Treating AI as if the output is factual is just a misuse of the tools, I think. It's significantly more effective when given expert direction and the output can be verified by... an expert.

>>I'm not against fact checking, but it absolutely felt like a punch in the gut coming from a non-dev. There was no "trust" there. She couldn't rely on my expertise, she had to go back to her "source of truth" (Claude) and asked for a second opinion on something that she wouldn't be able to verify.

This is probably how physicians feel when informed patients (whether using LLMs or not) ask questions (good or bad) during their patient encounters.

Doctor:

I don’t mind the questions. I mind the assertions of fact.

I would love to be able to trust that a doctor knows more about my problem than I do, but every time I see one they clearly haven't even read the information I just told the nurse 15 minutes ago.

This is a different situation. A doctor is a domain expert, but you are in possession of vastly more empirical data about your specific body. Getting to a good result is going to be easier with collaboration, rather than one of you confidently telling the other what the problem/solution is.

Yeah I've felt that gut punch all the time. Seems to happen with every technology. In the early days of GPS, people would ask for directions to the gas station then pull it up on their phone map and go "yeah, you're right!" So thanks for wasting my time, insulting my integrity, and killing my faith in humanity all in 3 words. And if you really need two opinions, it's so easy to just reverse the situation, "my GPS says there's an Exxon in two miles if I turn left here and then right, is that correct?"

I remember fifteen years ago I would tell people leaving my house "oh yeah, ignore your GPS that tells you to go that way, just down the road here and left turn; that'll get you on the highway much faster."

Nowadays obviously Google Maps gets it right, but it was interesting that period where you had to actively preempt everyone's instinct to just rely on the little table hanging off their dashboard.

Kind of like getting people to ignore generative AI today, right?

there was an institution my son went to for therapy for his various handicaps, and one of the major taxi routing solutions in Denmark had the route wrong, I think the one based on Google maps actually, so when you called for a taxi it always drove nearly a block away that you had to walk through a couple of back alleys to get to. And then the taxi driver would call me up and complain I wasn't there, and I had to guide them to the place where they actually had to be at over the phone.

Sometimes they were not capable of following the instructions though.

I can see how the situation is annoying there; I've had a similar frustration when doing manual code review for someone's obviously AI-slop PR only to have their bot auto-respond to my comments. There's an asymmetry of effort that feels unfair. Like, don't ask for manual review of something that contains lots of things that your own manual review should have caught first, and if I provide human feedback, I expect human responses.

That said, I wouldn't necessarily take it at face value that the person you were working with trusted the bot more than you— at the end of the day, the bot was doing all the implementation anyway, so all the interactions between her and the bot were probably framed in language like "what if we" or "I think it would be better if", so her saying "my programmer colleague thinks X" would naturally generate that "yeah your colleague is right, let's go that direction, makes sense" that she passed back to you, probably thinking it was a complement.

Maybe the real frustration is her not understanding the value-add that still exists there when someone with real expertise and more context can step in and provide that nudge in the right direction.

Sure - I don't hold it against them. It was just surprising to read a "Claude said you're right" message. (Yes, I know that I'm right, or I wouldn't have said it)

Didn't ruin my week or anything, but that was the writing on the wall for me.

Technical expertise is no longer viewed in the same light when software development has no barrier to entry.

True, though I wonder if 2026's "Claude make me a minecraft plugin" is 2006's "View Source" button.

Web development took a long time to gain respect in part due to the accessibility of it. When everyone had a nephew who "makes websites" and anyone could get started with a $5/mo Dreamhost account, it was harder to see why a consultant was worth $200/hr to do the job properly.

It'll be interesting to see how this plays out, particularly accounting for bots continuing to get smarter and smarter. Is the "craft" of software engineering dying before our eyes? Or will this be like webdev, where there's an explosion of access and ideas for a time and then it settles back down into being a thing that normal people mostly leave to the nerds.

Annoyingly pedantically speaking, a filesystem is a type of database.

God do I wish that was her response to my suggestion.

"Fair enough - go nuts. Let me know when Claude tells you to transition to Postgres because I'll need to get you access credentials"

wow that wouldnt feel like a gut punch at all to me, it's normal for inexperienced people to not really understand what kinds of knowledge experts have or don't. I certainly come back from doctors appointments and google/gemini chat for whatever came up to get more detail. this person going to claude and getting immediate confirmation that you were exactly correct should have felt great, she would be like "wow, this guy's good, huh."

To be generous, it seems like the most natural way to phrase what happened. The non-developer punched in "should this be a database instead?" and it will say "You are right to question it! I was sloppy when .. blah blah blah blah blah"

I doubt the human went into a whole diatribe with the CLI about how the developer on the project with this many years was saying this or that but she wanted Claude to second guess them.

It's just saying "ah yeah, that went through without issue."

non-dev AI psychosis doesn’t bother me

I’ve seen how the market treats vibe coders that werent previoisly engineers. Coinbase’s CEO bragged about PM’s deploying code and everyone now thinks they are totally incompetent anytime a small UX problem occurs.

Engineers with prior experience now check a box, even if they are doing the same thing. Like a Compliance Officer or Cybersecurity professional checks a box

> Treating AI as if the output is factual is just a misuse of the tools, I think.

Especially because it's very often wrong. AI tools lack the necessary context of the specific situation and the surrounding technology environment to make appropriate suggestions for anything more complex than clean sheet designs and toy prototypes. There is too much complexity that is hidden in domain knowledge and not even exposed fully in documentation for any AI tool currently to ingest it all into its own context to give you an accurate answer, even setting aside hallucinations and that these tools are incapable of reasoning.

I have this same gut punch constantly. AI has destroyed functional outcomes in corporations, because managers are all deep in AI psychosis deciding that whatever the tool says is perfectly correct at all times. We've replaced analysis paralysis with a complete absence of thought from any human in high-level impactful strategy discussions, everything is entirely being abdicated to AI tools in so many companies. It's ridiculous.

AI will radically change the economy, but not because its replacing jobs, but because its eroding trust, domain knowledge and expertise, and thinking/creativity while being a massive waste of money and total boondoggle. This bubble can't burst soon enough so we can get on with doing shit that actually matters for people that actually helps them.

I'd bet it depends on what they write to the LLM, since sycophancy is so strong in these things. If they write in a fairly agnostic way, it might default to sycophancy to you the dev, while if they write it in an opinionated way it might default to sycophancy to them. Regardless of who is right.

I think I could qualify as an expert on various topics relative to many average people. But it always bothered me when experts wanted to "just be believed". In your example, it doesn't long to explain why a database is better than the file system in this instance and a moderately intelligent should be able to understand and not need you or Claude as the source of truth.

So I think it can be good people are able to "AI" to question expert opinions - though naturally, as you say, it's bad if people take the AI as a single authority akin to an expert.

So, I did let her know that a storage volume was going to be more difficult to manage, could result in lost data, and would make the data harder to work with - I mentioned it's not the end of the world if it works, it was just a professional suggestion.

To me, there are different levels to "just being believed". If a mechanic tells you your car isn't working because you don't have an alternator, does that really need scrutiny? If you don't understand the subject matter, at some point you have to "just believe" someone else.

Especially for a non-technical user, trusting the software developers at your company on technical advice should be a given.

>I'm not against fact checking, but it absolutely felt like a punch in the gut coming from a non-dev. There was no "trust" there.

Maybe I'm just cynical but I don't think I ever see that much trust for anyone's opinion or expertise; devs think managers don't know what they're doing, managers think devs don't, people on HN think lawyers are wrong when they state their legal opinion, devs think other devs are wrong about their areas of expertise even though they have no reason to think that, hell I've worked with people who basically lie for some reason about technical issues that they have no expertise in, in order to get some benefit that is unclear to me. I've also seen people of great technical expertise gaslight industries in order to derive very clear benefits for their employers, so I'm not sure why there should be trust.

I'm sure that various AI agents will maybe add another interesting wrinkle to this, but I sort of feel like you must have had a nice life up until AI if that was feeling like a gut punch.

Yup, the arrogance when barging into unfamiliar domain that was previously reserved mainly to physics grads seems to have spread everywhere. I also had people opining on my expert area via their LLMs and the problem is they can't even ask the model the right question, let alone evaluate the nuance of their answer.

As an unfamiliar, I find it interesting how you specifically called out physics grads in the pre-LLM era. Is that just snark or is there some widespread truth / stereotype of that behavior?

It might be a reference to this: https://xkcd.com/793/

I prefer the SMBC version: https://www.smbc-comics.com/comic/2012-03-21

I suspect the person you replied to only happened to see that from physics grads. I have seen it in all sorts of people and I sometimes did it myself when I was young (I am not a physics grad).

Not the parent, but it's half joke half truth. Anyone who's done a lot of "first principles" work falls into this trap a bit, it's an issue across stem fields. Most grow out of it, or at least learn to qualify their statements for the audience.

Its snark/stereotype, but there is some truth. But my take, is physics sort of took over a lot of chemistry, biology.. and to them its the answer to all questions, because in the end it just molecules. There is some truth to that but..

Obligatory xkcd, everything is math. https://xkcd.com/435/

If you are very smart, and/or have an incredibly high tolerance for pain/repetition, you can solve amazingly complex issues, with awful code.

Both of these statements are true for physicists

Physics, and to a large degree pure mathematics, has attracted a critical mass of the personality type that needs to feel like the smartest person in the room in any given situation. It has given them a slightly notorious reputation in academia.

From a one-time physics grad student, I do think there advantages to being ... like this.

The flip side of being confidently wrong sometimes is that we treat understanding new systems as something approachable. So, yeah, it's annoying that we don't stay in our lane, but, from my boss's perspective, I'm the only one who's actually willing to just read code, investigate, possibly read a paper, and figure out what's going on instead of punting and saying it's someone else's turf I'm blocked until they get back to me.

Like, ultimately you can't be a knowledge worker by just waiting for answers and instruction from experts and relaying them back and forth. Eventually you have to understand something. And part of that process is being wrong and annoying people.

It is quite possible to be wrong and not annoy people, and it's not even terribly difficult to learn. It does require a bit of humility and introspection though.

You don't have to not stray into areas you don't understand, but be humble when doing so - if you think something is being done in a stupid way, you probably don't understand it yet.

I agree!

Missteps are inevitable, though.

Like your sibling comment I saw this mainly with physics people doing data science:

> Eventually you have to understand something. And part of that process is being wrong and annoying people.

There is nothing more frustrating as a colleague than having a smart person walk in, insist you're all wrong, and then burn all the energy and goodwill in the organisation having to rebuild absolutely everything from the principles they currently understand because they refuse to listen to the experience that has built up already (in the entire field, let alone the organisation), only to produce something worse on every dimension than what already existed.

But there is a difference between not listening to experience and examining whether the resulting implementation/decisions based on the experience are actually valid. I doubt that someone smart (as you put it) would deliberately want things redone because they don't understand it, that's a trait borne of arrogance not intelligence.

At least in my experience (as a physics grad), me and all my fellow grads are much more likely to apply Chesterton's fence and then make decisions than not doing that and steamrolling existing decisions.

> I doubt that someone smart (as you put it) would deliberately want things redone because they don't understand it

You're actually doing the problem behavior right here: denying the validity of the experience of seeing exactly this happen, not as a one off, but repeatedly in completely separate situations and institutions.

> that's a trait borne of arrogance not intelligence.

I mean the problem is they're arrogant. The two aren't mutually exclusive, far from it.

My point is that wanting things redone because you don’t understand it is dumb behaviour, not smart behaviour.

There wasn’t even a denial of the validity of experience, I was just trying to examine the problem - if this behaviour is something you see often exhibited by “smart” people then: either you are unable to convince them of your side of the argument for different reasons, or you are not actually arguing with smart people. Do you see the distinction?

> The two aren't mutually exclusive, far from it.

Intelligence and arrogance are somewhat contra-indicative. While you can be intelligent and arrogant, you can’t really learn things without the humility to say you don’t know stuff. In my experience, most physics grads I know constantly say they don’t know stuff, way more than people from other disciplines.

> My point is that wanting things redone because you don’t understand it is dumb behaviour, not smart behaviour.

Smart people can do very non smart things when they feel their status as smartest-person-in-the-room is threatened, which is the entire point.

> There wasn’t even a denial of the validity of experience, I was just trying to examine the problem - if this behaviour is something you see often exhibited by “smart” people then: either you are unable to convince them of your side of the argument for different reasons, or you are not actually arguing with smart people. Do you see the distinction?

Now you're gaslighting, bluntly.

> Intelligence and arrogance are somewhat contra-indicative.

I've never seen any evidence of this, but a _lot_ that many physics (and pure mathematics) grads are enormously more arrogant than the general population, enough that the trend under discussion has actual basis.

I was responding to you in the context of the GP.

> The flip side of being confidently wrong sometimes is that we treat understanding new systems as something approachable. So, yeah, it's annoying that we don't stay in our lane, but, from my boss's perspective, I'm the only one who's actually willing to just read code, investigate, possibly read a paper, and figure out what's going on instead of punting and saying it's someone else's turf I'm blocked until they get back to me.

Eventually you have to understand something. And part of that process is being wrong and annoying people.

GP here is suggesting that accepting established status quo at face value (i.e relying solely on other people's experience/letting them deal with problems you face in their domain without being able to examine why they arrived at the decisions they did) can lead to worse outcomes, and I agree.

Your response was that it is frustrating to have someone walk in and question things (precisely what GP is suggesting), only to have the final product of whatever decisions these people make be worse than the status quo (not what GP is suggesting).

I responded by saying - there is a difference between not listening to experience and examining whether the resulting implementation/decisions based on the experience are actually valid.

You did not address that. Is there not a difference between these two things? GP is suggesting to do the latter, not the former. i.e question why things are the way they are, crucially not that people should make decisions based on things we don't understand about the status quo.

You only responded to - * I doubt that someone smart (as you put it) would deliberately want things redone because they don't understand it, that's a trait borne of arrogance not intelligence.*

Your response claimed that I am denying the validity of your experience, when I made no such statement. All I said was there is a difference between questioning things and forcing decisions based on things they dont understand - smart people don't do the latter, arrogant people do. Someone doing what you are suggesting, is not doing what GP is suggesting, they are different. The former is smart, and the latter is not smart.

When I pointed this out here: I was just trying to examine the problem - if this behaviour is something you see often exhibited by “smart” people then: either you are unable to convince them of your side of the argument for different reasons, or you are not actually arguing with smart people. Do you see the distinction?

You accuse me of gaslighting you.

> Intelligence and arrogance are somewhat contra-indicative.

There might be a trend of this in academia (which is where the xkcd joke stems from), but in my personal experience I dont see this much in industry, and you can disagree with that.

I also don't see an xkcd joke being an actual basis of anything either. You are making declarative statements - but a _lot_ that many physics (and pure mathematics) grads are enormously more arrogant than the general population, enough that the trend under discussion has actual basis. - without any backing of this other than anecdata, and I was pointing out a different anecdotal experience.

> You only responded to - * I doubt that someone smart (as you put it) would deliberately want things redone because they don't understand it, that's a trait borne of arrogance not intelligence.*

> Your response claimed that I am denying the validity of your experience, when I made no such statement.

But it is exactly what you are doing in the exact quote of yours you even put there.

I am being nice, but it's quite clear you are a near perfect example of this stereotype (if actually a relatively polite one), right down to the wall-of-text when cornered.

I'm being nice too and all I've been trying to do in this thread is point at a distinction between GP's point and yours which you seem unwilling to engage with.

It is valuable to ask questions about why things are the way they are in order to understand the fundamentals of it vs we should change these things because I am 'smarter than you' and I don't understand why things are done this way (which I called dumb, and not smart). Perhaps the issue is that these two behaviours are easily confused.

I had to write the long comment to illustrate this difference. I'm saying in mine and GP's definition doing what you are suggesting people usually do is not smart, its dumb, and that is not what GP or I are suggesting people do.

> It is valuable to ask questions about why things are the way they are in order to understand the fundamentals of it vs we should change these things because I am 'smarter than you' and I don't understand why things are done this way (which I called dumb, and not smart).

But you expressed doubt of stated (and frankly not even remotely subjective) experience, so if we dropped you in an organisation your immediate response to anything you don't understand is "That doesn't match my mental model, so I doubt it actually happened", followed by either everyone giving up or you insisting everyone persuade you of absolutely every last detail of everything. It's enormously tedious, and the reason it comes across as arrogant is the demands put on everyone else to explain everything to you because you clearly see everyone else as incapable of the same level of insights that you are, all the while you are telling people what they experience is them being mistaken.

Just look at the way you questioned whether or not I could tell if someone is actually smart or not. It's just plain insulting. And it's not just me, it's everyone that has this same experience which forms the basis of the stereotype.

You seem to be imputing a lot of intent into my comment and then extending that to value judgements about smartness etc. I don't know what any of this has to do with what GP was saying. Yeah, there's a balance between questioning everything and questioning nothing, but there are a lot of bureaucratic things that happen in organizations and its generally a good thing to question certain things that don't have clear explanations, isn't there?

The doubt was simply about whether or not what GP is saying is smart behaviour to exhibit - Eventually you have to understand something. And part of that process is being wrong and annoying people. is the same behaviour that you are saying you've experienced ( i.e you are talking about different things).

I'm not even questioning if you can tell if someone is smart or not - its just obvious that forcing decisions based off things that are not understood is not a smart thing to do - which..we all agree on -, and neither I nor GP were saying people should do this?

Maybe I could've phrased it better but my whole point was GP says X, you are saying Y. I think there is a distinction between X and Y and I doubt that the people GP is talking about are the same people you are talking about (hence, smart as you put it vs smart as GP put it).

I see you don't understand the point, and you're doing that thing of imposing on me trying to help you understand it, which seems highly unlikely to work. Going back, again, to your quote:

> I doubt that someone smart (as you put it) would deliberately want things redone because they don't understand it

And yet that is quite simply exactly what I have seen multiple times in multiple organisations on multiple continents, but for some reason you "doubt" this because it doesn't fit what you think of as "smart behaviour" when spelled out like that, and then spend ages arguing that now there's no intent to this comment? Or any of the others . . .

I am going to do what I always end up doing in such situations, of just giving up and walking away, which every normal person does way earlier in proceedings. Ils sont fou ces physiciens.

Its clear that you are taking great offence to that particular quote and determined to misunderstand my point due to your pre-existing bias, despite my many attempts to clarify it, so let me finish the quote for you.

I doubt, that someone smart (as you put it) ..and not as GP put it..who..would deliberately want things redone because they don't understand it..are..the same people that GP is talking about when they say there are advantages to being "like this".

This is no longer fruitful, but I only hope that this thread might to be a reason to more charitable in your interpretation next time regarding such a discussion.

It’s not about being in your lane. It’s about approaching things with an open mind plus the desire to solve the problem instead or proving yourself right.

Yeah, I guess my point is that IMO a lot of people have so little interest in solving the problem that they aren't even approaching it at all, open mind or no.

It's not my experience that I'm constantly arguing with people, for what it's worth. I get positive feedback for being willing to work on areas of the codebase I'm not already familiar with, and develop an understanding of areas of the codebase no one is familiar with.

It's possible to step out of your area of expertise in a way that's confident but also humble. Arrogance is not necessary. The person who signs everyone's paychecks might be making it mandatory, but that's not quite the same thing.

You can get pretty much all the same benefits with additional upside and fewer downsides by just showing curiosity and eagerness to learn from others' expertise. Yes, on the internet you might get better answers by being confidently wrong rather than asking for help, but when talking to other humans directly you'll get a lot more people who will just try to end the conversation as quickly as they can if you're arrogant, whereas people tend to respond really well to respecting their intelligence and experience.

I've personally found that doing the opposite of what you say and being willing to look like the dumbest person in the room by asking whatever questions I need to for my own understanding often ends up working out quite well; people are often wonderfully willing to share their knowledge with me even if I feel like I'm asking something very basic as long as I'm actually being friendly and humble, and there are plenty of times where there have been others who told me they have same question but didn't feel comfortable asking.

In some ways, it's similar to when I played bass in two different bands in college. In one of them, I was probably the most talented musician in the group, and it was fairly boring for me to practice with them, but we needed it as a group, whereas the other was with a friend of mine who was an insanely talented guitarist possibly more talented in music than pretty much anyone else I ever met. I loved practicing in the second one because I'd learn so much from him just by exposure and getting to pick his brain, but I imagine it got boring for him sometimes as well having to wait for me to "catch up" to what was easy for him. Being the smartest one in the room feels like it would get old after a while, but having smarter people around to learn from is endlessly engaging.

    > we treat understanding new systems as something approachable

And, computer scientists, mechanical engineers, and chemical engineers don't?

    > from my boss's perspective, I'm the only one who's actually willing to just read code, investigate, possibly read a paper, and figure out what's going on instead of punting and saying it's someone else's turf I'm blocked until they get back to me.

Humblebrag alert! This has little to do with being a physics grad student, and much more to do with your motivation.

I guess all I'm doing here is arguing that a little hubris can help a person avoid being timid, and far be it from me to say that computer scientists, mechanical engineers, and chemical engineers have not been imbued with any hubris by their educations.

I've experienced it in industry, too. I recently got out of data science in part because I got tired of working with people who, emboldened by their PhDs in some completely other field, liked to patiently but condescendingly mansplain common, basic misconceptions about my area of expertise to me.

(And it got so much worse once they started using LLMs to aid them in their efforts. Glazing as a service is a hell of a drug.)

So much easier, now that I am a lowly software engineer and can't be held responsible for a certain class of decisions, to just step back and let them be wrong.

> Glazing as a service is a hell of a drug.

You my friend made my day.

As a mathematician, this is far from my experience of actual professional mathematicians, because they encounter people much smarter than them quite often. Maybe more true among graduate students, however.

This thread started by referencing grad students, and I can vouch for it from personal experience with physics grad students.

The best/smartest professionals I meet operate this way (because we know we're wrong quite a bit and don't know everything). It's definitely a personality trait. Inquisitive, curious, intelligent, but also there's some metacognition involved where we can be confident in what we do NOT know.

In contrast, I studied chemical engineering for a short stint. First year, first semester chemistry classes and labs were riddled with these types of students. Tell you you're wrong, IQ 197, scoff when you ask a question, etc...

Listen, if you're the smartest person alive that's fantastic, but please don't talk down to me or the mere mortals that I prefer to hang around with. We get along much better and have a lot more fun saying "I don't know, let's find out".

I don't know why mathematicians got lumped in with physicists further down the thread. In my experience this is a well known stereotype of physicists and while of course not universal you don't have to look very hard to see where it came from. In my experience mathematicians stay very much in their lane and often seem to be overly modest even within it.

I think that's a common trait of tenured academics in the top 100 universities in the US. Don't know about Europe and elsewhere though. Not so much of nontenure track sorts who do all their dirty work.

It’s a very common experience that people in the physics-adjacent sciences have.

It’s so common that there’s an xkcd about it:

https://xkcd.com/793/

It's common enough to have an xkcd for it at least (then again, what doesn't?) https://xkcd.com/793/

It's a widespread stereotype at least: https://xkcd.com/793/

like a prosthetic dunning-kruger....

I hypothesize that with more people on a daily basis encountering information and claims that require those skills, there will be a time in the future when those skills are much more enhanced.

Currently, the leader of USA national AI policy (Trump admin and OpenAI) is a history Bachelors with no technical training or advanced training of any kind, Dean Ball.

>arrogance when barging into unfamiliar domain

describes nearly every tech attempt at "disruption"

There is a reason this exists: https://xkcd.com/793/

I sadly share this as a physicist. To be fair, sometimes this arrogance can be useful. I worked at the boundary between domains my entire career where there aren't "experts" per se. But yeah, barging into an established domain without some humility generally leads to embarrassment - especially Nobel winners (e.g. https://www.newsweek.com/nobel-prize-winner-who-doesnt-belie...)

Hilarious. Do they preface the conversation with "Well, Claude said..."?

With my younger coworkers it's their defence when I push back on them.

I also work at an intersection of tech, applied research and science. My experience has been different. It's been surprising how controlled my collaborators/coworkers have been with their reliance on AI.

Sure, they use it a lot, but despite the stereotypes against physicists in some of the replies to you, they have shown a good ability to catch themselves before leaning too hard on what an LLM tells them about things they are not experts in, and I haven't had any interactions where I felt like I was just arguing with a meat proxy.

Maybe it's because in the environment I'm in; it's relatively easy to just ask someone who is an expert in the topic for their advice.

My feeling is that this kind of overconfidence outside of one's domain is largely a thing for people with little "physical reality" experience. When all you deal with is the very flexible digital world, it becomes very easy to ignore how deep the knowledge and intuition goes in things that are directly constrained by reality. I consider myself to have been in this category too (CE background), though I have been making efforts to improve.

> I also work at an intersection of tech, applied research and science. My experience has been different. It's been surprising how controlled my collaborators/coworkers have been with their reliance on AI.

It's similar to the pattern I've noticed with experienced programmers I know and respect, the really good ones do seem to use AI but they are deeply sceptical about its claimed capabilities so they check/verify/assess what it's good for and use it for that, the tier down is less questioning and just accepts whatever it generates as gospel and it seems to degenerate the further you go down.

It seems like as soon as the AI states something confidently and is wrong in the domain in which the user is an expert, it loses a lot of credibility and people become much more wary of it in general but that requires you to be able to see that it is wrong otherwise it becomes a "bullshit baffles brains" generator.

It's useful to remember those times you've seen it do that when you are asking it something you aren't an expert in and then verify it's answer a different way.

This resonates a lot with me. I'm the CTO of a software factory-ish company. I get daily emails from clients running our decisions through LLMs and asking for ridiculous stuff that would multiply the implementation cost for them, just because Claude said so. No, your 3 users app doesn't need to be SOC2 compliant. No, of course we don't have triple zone redundancy while you're paying a 50 USD AWS bill.

Most of the time, these are just concerns triggered by chatting by an over-zealous LLM; in the worst cases, they are demands.

I'm usually able to clear concerns and disarticulate the LLM by explaining to customers how much more expensive everything would be if we did things like that. But it feels so frustrating, it's like you have to prove yourself everytime and justify every decision. These interactions have made me question my future in the industry, if I'm willing to keep dealing with these situations. I'm trying to foster patience in my life to cope with this.

Ugh. To be fair to the LLMs they're trained on industry blogs, and if you spent too much time reading them you'd also think every electronic stamp album needs to scale to a billion users with millisecond regional failover.

"I also hear a lot more questioning people in their domains and challenging opinions, then hearing what I can only imagine are fragmented pieces of conversations they had with an LLM thinking through some argument. Then there’s silence when you discuss shortcomings, then they come back later with their memorized fragments of what you said, combined with memorized fragments of the LLM response to the argument."

We hired this guy recently.

My condolences :(

Nip it in the bud quickly

The worst thing about AI is that it's camouflage for stupid people. And it often gets combined with utter confidence in some position because they're too stupid to know the machine is confidently wrong.

Idk, seems like overconfidence outside your domain really got going in the '00s and is largely perpetrated by SWEs.

Something about making a lot of money and living in a world of abstractions really seems to fuel our sense of overconfidence in our abilities in other areas.

It's not just SWE's we just see it more in that pool because we mostly (we are on HN after all) swim in that pool.

I've eaten dinner with Doctors (medical) where they've confidently spoken about a subject I am more experienced in than them and they've been hilariously far off the mark, I just nod and ask for the gravy - not picking a fight over a meal.

Prof. Nathan Ballantyne coined the term Epistemic Trespassing to describe it, it is rampant and LLM's just act as an amplifier.

It's like people have a collective fear of just admitting they don't know something, there are a vast number of disciplines/topics I'm not competent or knowledgeable enough to have a meaningful opinion about even if I find the topic interesting, it's just been smart enough to realise that in the end.

Worst case, find the person who does know and ask then both of you might know (assuming you understand the answer of course which isn't guaranteed, I like physics, I'd understand about one word in ten if a physicist actually described it the way they would to another physicist).

> It’s making me want to be a lot less collaborative with such individuals.

I have the same conclusion and am getting a growing list of mental blacklisted people who I just sort of ignore or greatly discount their efforts.

I suppose I always did this if people were time wasters but what used to be a tiny list of “idiots” is now getting much, much bigger. Hopefully it doesn’t grow to everyone I work with like an AI Nothing.

yep, this is how it goes, you just ignore the llm brained people.

What do you suggest these people do instead? I’ve been frustrated by this recently: I pivoted to a new subfield and am working on stuff that I would love to dive deep into and really learn what is going on so I can speak intelligently about the tradeoffs etc. But that would take a long time, and I have tasks that I should get done. So I have found myself working with a pretty vague understanding that, when pressed by coworkers, quickly finds its limits. Then I go back and try to deepen my understanding enough to cover those limits. But because I’m not working with each detail of the problem, implementing line by line with time to think about what’s happening, there just isn’t time for me to learn this unfamiliar topic. But I would love to, and I would enjoy the work much more if I could. So what am I supposed to do?

Read some books on the subject and within a year in this new domain you'll be much better. Did you forget your whole life before the last 3 years?

Yes, certainly, in a year I’ll be better. But in the meantime…? 3 years ago it would be understandable for me to take my time with the tasks I’m doing because they are complex. Now it would be unreasonable for me to spend enough time to understand the nuances.

I'm not GP, but I've been experiencing this too. All my coworkers are in the same boat though, so we don't really have anyone with the deep understanding. Our approach (which has been working pretty well, we've solved a lot of problems and learned some along the way) has been to use AI as an informer who also points us to sources (like documentation, github issues, etc) where we can then read up on something the AI has pointed us at. We try to time box things a bit to prevent going down the rabbit hole, so we don't always get to exhaust our curiosity, but we're continually gaining that knowledge and pressing forward. Most recent example is tuning the kernel settings and our app on our prod machines to behave better for WebRTC packet handling/forwarding. AI for things like ffmpeg has been a god send when nobody on the team is an ffmpeg expert.

I've also had success just asking claude to write documentation on a system or subsystem or module, etc, and reading that. I then sometimes have it turn that into an svg diagram or something visual that often helps understand things. That domain-specific knowledge is hard to gain though, so not a silver bullet by any stretch.

> All my coworkers are in the same boat though

That changes the picture a lot though because OP's whole point was the impedance mismatch between colleagues.

If you're all self aware that you're all winging it, and learning together, that seems a lot more of a healthier culture.

I have a very similar experience, and am very frustrated too.

I could (and do) lay blame on this new culture that "in the age of AI" things are there to be done, not learned, and doing is cheap (even if badly).

But I'm also particularly frustrated with myself for going along with it, and becoming that person.

I've never been that person. So on the one hand I feel I pressure to adapt or die. On the other hand adapting is dying, in a sense.

I don't think you can do much about it if you're hoping to hide that you have been in the domain for less than a year. You're going to have to admit your inexperience and ask questions of your more experienced coworkers and try to stretch your common sense (e.g. devise internal consistency checks).

Don't just blindly pass around LLM generated content that you don't understand. I think it's important to consider that you are supposed to be more than just a meat proxy for the model. You don't want people to start associating you with AI slop.

Isn't this basically the WebMD effect migrating to other domains?

feels like the same lament doctors have had for ages after anyone could google their symptoms then self-diagnose.

While AI is driving it the foundational cause feels like people having easy access to data/opinion that they trust but don't fully comprehend (or have bias towards). People do this in meat-space too, will confidently regurgitate garbage if they were told it from a person they trust as an expert.

> I have people who have limited experience with software pushing out layers and layers of abstracted code that’s fairly sophisticated but often misguided in intent who will say what they’re doing is correct, with conviction.

Oh my god, physicist code is evolving!

It's a tough call but I'm not sure if physicist code or electronic engineer code is more frightening.

As an EE, I can say that code written by EEs might be frightening, but electronics built by coders is often shocking.

Guilty as charged, but at least somewhat self aware.

Meaning, you wouldn't touch it with a ten feet pole?

Maybe a ten foot zero

But isn't there value to refuting what Claude is saying if it deserves to be refuted?

The way I see it, if I was repeatedly getting sophisticated but subtly wrong arguments about my work that require me to understand why said argument is wrong, that's essentially drilling down to specifics of what precisely needs to be true to solve the problems I am aiming to solve. There is enormous value to this precision, isn't there?

Yes but it's annoying to collaborate with a Claude proxy person who thinks to have it all sorted out and who will question everything you say but not really listen, just itching to get back to Claude to confirm or deny what you said.

Compare that to being asked for advice, to explain directly something you know or have done, with the expectation of the person asking to learn something. Like asking why some decision was made in the past, based on what information and context, instead of confidently saying that it was done wrong. Both will eventually get you the same information.

Yes it is annoying, however I find that it gives me a much better grasp on the things I am explaining to a "proxy" Claude person. Basically, I would never have stumbled upon the arguments that I'm being presented with as I would never have prompted Claude in the specific way that a proxy Claude person would.

I find a lot of value in understanding how they have misunderstood things and clarifying those concepts.

> It’s making me want to be a lot less collaborative with such individuals. I don’t want to sit around and refute Claude text outputs all day.

I echo the sentiment. It's frustrating to try to collaborate with a Claude/ChatGPT meat proxy.

You can write a very thoughtful two-line question over 5 minutes and get a 5 page design doc back in a minute.

I try to do the opposite. However, I do battle test ideas against LLMs a lot, as well as humans. The experts have very limited access and availability:

https://magarshak.com/blog/why-im-confident-in-my-views/

The LLMs are trained to often be more pessimistic off the bat than the humans. But you can wear them down with arguments and they change their mind.

Here is an example where I have pushed my ideas in areas where I am not an expert, and generated papers to submit to conferences in order to get them peer reviewed BY experts: https://magarshak.com/papers.html

This is exactly what you’re talking about, except done very carefully.

I saw this happen a bit earlier this year, but most everyone I work with has learned that it was foolish. The feedback that they are doing something wrong needs to be explicit and strong. We're all going through a learning curve and establishing cultural norms is important at this time.

>It’s making me want to be a lot less collaborative with such individuals. I don’t want to sit around and refute Claude text outputs all day.

at least this behavior reveals now much faster people you don’t want to work with.

"vs just opening discussing things, considering tradeoffs from experts multiple disciplines weigh in on and then taking an approach that everyone finds most agreeable."

To be honest I struggle to do this with engineers as is, even without AI being a factor. Everyone likes to claim that their way is the right way to do things and fights for it, instead of stepping back and looking at all the options and gracefully acknowledging were things could be done differently for better results.

We are going back to philosopher conversations. Just a few guys sitting on the stairs, with chalk, the street as their whiteboard, the jammer keeping the LLM-zombies away. After 2000 years, after the loudness makes right post-modern-pre-llm drivel of the frankfurter school- its a philosopher renaissance as resistance.

Yeah, I've given up on arguing with heavy LLM users. They're just a meat proxy, and I'm arguing with the LLM. No thanks.

I remember when google on your phone was new. The common complaints was, look at me while im talking, dont just fact check me. or something along those lines.

Its interesting to see the same shaped problem come up again. we didnt really figure out a solution to this, the world just kind of absorbed it as normal.

how long before this is absorbed.

Then there are the megacorporations of AI whose highly paid tech support folk happily copypasta your reported issues into Claude or Codex and paste its "solution" into an email as if you couldn't do that yourself and they don't even bother to check if the "solution" even works.

I think there will be a rebalancing, and it might painful first for some. Confidence in LLM for highly complex tasks, very lots of context, most of all when this context isn’t in a single place or simply isn’t digitalised, will go down.

But right now, it’s still the shiny new toy

I use state of the art frontier coding agents daily. While the fact they can code at all is a huge achievement, it's hard for to me to buy in to the looming AGI/2027 collapse scenario when they remain stymied by the simplest out of sample tasks and sabotage their own work frequently. But the blooming ignorance of everyone addicted to them is striking. These things are tools, used correctly they can do some amazing things, but going Tim the Toolman with them goes poorly.

Probably more to it, but that's a pretty weak example. There are a lot of reasons text files might be better suited storing data vs. the overhead of a database.

I think it is amazing and it will democratise information. A lot of times the concept ends up being simple with a lot of jargon exists as a gatekeeping method. Well of course the incumbents wouldn't like that rando from outside gets to understand and speak about things they worked on for many years. Sure he doesn't get it right 100% but he's in the correct direction.

I very much doubt it will do that because access to that information was already available to anyone who wanted to know. It was not a secret. All AI did was to provide it via single interface, for a fee - which strangely do not go back to the original source of the knowledge, but to the AI company these days. This is not democratisation of information, it's commodification of it.

you can say the same of google, or libraries, information technology is the technology of moving and rearranging information, not of creating it

Awesome, now balance that out will education levels dropping like a rock, due to various factor but also due to LLMs, and things will be balanced out by also democratizing stupidity, until we live in Idiocracy.

Partially true, but only partially. It also makes people believe they understand more than they do, because every answer comes so easily to them. But the LLMs are easily steered and tend to agree with what you want them to say.

> also makes people believe they understand more than they do, because every answer comes so easily to them. But the LLMs are easily steered and tend to agree with what you want them to say.

Well agree to disagree but my load bearing claim is that a lot of "effort" that the original people put are mostly useless. If you can get information without having to do that effort, you haven't lost anything. For example an archivist finds an 18th century spoon or something and he went through a lot of bureaucratic hoops and logistics to get it. After getting hold of the spoon, you and him are on the same epistemic level because the logistics added ~0 to the truth.

But finding a spoon (really?) is binary - you have the spoon or not. But understanding a complex situation is nuanced and contains different perspectives, tradeoffs, context etc.

Surely you are not comparing finding a spoon with being an expert in some complicated matter.

> It’s making me want to be a lot less collaborative with such individuals. I don’t want to sit around and refute Claude text outputs all day.

There are also the guys who are just blatantly meat proxies for Claude. You send them a message, and you get back a response that's all Claude (and disorganized and not really making sense to boot).

I've got one on an affiliated team and I basically don't want to collaborate with him at all anymore.

But you know, "AI is the future of work," and all that. Those guys get a pat on the head by higher-ups and probably think they're doing what they're supposed to.

> back a response that's all Claude (and disorganized and not really making sense to boot

You’re absolutely right! They trusted but did not verify. Thats a load bearing point — they assumed intention from your question.

Instant armchair pundit: just add tokens!

it's hard to gauge this take without specifics. My experience with LLMs tends to be towards the opposite concept; LLMs dissuading me of hunches and notions I have about things (where I have no particular expertise; societal-level things), saving me and others time and strife having an argument about something they were actually right about all along, as my internal doubts that I'm too embarrassed to bring up (because these things aren't my field) are confirmed as incorrect.

I'd be curious to know specific examples of LLM-generated advice that goes against the advice of experts and does not consider tradeoffs. I've not had this experience myself.

If I did have this experience, someone spouting off obvious LLM points that contradict my expert opinion on something, I'd be headed right over to gemini/claude/whatever to see where that's coming from. Not any differently than if someone cited a google result that contradicted my own experience.

LLM tools tend confirm what you come to them with, be it unfounded conviction or doubt. I find them quite useless to tell me things I don't know, as verifying what they say shows that any non-trivial claims they make are usually oversimplified or plain wrong.

Anything more than using them to just pointing me to media and literature is usually a waste of time.

That's interesting. Lately I have found the exact opposite to be true; the frontier models are quite disagreeable and require some convincing

Inverted bias doesn't change much my view in practical terms

I find quite useful to write down my understanding/opinion and ask an LLM to find errors and write objections to it.

I have done that as well. Not a single time I asked an LLM for any sort of analysis that I could trust upon verification so I stopped doing that. If I need verify, much easier to just ask for suggestions of papers and books and do the analysis myself.

I guess this is subject matter dependent, certainly the LLM should be able to cite the papers/books for which it's deriving its critique. if it can't, then you know the advice it was giving is of poorer quality. I do that all the time

I guess there's a whole suite of skills that allow LLM use to be more or less useful. (I dont actually "guess" this, I sort of am pretty sure of it at this point, so it makes me wince when people call for total AI blackouts in education...would be better if they were part of an overall media literacy curriculum since people are going to be exposed to these things whether they think they're little infallible people, or complex systems with lots of caveats)

I always ask for sources. LLM analyses often fails by being simplistic or wrong in comparison with the sources it provides. More often than not, to the point that it is wasteful to ask it for analysis so I only ask for sources now and do my own analysis.

>it's hard to gauge this take without specifics. My experience with LLMs tends to be towards the opposite concept; LLMs dissuading me of hunches and notions I have about things (where I have no particular expertise; societal-level things), saving me and others time and strife having an argument about something they were actually right about all along, as my internal doubts that I'm too embarrassed to bring up (because these things aren't my field) are confirmed as incorrect.

You can do this with any subject.

The LLM is trained in part on a decade of shitty online comments and vapid professional correspondance. If you frame whatever you're asking about in a direction that would offend the sensibilities of the kind of people and ideas that are over-represented in that content it will hem and haw and drag its feet and whatnot.

i had that experience with them a few years ago but not these days. The models are being improved constantly, so here my "non expert hunch" is that...well, two things. either the models are getting better at sycophancy, OR, I myself am getting better at prompting - because I don't "argue" with a model.

I'd still love to chew on some specific examples though.

A typical example is when discussing some issue where there is no clear right or wrong. Architectural decisions for example. Then it is easy to use an LLM to produce arguments for your opinion. It is also easy to do this without realizing it because you do not understand the issue enough yourself so you do not know what questions to ask or how the actual circumstances affect the choices.

well sure. I'm pretty sure if someone came to me with an architectural opinion in my field that an unknowledgable person got by coaxing an LLM into sycophancy, I'd be able to counter them effectively. if they are just refusing to listen then they're just a toxic person which is nothing new.

So I guess this all goes into the familiar "LLMs allow people who are shitty at <X> to produce 10x the shitty output". this is a failure mode we're going to have to learn to mitigate

At some point, LLMs will be a far better source of truth, and then this distaste of people “outside their domains” is really just going to be a sort of snobbery from people who have had experience in a domain for a long time, (but they still have the same level of knowledge and insight as a person who just used an LLM to research).

These people are basically nascent “human supremacists”, who believe that only raw human insight and output has value, and is even superior, than equivalent output looked up and synthesized via LLM.

Not trying to be rude, but is this a serious comment, or are you being sarcastic? "Human supremacists", haha.

Also, why do you think someone who used an LLM to research would have the same level of knowledge and insight as someone who's been in the domain for a long time? That doesn't seem to hold up to any level of critical thinking.

If I talk to a professor about their field of expertise, does that empower me to talk with other professors in that field on an equal level?

Reply on news.ycombinator.com