Comment by saaaaaam | Hacker News Reader

Comment by saaaaaam | Hacker News Reader

Parent Full thread

>it feels like reading an impression of a literature book by a high school English class’s most overconfident student who’s only ever read LinkedIn-speak.

Claude is very much the “stupid person’s idea of an intelligent person”[0] which, I suspect, is why it is so popular.

It certainly explains why half the internet is huge chunks of Claude-authored gibberish copied and pasted and published. If people didn’t think it sounded clever they wouldn’t put their name behind its ramblings - but very few of them seem to realise that a lot of people see straight through the bullshit and know instantly that they didn’t write it themselves.

But equally, a lot of people can’t tell, and read whatever it is and think “that person must be clever!” So you have people incapable of coherently expressing thoughts who are using Claude to write on their behalf, with the result that the people they want to think of them as clever think less of them and the people who can’t distinguish clever from AI slop think they are clever.

And the people who can’t tell don’t care, and the people copying and pasting Claude slop seemingly don’t care either.

And then I remember that more than half of the US populations reads at Grade 6 or lower[1], and nearly 1 in 5 people in England is functionally illiterate[2], and I simultaneously despair of - and am thankful for - the bubble of literacy I inhabit.

[0] https://quoteinvestigator.com/2018/01/05/clever/ [1] https://www.thenationalliteracyinstitute.com/2024-2025-liter... [2] https://literacytrust.org.uk/parents-and-families/adult-lite...

Reminds me of current day politics. Lots of public statements which are obviously false, and you would think the politician knows they are false, but utter them anyway because they also know lot of their supporters buy what they are saying anyway.

Now politicians also know something about their supporters so they will adapt their statements to what they think they can get away with it. But, I wonder if this leads to a two-party-system where one party attracts stupid followers and another attracts the smarter ones?

In terms of AI, we might see LLMs specialized to attract more stupid audience and others meant to attract those who appreciate correctness and facts.

>Lots of public statements which are obviously false, and you would think the politician knows they are false, but utter them anyway because they also know lot of their supporters buy what they are saying anyway.

Yes, we truly live in a “post truth” era.

> But, I wonder if this leads to a two-party-system where one party attracts stupid followers and another attracts the smarter ones?

I think all political parties aim at the lowest common denominator.

The alt-right party in Hungary - Fidesz - at least since 2006, and the Republican Party since at least Trump, definitely targeted people who are fine with a broken belief systems, in other words, they are fine with continuous cognitive dissonance.

In case of Hungary, in the past 20 years, they moved contradictions inside an article, to inside a paragraph to inside a single word. Trump is on the single word/expression level since 2016 for sure: “alternative facts”.

there's a wide array of assessments when it comes to reading comprehension. the one you refer to, the GRA, sets the 'sixth grade level' as whether or not a reader understands the author's main points, is able to answer conceptual questions related to the text, and then apply those to relevant situations. beyond this level is the ability to essentially be skeptical of a text and to know how to critically analyze it. so if your comprehension level stops before this you get 'big words in complex sentence structure sounds smart and right so it is smart and right' even if the reasoning and process is poor

it makes me think about how people engage with movies and television - as passive, plot-and-character driven consumption (eg I hope Walter White survives) with no critical analysis of how and why the writers added ABC thematic element (eg Walter White as a motif of a toxically masculine narcissist with specialized knowledge as a larger critique how mass media tends to valorize their male leads in the same vein as many other prestige shows at the time like Mad Men), and the larger, downstream sociocultural impact that piece of media has on how people see the world (eg people who now have the Heisenberg tattoo, unironically)

there's been some musings on why this the case like Hofstadter's Anti-Intellectualism in American Life - the valorization of obedience and trust in hierarchy and the state are net wins if you're an institution that seeks to increase it's power, whether religious or governmental. I was talking about this with a few friends the other day and it's a dismal future reality where not only did we make anti-intellectualism normalized and politically legitimate in the USA (eg Fox News, clickbait articles, and all the other forms of yellow journalism that have emerged), we now have tools by which individuals can even further remove themselves from having to critically engage with thoughts, feelings. I heard a story about how someone scanned a group activity at a baby shower into ChatGPT and had it answer for them instead of, well, socially interacting with the other guests and forming a memory of the moment with their friends

the counterargument to that might be that Claude/ChatGPT/etc have more epistemic rigor than your average American (sure) but the sycophancy of modern day LLMs is an actual danger that enables more harm than good. it does seem as if Claude is the only one interested in guarding against some small amount of it (though to the detriment of people just trying to get work done. as an aside, I get the feeling Mythos was intended to be the bespoke enterprise solution without the guardrails but the Anthropic marketing department or some power-hungry department lead made it about how dangerous/effective it was from a security perspective which threw a wrench in things). but then I think about people like my parents asking ChatGPT which specific house to buy in their retirement only to later find out the house was sold weeks ago, or just in bad condition, or in a neighborhood where the housing value has already reached equilibrium, it makes me think about how it's not enough and the future is bleak

I'll also say that I think Claude sounds the way that it does because it, like many other LLMs, are RLHF trained largely by lowly paid gig-workers, many of them ESL speakers. if their trainers were, for example, dedicated and highly trained academics, scientists, and other researchers, you'd likely see a lot more concise and more importantly skeptical reasoning and responses. but that won't happen in our current reality of capitalist-driven development so we get encoded solutions like MoE that still largely depend on the messy, imprecise RLHF training at baseline

in the right hands, I do think AI is a wonderful tool. one of the first things I did with it was to create a research skill that reviews white papers from the lens of someone who knows how to read/interpret research methodology, is aware of things like p-hacking, and deterministically assigns weight according to the hierarchy of evidence. even still, I'll still read the studies because there's so often nuance that's missed if the sub-agent read only a search snippet but that takes effort, time, and the practiced knowledge of critical analysis to even want to do it

I’m inherently skeptical of big walls of text like this these days.

(So here’s a big wall of text of my own!)

However, a lot of what is written here makes sense.

And particularly “if your comprehension level stops [here] you get 'big words in complex sentence structure sounds smart and right so it is smart and right' even if the reasoning and process is poor”

This is exactly the problem.

And another point you make:

> but the sycophancy of modern day LLMs is an actual danger that enables more harm than good

I don’t think it is necessarily the sycophancy that is the biggest problem (though that is definitely a problem) but rather the combination of authoritative sounding text plus “complete answers” which sound wholly believable but are deeply flawed unless you have domain expertise.

I moderate a forum that deals with people who face a relatively common but somewhat complex (and nuanced) set of legal problems.

The purpose of the forum is peer support, shared experience (“lived experience”) and community.

It’s not legal advice, though moderators will sometimes step in to highlight relevant legal resources (e.g. case law/precedent or primary legislation/instruments).

Prior to AI infecting the forum someone would post their problem, people would respond with their often incomplete or poorly communicated thoughts, the OP would ask more questions - or argue - and a dialogue would occur. That created a community and people would post updates and ask more questions and find common shared experience. Many of them became correspondents with each other and some became actual friends.

In the past 12-18 months the discourse has changed from “here is my personal experience and here is what I did” to “here’s a bunch of stuff an AI says and I’m pretending it is me giving advice”.

Almost without exception the person who has started the thread will react positively to the AI generated content, even when it is egregiously incorrect - but won’t ask questions.

More problematically, these AI posters will often argue specific incontestable points of law “because I asked ChatGPT/Grok/Claude and it says this” and ChatGPT clearly cannot be wrong. And the border of precedence seems to be ChatGPT, Grok and then Claude some way behind.

I’m slowly seeing a pushback from people as “normies” begin to spot AI. But it’s ruined a community because the advice sounds so authoritative and complete that people won’t argue or ask questions.

As a result we have banned AI generated posts and remove repeat infringers.

That’s significantly reduced the volume of posting (below what it was pre-AI) but has significantly increased the value the members are getting.

I do appreciate the thoughtful response to a really long wall of text lol. and yes, I agree - I think that'll be the lesson that society is going to take probably far too long to learn, to not see everything as a nail that AI can hammer at. a lot of tech companies are in essentially a 'fuck around and find out' phase with AI taking over code review, testing, etc. combined with the expectation of shipping 3X the amount of code, we've enshittified the entire SDLC. and so we have near-daily incidents, data leaks, etc, something that I was able to measure and report on at my old place of work to, well, no avail

it's the old tortoise vs hare parable, I think. go fast, make a bunch of mistakes, get too arrogant, and you lose out. your forum might be slightly lower engagement now while people are caught up in the latest fad but your rules are proactive for a future where average people hopefully realize that you can't trust an LLM that has zero context, no real harness and determinstic tests to speak of, and a propensity towards probabilistic rabbit holes that result in hallucinations. at least that's the kind of space I'd look for now and largely why I've given up on a lot of other forums

That’s quite encouraging to hear, because it aligns with what we are trying to do.

Which is basically weather the AI storm and come out the other side with something that is essentially purely human.

And then we might - where appropriate - use AI to help surface or explain relevant external content. “Idiots guides” but human reviewed.

You are fighting a good fight! Props.

Good fight / entirely thankless fight maybe.

I can’t help feeling like this is the last gasp of the old internet. Those tiny corners of expertise can so easily be eliminated through a few months of “AI! SHINY!” and there’s no coming back. I’ve seen a couple of other communities decimated by AI. The participants start posting AI slop and then remarkably quickly everyone else just stops commenting. It’s awful.

The problem is one of expertise, sometimes general, sometimes specific.

If you don't know better, you don't know better to question what the AI says.

I've seen this in the work environment with a coworker who insisted that I implement my side of the control system using the control law ChatGPT recommended instead of building off the empirically tuned control law. I eventually sectioned off a part of the codebase for him to work on independently.

Needless to say he didn't get a whole lot farther.

Later characterization of the entire system end-to-end showed the existing system was already close to the theoretical limits and ChatGPT's tearup would have bought us precisely nothing except for more work to tune the new control loop.

And I see this in everything that requires expertise. You need to know enough to know when it's bullshitting you, and it's hard to be enough of an expert in everything to tell when it's bullshitting you for something you aren't enough of an expert in.

I see it as a problem of context which maps to my theoretical understanding of LLMs. models trained on large data sets will probabilistically veer towards the median in all aspects - reasoning, assumptions, environments, etc. specialized context about your specific codebase's solutions don't exist unless you add them in, either in the prompt, as a skill or rule, or more generally in the harness via memories, tests, etc (though ideally a combination of all of the above). without that the LLM will suggest the median solution for the median codebase according to some ephemeral, unqualifiably trained understanding of best practices

it makes me wonder if the solution that businesses/users need to implement is just the same solution to everything since the beginning of time ie standardization. skills/harnesses/agents.md/etc maintained by codeowning teams that must be invoked for AI-assisted code changes on ABC part of the codebase, these existing as replacement for the bevy of other documentation required for the days of hand-written code. a company-wide orchestration skill knows how to search and pull down the relevant .mds, cleans it as cruft at the end of a session, every merge with a short changelog saved to a corpus somewhere with a TOC + appendix that an LLM can navigate to and read for context, major changes in the logic documented in the working skill doc, all of it generally automated but requiring HITL vetting

this wouldn't fully solve the problem of subject matter expertise but it seems like it would remove a lot of the friction for new employees and other teams with dependencies on your work or with whom you have dependencies

> it makes me think about how people engage with movies and television - as passive, plot-and-character driven consumption (eg I hope Walter White survives) with no critical analysis of how and why the writers added ABC thematic element (eg Walter White as a motif of a toxically masculine narcissist with specialized knowledge as a larger critique how mass media tends to valorize their male leads in the same vein as many other prestige shows at the time like Mad Men), and the larger, downstream sociocultural impact that piece of media has on how people see the world (eg people who now have the Heisenberg tattoo, unironically)

That's not the only smart-person way to read that show. And even if a character has flaws, or even if it's an outright villain, people can still like the character. If I tattoo Scar on me from the Lion King, does it mean I didn't understand that he's not a positive character? I can still think he's cool. I'm sure people also put Darth Vader tattoos on them. Also you're using phrases of political ideology that one doesn't have to subscribe to in order to enjoy the series.

sure, that's very much the 'just let people like things' argument where literal white supremacists can enjoy Rage Against the Machine in spite of the music literally being in total opposition of their ideology

everyone's free to enjoy media however they want, with whatever level of interpretation they like. I provided the BB references as short examples, they aren't meant to represent the definitive diagetic experience of the show. if you have a different view, great. if it was triggering for you to hear 'toxically masculine', also fine but... might be something worth self-examination on given that it's very much also a clinical and academic term [0] as much as it is one steeped in the artificially manufactured culture wars by people who don't want to change their anti-social behaviors

I would also say that understanding Scar as a villain is the sixth grade reading level understanding of the character. and you're free to stay at that level of understanding. someone who wants to engage more critically might map the character to Claudius, comparing and contrasting how they're characterized given the context of the audience for Disney and Shakespeare, and appreciate the character that way, as a standardized trope utilized throughout all other forms of media. they may even get a tattoo of Scar, symbolizing their dive into the analysis

my point is not that people should or shouldn't engage critically. it's that this practice of critical engagement, of being skeptical and analytical provides you with the skills to not be a total sucker who falls for the latest manufactured fad that someone with a strong theoretical understanding of semiotics and social capital created (ie most modern marketers). the pertinent example being how people engage with AI - seeing it either as a specialized tool with a set of flaws that need to be accounted for and checked against or as just some kind of authoritative voice because it sounds smart and so-called smart people like Elon are terrified of it and AGI. which, again, if you prefer the latter engagement all power to you but the chances of you taking some really bad advice forward is not negligible

[0] https://www.wi.edu/news-Shifting-the-Conversation-From-Toxic...

You're the live stereotype of that middle-of-bell-curve meme thing. Using thesaurus words is no longer impressive. And that academic world you refer to is a navel gazing self referential nothingburger.

It's like a jumbled up string. You pull on the two ends and it turns out to be just a loop, it resolves to a big null.

you can have whatever opinion you want of me, friend, but for your own good I hope you develop better ways of processing your feelings so that you're not so reactive when someone mildly disagrees with you

There's less consensus around what you're talking about that you imagine, which was my initial point. Smart people aren't just this academic bubble that you think it is. Not seeing such views made you develop this association of "smart" has to mean "interprets cultural products according to my political ideology". My reactive reaction was to point out that this isn't so.

I didn't make any claim about being smart or not aside from brief mockery of people who care about the word and things like IQ tests

I'd personally trust a rigorous, well-evinced, and methodical dissection of a topic by someone with completely average intelligence over a lazy broad generalization by someone who scored high on a paper test that people (wrongly) assume maps onto the ability to assess and analyze the world [0]. which isn't to say there isn't any clinical significance from something like the Stanford-Binet, it's just that there's a wide gulf between the lay person's understanding of 'high IQ' and clinical applications [1]. and, accordingly, I am someone who did score high enough on a psychologist administered IQ test when I was but a wee lad that universities were sending my family letters requesting that I be a part of their high-IQ child studies but you're convinced I'm middle of the curve so how valid is it, even?

the thing I value from academia is predominantly peer review and constant critique by people whose entire job is to perform epistemically rigorous analysis. this is something that's very uncommon when it comes to culture warriors on social media. and you must be very far removed from any kind of academia if you think they all universally agree - put a post-structuralist in a room with someone who still favors deconstructionism and you won't hear the end of it

funnily enough, clinical psychology is the research I linked to about toxic masculinity that you broadly generalized as worthless ivory tower ideology. so it wouldn't make sense then to value something like 'smarts' or 'IQ' since those, too, are concepts that emerged from the very same ivory towers, in those exact departments rife with ideological bias

[0] https://som.yale.edu/news/2009/11/why-high-iq-doesnt-mean-yo...

[1] https://opened.cuny.edu/courseware/lesson/48/student/?sectio...

I've participated in peer review from both sides, and it's much less impressive than people imagine from the outside.

I wish I could /. friend people on hacker news; the enjoyment I felt from your posts in this thread almost knocked me over in delight :)

Yes, I threw in a smiley just to be a dick. You even quote LeGuin, thank you for being you.

> (eg Walter White as a motif of a toxically masculine narcissist with specialized knowledge as a larger critique how mass media tends to valorize their male leads

Very confused about this: how would you write the character differently?

The whole point of leads, whether male or female, is to be larger than life. All their characteristics, including growth or change, are exaggerations of reality.

This specific character is supposed to be more villainous than reality, and if you take that away then you're simply going to have a show no one will watch.

> I'll also say that I think Claude sounds the way that it does because it, like many other LLMs, are RLHF trained largely by lowly paid gig-workers, many of them ESL speakers. if their trainers were, for example, dedicated and highly trained academics, scientists, and other researchers, you'd likely see a lot more concise and more importantly skeptical reasoning and responses. but that won't happen in our current reality of capitalist-driven development so we get encoded solutions like MoE that still largely depend on the messy, imprecise RLHF training at baseline

No really, that's not particularly accurate, they use so much gig work because no-one else wants to work for them not because they would be unwilling to pay a little extra, or only want the absolute cheapest labor they can get on the planet.

They want senior white collar professionals and scientists and researchers especially since these companies already on some level believe their models are as good as any senior employee in any field (it's probably the models generating text saying that, but that's besides the point). But who's going to work on contract for a company that wants to automate them out of a job? Realistically no-one unless they get some shares in the thing that will destroy their future earnings potential and ability to control their own destiny if it works out.

But they can find enough educated white collar professionals on unemployment or in unstable academic employment that will take an extra job on even if it's only 50 $/h or 70 $/h and compromise on any solitary they might have but the work output you get from that is only going to be as good as what you ask for, if they had better respect for the professions they want to automate, it would be better.

Like is that an acceptable wage in the US for difficult skilled work, not particularly but it's not rock bottom exactly, and it's not bad for other English speaking countries, working conditions and stated mission are more of an issue than being cheap.

Training pipeline on a modern LLM is also going to be quite indirect during the long tail of post training, and heavy on automated RL, the human feedback might end up getting used in the form of automated grading guidelines like what you did for research, with the same issues as that, compounded by the input being LLM generated and models being biased towards model output by default. It's more of a feedback on the loop rather than in the loop.

so why don't people want to work for them? they don't get paid enough? what if they were paid more? what if they were FTE with all the benefits? what if AI projects were nationalized and trainers were funded by grants? what if we increased the NIH budget, made peer review and journals far less exploitative of researcher's time, and got rid of academic middle management, focusing mostly on paying more towards actual research and academia?

definitely a utopian vision that is not likely to happen in our current reality but I like to imagine better worlds that are possible. as LeGuin once said, "We live in capitalism, its power seems inescapable — but then, so did the divine right of kings. Any human power can be resisted and changed by human beings."

I suppose they don't because they want to be respected, and the LLM companies that got big think they are past needing to have respect for the people who's jobs and ways of living they want to automate. We saw that in how the leading AI company for this kinda of pure reasoning and advanced logic (OpenAI by a mile) treated reporting when they made those 10 recent discoveries in mathematics, they didn't bother crediting the work their models generated a proof by building upon, the headline was just all about their models and how great they are.

You could make it a state capitalist society with theoretical public ownership and this disrespect towards people won't go away. Like LeGuin - a famous non-anarchist - pointed out in the dispossessed simply liberating the social relations and saying you have no rulers is not enough to build a society without unjustified power, and what power even is or isn't justified will rarely be an easy matter.

I don't know you can build this technology otherwise that is without coercion, with consent, the people that want it have convinced themselves it's too important to try to justify to anyone else why they need it, I think you could eventually do it. But for us at the very start of the development of what became this systems we went into it with a handful of admittedly brilliant people so convinced they have a right to reshape the world they didn't care if anyone else agreed to this, they would have been bad anarresti. If you wanted to build it in a fair way it might be another generation or more before the project would be complete, you would have to first convince people this is something that should be built in the first place, not just that you can build it in a safe way.

> You could make it a state capitalist society with theoretical public ownership and this disrespect towards people won't go away

this is essentially the PRC in the 21st century post-Deng* - there's probably a cultural difference at play here too given how embedded the CCP is within academic and business institutions, how kinship tends to be extended on a filial level leading to larger networks. mobilizing a large group of subject-matter experts for post-training annotation (eg - https://ojs.aaai.org/index.php/AAAI/article/view/29907) seems to be fairly simple as an ask but, like you said, it doesn't matter ultimately with their largest firms like DeepSeek utilizing automated RL to skip that whole step and this type of training isn't the norm

I sometimes wonder if the problem with the PRC's sinking back to exploitative labor standards would happen in a vacuum. if you weren't surrounded by adversarial nation states with leaders looking to squeeze every advantage, would you, yourself, resort to the race-to-the-bottom of profit sharing?

a much better SF visionary than I would be able to write a story about a very slow growing AI project* trained for specialized use, and how this was the norm and everyone was happy because it was happening at a reasonable pace relative to hardware capabilities (presumably also reduced to avoid all of the slave labor inside of the rare earth trade). I think it's the capitalism side of things that says 'we should be everything at the fastest speed for everyone' that we get things like Claude and ChatGPT which necessitate outsized, disproportionate resources that doesn't allow hardware efficiencies to catch up and mitigate the worst of the externalized costs

*realistically it's maybe more accurate to say post-Jiang given the extent of opening up Chinese labor markets but Deng's trajectory steered the way

*Ted Chiang's Lifecycle of Software Objects does this a bit but it's more of a social commentary on the capitalist abandonment of the functional for the shiny than it is a sharp political critique of the pace of modernization and its effects on people and the environment

Reply on news.ycombinator.com