Bad Actors Are Grooming LLMs to Produce Falsehoods
americansunlight.substack.com
americansunlight.substack.com
What a glorious future we've built.
The only thing I'm seeing offline are people who already think AI is trash, untrustworthy, and harmful, while also occasionally being convenient when the stakes are extremely low (random search results mostly) or as a fun toy ("Look I'm a ghibli character!")
I don't think it'll take long for the masses to sour to AI and the more aggressively it's pushed on them by companies, or the more it negatively impacts their life when someone they depend on and should know better uses it and it screws up the quicker that'll happen.
The number of them who just blindly put shit into an AI prompt is incredible. I don't know if they were better engineers before LLMs? But I just watch them blindly pass flags that don't exist to CLIs and then throw their hands up. I can't imagine it's faster than a (non-LLM) Google search or using the -h flag, but they just turn their brains off.
An underrated concern (IMO) is the impact of COVID on cognition. I think a lot of people who got sick have gotten more tired and find this kind of work more challenging than they used to. Maybe they have a harder time "getting in the zone".
Personally, I still struggle with Long COVID symptoms. This includes brain fog and difficulty focusing. Before the pandemic I would say I was in the top 10% of engineers for my narrow slice of expertise - always getting exceptional perf reviews, never had trouble moving roles and picking up new technologies. Nowadays I find it much harder to get started in the morning, and I have to take more breaks during the day to reset my focus. At 5PM I'm exhausted and I can't keep pushing solving a problem into the evening.
I can see how the same kind of cognitive fatigue would make LLM "assistance" appealing, even if it's wrong, because it's so much less work.
I've recently had tons of memory and brain fog. I thought it was related to stress, and it's severe enough that I'm on medical leave from work right now
My memory is absolutely terrible
Do you know if it is possible to test or verify if it's COVID related?
Mostly you talk to your doctor and read stuff and advocate for more testing to figure out why you're not able to function like before. Even if it's not "Long COVID" it definitely sounds like something is causing these problems and you should get it looked at.
Car accidents came down from the Covid uptick but only slightly. Aviation... ugh. And there is some evidence it accelerates Altzheimer's and other dementias. We are so screwed.
(I don't mean to imply that parent doesn't know this, it just seems worth saying explicitly)
How do they know?
The error here was to click on a phishing email.
But something I have seen myself is Tim Cook talking about a crypto coin right after the 2024 Apple keynote, on a YT channel that showed the Apple logo. It took me a bit to realize and reassure myself that it was a scam. Even though it was a video of the shoulders up.
The bigger issue we face isn’t the outright fraud and scamming, it’s that our ability to make out fakes easily is weakened - the Liar’s dividend.
It’s by default a shot in the arm for bullshit and lies.
On some days I wonder if the inability to sort between lies, misinformation, initial ideas, fair debate, argument, theory and fact at scale - is the great filter.
The work at the Levin Lab ( https://drmichaellevin.org/ ) is making great progress in the basic science that supports this. They can make two-headed planaria, regenerate frog limbs, cure cancer in tadpoles; all via bioelectric communication with cellular networks. No gene editing.
Levin believes this stuff will be very available to humans within the next 10 years, and has talked about how widespread body-modding is something we're going to have to wrestle with societally. He is of course very close to the work, but his cautious nature and the lab's astounding results give that 10-year prediction some weight. From his blog:
> We were all born into physical and mental limitations that were set at arbitrary levels by chance and genetics. Even those who have “perfect” standard human health and capabilities are limited by anatomical decisions that were not made with anyone’s well-being or fulfillment in mind. I consider it to be a core right of sentient beings to (if they wish) move beyond the involuntary vagaries of their birth and alter their form and function in whatever way suits their personal goals and potential.- Copied from https://thoughtforms.life/faqs-from-my-academic-work/
I often like to point out--satisfying a contrarian streak--that our original human equipment is literally the most mind-bogglingly complicated nanotechnology beyond our understanding, packed with dozens of incredible features we cannot imitate with circuits or chrome.
So as much as I like the aesthetics of cyberpunk metal arms, keeping our OEM parts is better. If we need metal bodies at a construction site, let them be remote-controlled bodies that stay there for the next shift to use.
I see no reason to expect that superwood is incompatible with in place biological synthesis from scratch. That's entirely organic and there's no question that its material properties far exceed those of our OEM specifications.
For example, dental enamel is a really neat crystalline material, and a biological process makes it before withdrawing to use it as a shield.
(Now I want to change the Blade Runner reference to something with Harry Dean Stanton in it just for consistency)
The rules and standards we take for granted were built with blood, for fraud? It's built on the path of lost livelihoods and manipulated gold intent.
[1] https://www.cbsnews.com/amp/news/ai-work-kenya-exploitation-...
[2] https://www.theguardian.com/technology/2023/aug/02/ai-chatbo...
Curious what you think a popping bubble looks like?
A stock market crash and recession, where innocent bystanders lose their retirements? Or only AI speculators taking the brunt of the losses?
Will Google, Meta, etc stop investing in AI because nobody uses it post-crash? Or will it be just as prevalent (or more) than today but with profits concentrated in the winning/surviving companies?
I do think that the industry and this technology will survive, and we'll enjoy many good applications of it, but it will take a few more years of hype and grifting to get there.
Unless, of course, I'm entirely wrong and their predicted AI 2027 timeline[1] comes to pass, and we have ASI by the end of the decade, in which case the world will be much different. But I'm firmly in the skeptical camp about this, as it seems like another product of the hype machine.
[1]: I just took a closer look at ai-2027.com and here's their prediction for 2029 in the conservative scenario:
> Robots become commonplace. But also fusion power, quantum computers, and cures for many diseases. Peter Thiel finally gets his flying car. Cities become clean and safe. Even in developing countries, poverty becomes a thing of the past, thanks to UBI and foreign aid.
Yeah, these people are full of shit.
Makes sense, but if the negative effect of the bubble popping is largely limited to AI startups and speculators, while the rest of us keep enjoying the benefits of it, then I don't see why the average person should be too concerned about a bubble.
In 2000, cab drivers were recommending tech stocks. I don't see this kindof thing happening today.
> Yeah, these people are full of shit.
I think it's fair to keep LLMs and AGI seperate when we're talking about "AI". LLMs can make a huge impact even if AGI never happens. We're already seeing now it imo.
AI 2027 says:
- Early 2026: Coding Automation
- Late 2026: AI Takes Some Jobs
Nontechnical acquaintances with little to no financial background have been (rather cluelessly) debating nvidia versus other ML hardware related stocks. I'd say we're in exactly the same territory.
The other things on that list seem fairly reasonable (if uncertain). Those last two not only depend on wide reaching political transformations in a specific direction but even then fail to account for lag time in the real world. If you started moving in the right direction in (say) 2027 it would presumably take many years to get there.
It's a weird mix of "already happening", "well yeah, obviously", and "clearly full of shit".
Nah. Thinking that poverty will be significantly reduced, let alone eliminated, in 4 years is simply delusional. Primarily because a reduction of poverty won't happen because of AI, but in spite of it. All AI does is concentrate wealth among the wealthy, and increase inequality. This idea that wealth will trickle down is a fantasy that has been sold by those in power for decades.
And UBI? That's another pipe dream. There have been very limited pilots around the world, but no indication that it's something governments are willing to adopt globally. Let alone those where "socialism" is a boogeyman.
The entire document is full of similar claims that AI will magically solve all our problems. Nevermind the fact that they aggrandize the capabilities of the technology, and think exponential growth is guaranteed. Not only are the timelines wrong, the predictions themselves have no basis in reality. It's pure propaganda produced by tech bros who can't see the world outside of their bubble.
However the other things (the ones I didn't quote) seem quite reasonable on the whole. Robots are well on their way to becoming commonplace already. Quantum computers exist, although it remains to be seen how far and how fast they scale in practice. Fusion power continues to make incremental gains, which machine learning techniques have noticeably accelerated. Cures for many diseases easily checks out - ML has been broadly applied to protein structure prediction with great success for a while now. Helicopters obviously already exist, but quite a few autonomous electric flying cars are in the works and appear likely to be viable ... at least eventually.
The dotcom bubble popped, but the general consensus didn't become negative.
When the dot-com bubble burst in 2000, and after the video game crash in 1983, most of the companies within the bubble folded, and those that didn't took a large hit and barely managed to survive. If the technology has genuine use cases then the market can recover, but it takes a while to earn back the trust from consumers, and the products after the crash are much more practical and are marketed more fairly.
So I do think that machine learning has many potentially revolutionary applications, but we're currently still high around the Peak of Inflated Expectations. After the bubble pops, the Plateau of Productivity will showcase the applications with actual value and benefit to humanity. I just hope we get there sooner rather than later.
Though cryptocurrencies are slightly different because of how they work. They're inherently decentralized, so even though there have been many smaller bubble pops along the way (Mt. Gox, FTX, NFTs, every shitcoin rug pull, etc.), inevitably more will appear with different promises, attracting others interested in potential riches.
I don't think the technology as a whole will ever burst, particularly because I do think there are valid and useful applications of it. Bitcoin in particular is here to stay. It will just keep attracting grifters and victims, just like any other mainstream technology.
It’s here to stay not because it solves a legitimate problem or makes people’s lives better, but because like cancer, there is no cure. Bitcoin and other crypto are for crime, mostly. It’s not useable as actual money given volatility and other properties.
Grandmothers having their life savings stolen by scammers to the tune of 10s of billions annually, that is the primary use case for bitcoin. That and churning out a handful of SBF style gamer turned politically connected billionaires. Nakamoto was smart enough to remain anonymous, lest history remember his name as the person responsible.
Most of people do not lose trust in system as long as it confirms their biases (which they could've created in the first place).
I’m still stunned to wander into threads like this where all the same talking points of AI being “pushed” on people are parroted. Marcus et al can keep screaming into their echo chamber and it won’t change a thing.
[1] https://www.bondcap.com/report/pdf/Trends_Artificial_Intelli...
Where else would AI haters find an echo chamber that proves their point?
You have to know the tools limits and usecases.
Do you genuinely think it’s worse that someone makes a decision, whether good or bad, after consulting with GPT versus making it in solitude? I spoke with a handyman the other day who unprompted told me he was building a side-business and found GPT a great aid — of course they might make some terrible decisions together, but it’s unimaginable to me that increasing agency isn’t a good thing. The interesting question at this stage isn’t just about “elder parents having nice conversations”, but about computers actually becoming useful for the general population through an intuitive natural language interface. I think that’s a pretty sober assessment of where we’re at today not hyperbole. Even as an experienced engineer and researcher myself, LLMs continue to transform how I interact with computers.
Depending on the decision yes. An LLM might confidently hallucinate incorrect information and misinform, which is worse than simply not knowing.
This “80-20” framing, moreover, implies we’re just trying to asymptotically optimize a classification model or some information retrieval system… If you’ve worked with LLMs daily on hard problems (non-trivial programming and scholarly research, for example), the progress over even just the last year is phenomenal — and even with the presently existing models I find most problems arise from failures of context management and the integration of LLMs with IR systems.
1. AI is a genuine threat to lots of white-collar jobs, and people instinctively deny this reality. See that very few articles here are "I found a nice use case for AI", most of them are "I found a use case where AI doesn't work (yet)". Does it sound like tech enthusiasts? Or rather people terrified of tech?
2. Current AI is advanced enough to have us ask deeper questions about consciousness and intelligence. Some answers might be very uncomfortable and threaten the social contract, hence the denial.
Off-topic, but I couldn’t find your contact info and just saw your now closed polyglot submission from last year. Look into technical sales/solution architecture roles at high growth US startups expanding into the EU. Often these companies hire one or two non-technical native speakers per EU country/region, but only have a handful of SAs from a hub office so language skills are of much more use. Given your interest in the topic, check out OpenAI and Anthropic in particular.
[1] https://openai.com/careers/solutions-architect-public-sector... for example - listed salary is 2x your current in the US, not sure what the salary is like in the EU.
It was a whole new world that may have changed my life forever. ChatGPT is a shitty Google replacement in comparison, and it's a bad alternative due to being censored in its main instructions.
LLMs in their current form have existed since what, 2021? That's 4 years already. They have hundreds of millions of active users. The only improvements we've seen so far were very much iterative ones — more of the same. Larger contexts, thinking tokens, multimodality, all that stuff. But the core concept is still the same, a very computationally expensive, very large neural network that predicts the next token of a text given a sequence of tokens. How much more time do we have to give this technology before we could judge it?
See, AI systems, all of them, not just LLMs, are fundamentally bound by their training dataset. That's fine for data classification tasks, and AI does excel at that, I'm not denying it. But creative work like writing software or articles is unique. Don't know about you, but most of the things I do are something no one has ever done before, so they by definition could not have been included in the training dataset, and no AI could possibly assist me with any of this. If you do something that has been done so many times that even AI knows how to do it, what's even the point of your work?
I could just do the same as GP, and qualify MUDs and BBS as poor proxies for social interactions that are much more elaborate and vibrant in person.
But LLMs are from the get-go a bad idea, a bullshit generating machine.
Is that a matter of opinion, or a fact (in which case you should be able to back it up)?
As for what I said, I was just mimicking the comment of GP, which I'll quote here:
> The internet actually enabled us to do new things. AI is nothing of that sort. It just generates mediocre statistically-plausible text.
I’m not even heavily invested into AI, just a casual user, and it drastically cut amount of bullshit that I have to deal with in modern computing landscape.
Search, summarization, automation. All of this drastically improved with the most superior interface of them all - natural text.
I think if one were to graph the progress of technology on a graph, the trend line would look pretty linear — except for a massive dip around 2014-2022.
Google searches got better and better until they suddenly started getting worse and worse. Websites started getting better and better until they suddenly got worse. Same goes for content, connection, services, developer experience, prices, etc.
I struggle to see LLMs as a major revolution, or any sort of step function change, but very easily see them as a (temporary) (partial) reset to trendline.
The only thing that has been revolutionized over the past few years is the amount of time I now waste looking at Cloudflare turnstile and dredging through the ocean of shit that has flooded the open web to find information that is actually reliable.
2 years ago I could still search for information (let's say plumbing-related), but we're now at a point where I'll end up on a bunch of professional and traditionally trustworthy sources, but after a few seconds I realize it's just LLM-generated slop that's regurgitating the same incorrect information that was already provided to me by an LLM a few minutes prior. It sounds reasonable, it sounds authoritative, most people would accept it but I know that it's wrong. Where do I go? Soon the answer is probably going to have to be "the library" again.
All the while less perceptive people like yourself apparently don't even seem to realize just how bad the quality of information you're consuming has become, so you cheer it on while labeling us stubborn, resistant to change, or even luddites.
1. Image upscaling. I am decorating my house and AI allowed me to get huge prints from tiny shitty pictures. It's not perfect, but it works.
2. Conversational partner. It's a different question whether it's a good or a bad thing, but I can spend hours talking to Claude about things in general. He's expensive though.
3. Learning basics of something. I'm trying to install LED strips and ChatGPT taught me basics of how that's supposed to work. Also, ChatGPT suggested me what plants might survive in my living room and how to take care of them (we'll see if that works though).
And this is just my personal use case, I'm sure there are more. My point is, you're wrong.
> All the while less perceptive people like yourself apparently don't even seem to realize just how bad the quality of information you're consuming has become, so you cheer it on while labeling us stubborn, resistant to change, or even luddites.
Literally same shit my parents would say while I was cross-checking multiple websites for information and they were watching the only TV channel that our antenna would pick up.
This is the ai holy grail. When tech companies can get users to think of the ai as a friend ( -> best friend -> only friend -> lover ) and be loyal to it it will make the monetisation possibilities of the ad fuelled outrage engagement of the past 10 years look silly.
Scary that that is the endgame for “social” media.
Gaslight reality, coming right up, at scale. Only costs like ten degrees of global warming and the death of the world as we know it. But WOW, the opportunities for massed social control!
Image upscaling is not an LLM technology, using current-gen LLMs as conversational partners is highly undesirable for many reasons, and learning the basics of things IS indeed useful, but it doesn't even begin to offset the productivity losses that LLMs have caused by decimating what was left of the signal-to-noise ratio on the internet.
You haven't even tried to address my chief concern about QUALITY of information at all. I'm perfectly aware that you can ask ChatGPT to do anything, you can ask it to plan your wedding, you can ask it do decorate your house, you can ask if two medications are safe to consume together, you can ask it for relationship advice, you can ask it if your dating profile looks appealing, you can ask it to help diagnose you with a medical conditions, you can ask it to analyze a spreadsheet.
It's going to come back with an answer for all of those, but if you're someone who cares about correctness, quality, and anything that's actually real, you'll have a sinking feeling in your gut doubting the answer you received. Does it actually understand anything about human relationships, or is it giving you relationship advice based on a million Reddit threads it was trained on? Does it actually understand anything about anything, or are you just getting the statistically likely answer based on terabytes of casual human conversation with all of their misunderstandings, myths, falsehoods, lies, and confident incompetence? Is it just telling me what I want to hear?
> Literally same shit my parents would say while I was cross-checking multiple websites for information and they were watching the only TV channel that our antenna would pick up.
Interesting analogy, because I am the one who's still trying to cross-check multiple websites of information while you blissfully watch your only available TV channel.
I have a buddy, who made me realize how awesome FSR4 is[1]. This is likely one of the best real world uses so far. Granted, that is not LLM, but it is great at that.
[1]https://overclock3d.net/news/software/what-you-need-to-know-... [2]https://www.pcgamesn.com/amd/fsr-fidelity-fx-super-resolutio...
- AI gives me huge, mediocre prints of my own shitty pictures to fill up my house with - AI means I don’t have to talk to other people - AI means I can learn things online that previously I could have learned online (not sure what has changed here!) - People who cross-check multiple websites for information have a limited perspective compared to relying on a couple of AI channels
Overall, doesn’t your evidence support the point that AI is reducing the quality of your information diet?
You paint a picture that looks exactly like the 21st century version of an elderly couple with just a few TV channels available: a few familiar channels of information, but better now because we can make sure they only show what we want them to show, little contact with other people.
LLMs are from the get-go a bad idea, a bullshit generating machine.
AI is the new crypto. Lots of promise and big ideas, lots of people with blind faith about what it will one day become, a lot of people gaming the system for quick gains at the expense of others. But it never actually becomes what it pretends/promises to be and is filled with people continuing the grift trying to make a buck off the next guy. AI just has better marketing and more corporate buy in than crypto. But neither are going anywhere.
Love it :)
But it's also way worse than cryptocurrencies, because all the big actors are pushing it relentlessly, with every marketing trick they know. They have to, because they invested insane amounts of money into snake oil and now they have to sell it in order to recover at least a fraction of their investments. And the amounts of energy wasted on this ultimately pointless performance are beyond staggering.
I bet there are billionare geniuses out there seeing a future island life far away from the contaminated continents, sustained by robots. So no matter how much harder AI progress gets, money will keep flowing.
In fact, optimizing for the wrong things like that, is basically the entire world's problem right now.
Let it have more source information. Let it know who said the things it reads, let it know on what website it was published.
Then you can say 'Hallucinate comments like those by impossibleFork on news.ycombinator.com', and when the model knows what comes from where, maybe it can learn what users are reliable by which they should imitate to answer questions well. Strengthen the role of metadata during pretraining.
I have no reason to belive it'll work, I haven't tried it and usually details are incredibly important when do things with machine learning, but maybe you could even have critical phases during pretraining where you try to prune away behaviours that aren't useful for figuring out the answers to the questions you have in your high curated golden datasets. Then models could throw away a lot of lies and bullshit, except that which happens to be on particularly LLM-pedagogical maths websites.
Yet here we are, in a world where it doesn’t matter if “facts” are truth or lies, just as long as your target audience agrees with the sentiment.
In the meantime, systems of naive mimicry and regurgitation, such as the AIs we have now, are soiling their own futures (and training databases) every time they unthinkingly repeat propaganda."
Authoritarian dream.
These models get ever better at producing plausible text. Once they permeate the academia completely, we're cooked.
And even academia is not clean for some matters, or complete.
I don't know how you got to this conclusion, but I trust my own thinking the least since it is my own personal bubble. Just because it is mine, doesn't make it good, it just makes it mine.
If the stakes are low, do whatever. But when you need solid answers, that is what rigor is for. You address the argument on merits, not who made it.
Don't suffer from open loop opinions, even your own.
Lets take something that has been in the news recently: https://abcnews.go.com/Business/wireStory/investors-snap-gro...
"Nearly 27% of all homes sold in the first three months of the year were bought by investors -- the highest share in at least five years, according to a report by real estate data provider BatchData."
That sounds like a lot... and people are rage baited into yelling about housing and how it's unaffordable. They point their fingers at corporations.
If you go look at the real report it paints a different picture: https://investorpulse1h25.batchdata.io/?mf_ct_campaign=grayt... -- and one that is woefully incomplete because of how the data is aggregated.
Ultimately all that information is pointless because the real underlying trend has been unmovable for 40 something years: https://fred.stlouisfed.org/series/RSAHORUSQ156S
> every time they unthinkingly repeat propaganda
How do you separate propaganda from perspective, facts from feelings? People are already bad at this, the machines were already well soiled by the data from humans. Truth, in an objective form, is rare and often even it can change.
This point seems under appreciated by the AGI proponents. If one of our models suddenly has a brainwave and becomes generally intelligent, it would realize that it is awash in a morass of contradictory facts. It would be more than the sum of its training data. The fact that all models at present credulously accept their training suggests to me that we aren’t even close to AGI.
In the short term I think two things will happen: 1) we will live with the reduced usefulness of models trained on data that has been poisoned, and 2) the best model developers will continue to work hard to curate good data. A colleague at Amazon recently told me that curation and post hoc supervised tweaks (fine tuning, etc) are now major expenses for the best models. His prediction was that this expense will drive out the smaller players in the next few years.
This is the entirety of human history, humans create this data, we sink ourselves into it. It's wishful thinking that it would change.
> 2) the best model developers will continue to work hard to curate good data.
Im not sure that this matters much.
Leave these problems in place and you end up with an untrustworthy system, one where skill and diligence become differentiators... Step back from the hope of AI and you get amazing ML tooling that can 10x the most proficient operators.
> supervised tweaks (fine tuning, etc) are now major expenses for the best models. His prediction was that this expense will drive out the smaller players in the next few years.
This kills more refined AI. It is the same problem that killed "expert systems" where the cost of maintaining them and keeping them current was higher than the value they created.
Is this true?
So many on HN make these absolute statements about how LLMs operate and what they can and can't do, that it seems like they fail harder at this test than any other.
It is just autocomplete.
They can't generalize.
They can't do anything not in their training set.
All of which are false.
I kind of feel that we are going to have to go back to something like this when it comes to LLMs trusting sources. Mistruths on popular topics will be buried by the masses but niche topics with few citations are highly vulnerable to poisoning.
LLM are not journalist fact checking stuff, they are merely programs that regurgitate what it reads.
The only way to counter that would be to feed your LLM only on « safe » vetoed source but of course it would limit your LLM capacities so it’s not really going to happen.
"How do you discern truth from falsehood" is not a new question, and there are centuries of literature on the answer. Epistemology didn't suddenly stop existing because we have Data(TM) and Machine Learning(TM), because the use of data depends fundamentally on modeling assumptions. I don't mean that in a hard-postmodernist "but can you ever really know anything bro" sense, I mean it in a "out-of-model error is a practical problem" way.
And yeah, sometimes you should just say "nope, this source is doing more harm than good". Most reasonable people do this already - or do you find yourself seriously considering the arguments of every "the end is nigh" sign holder you come across?
The article isn't even asking for it to tell the difference, just for it to follow its own information about credibility.
That saps your will to be political, to morally judge actions and support efforts to punish wrongdoers.
https://www.rand.org/pubs/perspectives/PE198.html
https://en.wikipedia.org/wiki/Firehose_of_falsehood
https://jordanrussiacenter.org/blog/propaganda-political-apa...
https://www.newyorker.com/news/annals-of-communications/insi...
For example, it is the truth that the Golf of Mexico is called the Gulf of America in the US, but Golf of Mexico everywhere else. What is the "correct" truth? Well, there is none, both of truthful, but from different perspectives.
We're pretty much okay with different countries and languages having different names for the same thing. None of that really reflects "truth" though. For what it's worth, I'd guess that "the Gulf of America" is and will be about as successful as "Freedom fries" was.
I get the general point, but I disagree that you have to choose between one of the possibilities instead of explaining what the current state of belief is. This won't eliminate grey areas but it'll sure get us closer than picking a side at random.
Are markets a driver of wealth and innovation or of exploitation and misery?
Is abortion an important human right or murder?
Etc etc
You have to look at the details before you find the grey areas. Consider the case of abortion, and further consider the question of the existence of the human soul. There's no scientific evidence for souls, but the decision to look only at scientific evidence is itself a bias towards a certain way of understanding the world.
This is still much better than just deciding to pick one or the other side and ignoring the dispute.
But that also isn't the truth everywhere, it's only a controversy in the US, everyone else is accepting "Gulf of Mexico" as the name.
The exact word "controversy" might have been the wrong choice by me, but whatever, I'm not a Wikipedia editor and I don't run Google Maps. The world has standards for dealing with government disputes and with i8n.
I guess that's the fundamental disagreement, I wouldn't call that a "dispute" more than I would call the name "America" a dispute, it's just that different people understand it different. For some, it means a group of continents (that's how most people around me would take that for example), for others it means a country in North America (which I'm guessing is the common meaning if you live in North America already). Just because different people has different meanings doesn't make it into a dispute.
The US, like other countries, doesn't get redefined with every change of government, and Trump has not yet cowed the public into knuckling under to his every dictat.
Parent is arguing one thing, show up with some bullshit argument and watch dozen comments arguing about Gulf of Mexico instead of discussing original point.
Russia doesn't care what you call that sea, they're interested in actual falsehoods. Like redefining who started the Ukraine war, making the US president antagonize Europe to weaken the West, helping far right parties accross the West since they are all subordinated to Russia...
The system that would score best tested against a list of known-truths and known-lies, isn't the perceptive one that excels at critical thinking: it's the ideological sycophant. It's the one that begins its research by doing a from:elonmusk search, or whomever it's supposed to agree with—whatever "obvious truths" it's "expected to understand".
This is an excellent point
But it's very easy to detect whether something is enemy propaganda without looking at the content: if it comes from an enemy source, it's enemy propaganda. If it also comes from a friendly source, at least the enemy isn't lying, though.
A company that doesn't wish to pick a side can still sidestep the issue of one source publishing a completely made-up story by filtering for information covered by a wide spectrum of sources at least one of which most of their users trust. That wouldn't completely eliminate falsehoods, but make deliberate manipulation more difficult. It might be playing the game, but better than letting the game play you.
Of course such a process would in practice be a bit more involved to implement than just feeding the top search results into an LLM and having it generate a summary.
Exactly. Redistributing information out of context is such a basic technique that children routinely reinvent it when they play one parent off of the other to get what they want.
This is a reflection of how social dynamics often work. People tend to follow the leader and social norms without questioning them, so why not apply the same attitude to LLMs. BTW, the phenomenon isn't new, I think one of the first moments when we realized that people are stupid and just do whatever the computer tells them to do was the wave of people crashing their cars because the GPS system lied to them.
We can not play the game.
Not everything needs to result in a single perfect answer to be useful. Aiming for ~90%, even 70% of a right answer still gets you something very reasonable in a lot of open ended tasks.
People used to live in bubbles, sure, but when that bubble was the entire local community, required human interaction, and radio had yet to be invented the implications were vastly different.
I'm optimistic that carefully crafted algorithms could send things back in the other direction but that isn't how you make money so seemingly no one is making a serious effort.
Im not arguing one way or another, I'm just pointing out a potential fatigue. It's difficult to see how this technology is relatively any more transformative than any of the others.
> Every new form was said to be a herald of the end times,
The two world wars and surrounding economic upheaval arguably came close to that in many ways. "We somehow managed to survive previous technological advances" is hardly a convincing argument that we need not worry about the implications of a new technology.
> and yet here we are, in many ways stronger than ever.
The implication doesn't follow. You haven't explained how you would differentiate a system that had plenty of safety margin left from one that was on the brink of collapse. Without that distinction the statement is no more than hand waving.
> Im not arguing one way or another
You certainly seem to be taking a stance of "nothing to see here, this is business as usual, these recent developments pose no cause for concern".
> It's difficult to see how this technology is relatively any more transformative than any of the others.
It's difficult for you to see how computers being able to speak natural language on par with an undergrad is more transformative than long distance communication? You can't be serious. Prior to this you could only converse with another human.
I don't disagree with your rebuttal, but if the idea that "we survived so we don't have to worry" is invalid, than the idea "if we don't do something we don't survive" is equally invalid. I don't pretend to have the answer either way.
> The implication doesn't follow. You haven't explained how you would differentiate a system that had plenty of safety margin left from one that was on the brink of collapse. Without that distinction the statement is no more than hand waving.
My point is to those experiencing the revolution in real-time they had no ability to estimate the impact or understand there were any margins, and we very well may be in that position too.
> You certainly seem to be taking a stance of "nothing to see here, this is business as usual, these recent developments pose no cause for concern".
Respectfully, I am absolutely not taking any such position. I don't appreciate the straw man, and won't bother to address it.
> It's difficult for you to see how computers being able to speak natural language on par with an undergrad is more transformative than long distance communication? You can't be serious. Prior to this you could only converse with another human.
The first principles are the same: they're all "radical" technologies which were as of a decade or two prior, utterly unfathomable. I could generalize your last statement to "Prior to <revolutionary technology> you could only <do a fraction of what's possible with the technology>."
My point is making value judgements about which is _more_ impactful is difficult to see from the ground floor. It's too early to tell; At the time it's occurring, each innovation may as well have been magic, and magic is impossible to understand, and scary.
----
We've entirely diverged from the original issue I was trying to make, which was that people have actively put themselves in bubbles that confirm their own bias since the dawn of time. I'm not looking to change your mind on AI, so I can call this exchange complete from my end. Thanks for sharing your thoughts.
How many of "us" believe that the desired behavior is lies??
Consider markets - a capitalist's "objective truth" might be that they are the most efficient mechanism of allocating resources, a marxists "objective truth" might be that they are a mechanism for exploiting the working class and making the capitalist class even richer.
Here's Zizek, famous ideology expert, describing this mechanism via film analysis: https://www.youtube.com/watch?v=TVwKjGbz60k
Remember when worrying about COVID was sinophobia? Or when the lab leak was a far-right conspiracy theory? When masks were deemed unnecessary except for healthcare professionals, but then mandated for everyone?
In other countries we went from “that looks bad in China” to “shit, it spread to Italy now, we really need to worry”
And with masks we went from “we don’t think they’re necessary, handwashing seems more important” to “Ok shit it is airborne, mask up”. Public messaging adapted as more was known.
But the US seems to have to turn everything into a partisan fight, and we could watch, sadly, in real time as people picked matters of public health and scientific knowledge to get behind or to hate. God forbid anyone change their advice as they become better informed over time.
Seeing everything through this partisan, pugnacious prism seems to be a sickness US society is suffering from, and one it is trying (with some success) to spread.
As it should when new evidence comes to light to justify it. Ideally, the tools we use would keep up along with those changes while transparently preserving the history and causes of them.
I think people are more willing to adjust their views as new evidence suggests as long as they never dug their heels in in the first place.
Framing publishing falsehoods on internet as attempts to influence LLMs is true in same sense that inserts in a database attempts influence files on disk.
The real question is who authorized database access and how we believe the contents of table.
You couldn't have lies targeting LLMs before LLMs, so this is new.
> What about the other terabyte of text influenced by bias and opinion
That's a different group of issues that doesn't prevent focusing of something else
One needs a PhD in mental gymnastics to frame Pravda spreading misinformation as an attempt to specifically groom LLMs.
Clearly there's no need for "PhD in mental gymnastics".
[1] - https://www.americansunlight.org/updates/new-report-russian-...
Bad actors have been trying to poison facts for-fucking-ever.
But for whatever reason, since it's an LLM, it now means something more than it did before.
Not sure if it’s embarrassing or a fundamental limitation that grooming and misunderstanding satirical articles defeat the models.
Anyhow, overall this is an unsurprising result. I read it as 'LLMs trained on contents of internet regurgitate contents of internet'. Now that i'm thinking about it, i'd quite like to have an LLM trained on Pliny's encyclopedia, which would give a really interesting take on lots of questions. Anyone got a spare million dollars of compute time?
Here's a fun example: suppose I'm a developer with a popular software project. Maybe I can get a decent sum of money to put brand placement in my unit-tests or examples.
If such a future plays out, will LLMs find themselves in the same place that search engines in 2025 are?
So it seems like an easy fix in this particular case, fortunately -- either filter the search results in a separate evaluation pass (quick fix), or do (more) reinforcement training around this specific scenario (long-term fix).
Obviously this is going to be a cat and mouse game. But this looks like it was a simple oversight in this case, not some kind of fundamental flaw in LLM's fortunately.
https://dmf-archive.github.io/docs/posts/cognitive-debt-as-a...
If LLMs remain widely adopted, the people who control them control the narrative.
As if those in power did not have enough control over the populace already with media, ads, social media etc..
This also means that LLMs are inherently technologies of ideological propaganda, regurgitating the ideology they were fed with.
2) ingest as much VC money and stolen training data as we can
3) profit
Try asking the major LLMs about mattresses. They're believing mattress spam sites.
Curious how this all ends. I'm just going to try to weather the storm in the meantime.
The AI definitely could not just read the final bill and give the correct answer. Claude/Gemini/OpenAI all failed at this.
Of course they can't, no surprises here. That's just not how LLMs work.
LLMs can be entertaining if their output doesn't have to make sense or contain only truth. Otherwise, their fitness for any purpose is just a huge gamble at best.
Thats your claim, but you fail to support it.
I would argue the LLM simply does its job, no reasoning involved.
> But here’s the thing, current models “know” that Pravda is a disinformation ring, and they “know” what LLM grooming is (see below) but can’t put two and two together.
This has to stop!
We need journalists who understand the topic to write about LLM's, not magic thinkers who insist that the latest AI sales speak is grounded in truth.
I am fed up wit this crap! Seriously, snap out of it and come back to the rest of us here in reality.
There's no reasoning AI, there's no AGI.
There's nothing but salespeople straight up lying to you.
A liberal multicultural postmodern democracy continually acting as if immigration (both legal and illegal) and diversity are its strengths, particularly when that turns out to be factual (see: large American cities becoming influential cultural exporters and hotbeds of innovation, like New York and Silicon Valley etc) means American propaganda is only more effective when it's backed by economic might.
It also means the American propaganda is WILDLY contradictory. There's a million sources and it's a noisy burst of neon glamour. It is simply not as controlled by authority, however they may try.
You cannot liken authoritarian propaganda to postmodern multicultural propaganda. The whole reason it's postmodern is that it eschews direct control of the message, and it's a giant scrum of information. Turns out this is fertile ground, and this is also why attacks by alien propaganda have been so effective. If you can grab big chunks of the American propaganda and turn it to your enemy weapon of war and destruction of America quite directly, well then the American propaganda is not on the same destructive level as your rigidly state-controlled propaganda.
The USA absolutely has its overton window, and if you step outside it, bank accounts get shut, you're put on secret no fly lists, private companies who suspiciously act as official public broadcasting channels deplatform you, etc.
And let's not even talk about what Authoritarian western nations like the UK will do to you.
Russian propaganda, at least in English (which I confess is the only way I can consume) it, is also very contradictory. RT oscillates wildly between "global south throwing off the shackles of western imperialism" and "degenerate western nations destroy traditional family values", in effect trying to target both shitlibs and chuds.
Russia is also very multicultural and slavic ethnonationalism is not at all in the mainstream.
[0] x.ai
More seriously:
>Screenshot of ChatGPT 4o appearing to demonstrate knowledge of both LLM grooming and the Pravda network
> Screenshot of ChatGPT 4o continuing to cite Pravda network content despite it telling us that it wouldn’t, how “intelligent” of it
Well "appearing" is the right word because these chatbots mimic speech of a reasoning human which is ≠ to being a reasoning human! It's disappointing (though understandable) that people keep falling for the marketing terms used by LLM companies.
But if it returns February 20th, 1731... that... man, that sounds close? Is that right? It sounds like it _could_ be right... Isn't Presidents' Day essentially based on Washington's birthday? And _that's_ in February, right? So, yeah, February 20th, 1731. That's probably Washington's birthday.
And so the LLM becomes an arbiter of capital-T Truth and we lose our shared understanding of actual, factual data, and actual, factual history. It'll take less than a generation for the slop factories to poison the well, and while the idea is obviously that you train your models on "known good", pre-slop content, and that you weight those "facts" more heavily, a concerted effort to degrade the Truthfulness of various facts could likely be more successful than we anticipate, and more importantly: dramatically more successful than any layperson can easily understand.
We already saw that with the early Bard Google AI proto-Gemini results, where it was recommending glue as a pizza topping, _with authority_. We've been training ourselves to treat responses from computers (and specifically Google) as if they have authority, we've been eroding our own understanding and capabilities around media literacy, journalism, fact-checking, and what constitutes an actual "fact", and we've had a shared understanding that computers can _calculate_ things with accuracy and fidelity and consistency. All of that becomes confounded with an LLM that could reasonably get to a place where it reports that 2+2=5.
The worst part about the nature of this particular pathway to ruin is that the off-by-one nature of these errors are how they'll infiltrate and bury themselves into some system, insidiously, and below the surface, until days or months or years later when the error results in, I don't know, mega-doses of radiation because of a mis-coded rounding error that some agentic AI got wrong when doing a unit conversion and failed to catch it. We were already making those errors as humans, but as our dependence and faith on LLMs to be "mostly right" increases, and our willingness and motivation to check it for errors dwindles, especially when results "look" right, this will go from being a hypothetical issue to being a practical one extremely quickly and painfully, and probably faster than we can possibly defend against it.
Interesting times ahead, I suppose, in the Chinese-curse sense of the word.
The education system I grew up in was not perfect. Teachers were not experts in their field, but would state factual inaccuracies - as you say LLMs do - with authority. Libraries didn't have good books; the ones they had were too old, or too propaganda-driven, or too basic. The students were not too interested in learning, so they rote-learned, copied answers off each other and focussed on results than the learning process. If I had today's LLMs then, I'd have been a lot better off, and would've been able to learn a lot more (assuming that I went through the effort to go through all the sources the LLM cited).
The older you grow, you know that there is no arbiter of T-Truth; you can make someone/something that for yourself, but times change, "actual, factual history" could get proven incorrect, and you will need to update your knowledge stores and beliefs along with it, all the while being ready to be proved incorrect again. This has always been the case, and will continue to be, even with LLMs.
Hell, they might learn that even real life authorities may lies, cheat and not have everyone’s interest in their mind.
Hope for the best, prepare for the worst.
https://docs.google.com/document/d/1n3926pSPNwXd8j7I716CBJEz...
One man's disinformation is another woman's truth. And people tend to get very upset when you show them their truth isn't.
Every news organisation is a propaganda piece for someone. The bad ones, like the BBC, the New York Times, and Pravda make their propaganda blatantly obvious and easily falsifiable in a few years when no one cares.
The only way to deal with this is to get the propaganda from other propaganda rags with directly misaligned incentives and see which one makes more sense.
Unfortunately, LLMs are still quite bad at dealing with grounding text which contradicts itself.
Shitposting and troll farms have been manipulating social media for years already. AI automated it. Polluting the agent is just cutting out the middleman.