The AI bullshit singularity
successfulsoftware.net
successfulsoftware.net
In the last internet revolution (web search), results started high quality because the inputs were high quality - bloggers and others just wanted to document and share knowledge. But over time, many interests (largely commercial) figured out how to game the system with SEO, and quality of search results has decreased as search's incentive structure led to lower quality data being indexed.
We're at the start of the LLM revolution now - models are trained on high quality inputs (which may be as rare as "low-background steel" in the future). But the models allow the mass production of lower quality outputs with errors and hallucinations; once those get fed back into new models, are we doomed to decreasing effectiveness of LLMs, just as we've seen with search? Will there be LLMO (LLM optimization) to try to get your commercial interests reflected in the next generation models?
I think we've got a few golden years of high quality LLMs before that negative feedback loop really starts to hurt like it did in search.
Open source AI needs to get a lot of investment for this to be mitigated. Relying on market incentives to drive development without the possibility of forking is too dangerous given the high stakes.
If we can only dicuss ideas which have never bee shared we can close this website right now
It’s just really clear that a giant text averaging machine can only go so far, and while we do see some higher level emergent properties, we’re going to have to move beyond the current state of the art in the next decade, and such future systems may be much less affected by the internet’s bullshit. Even without the bullshit singularity I think such measures would be a necessity and we would see them developed soon.
It's not really a text averaging machine, it's a pattern matching machine.
Right now the "depth" of the patterns it can match can only go so far, but in a few years with more advances in chips and memory the depth is going to increase and the patterns it can match will fan out accordingly.
Since all deployed models produce some kind of effect and feedback from the world there is an opportunity there to collect data targeted on the current level of the model, the most useful kind of data. That's why I think AI will be ok even with the proliferation of bots online. It's not 100% pure synthetic data in a loop, it is a agent-environment loop.
tl;dr Models learn better from their own experiences, not ours.
I just finished rereading Do Androids Dream for the first time in 20 years or so, and was astonished at how similar his andys really are to LLMs in the polluted / destroyed reality there. How confusing and corrupting they are to organic life. PKD describes the androif brains as neural networks with thousands of layered pathways and trillions of weighted parameters, and it's as if he was able to accurately conceive of what linguistic and "emotional" strengths and weaknesses those constructs would actually have, decades before LLMs existed. And there's this one amazing line where Deckard calls them "Life thieves". What else should we call what Sam Altman and others are building - but theft of human ingenuity, creativity, and basic reason for living, and the utter annihilation and suppression of people like this 16 year old kid who dare to hope they can contribute something more original in life than being a servant of a tech company building this shit, or a tiktok influencer who writes prompts?
What should that kid hope: That their work becomes noticeable enough to be immediately stolen and their name turned into a prompt?
Life thieves.
Just as a further aside, I had dinner tonight with a friend who's a fairly famous animator in the commercial realm, and I brought up this post. He's just sure the kid's screwed and the genie is out and creative is basically over. He's turning to building wooden clocks.
But his reaction, and the reactions I see here every time this comes up, remind me of something else. They remind me of how people react when someone is robbed. Everyone has some reason why it's bad but not that bad, it was inevitable, it'll be okay, etc. Or they go around wondering what they're going to do now. Or they paper it over with optimism. Surprisingly few people get robbed and are willing to realize they were robbed, and become wildly pissed off about it. Most people have some sort of flight reaction, as evidenced by e.g. the promotion of "prompt writing" or learning to use a paid API to do what was formerly your own creative job that now spits your own work back at you.
I say sue the shit out of all these content thieves.
If the AI was good at detecting it, it wouldn't matter if the AI-generated content sucked, yes?
Even a low probability of detection would help. Let's say our algorithm is 50% likely to detect AI junk. That means that half the junk data won't make it in to model. Even 20% would probably be worthwhile, especially if it also threw away human-generated junk (and let's be realistic here: there is, and always has been, no shortage of terrible and/or wrong human-generated content).
Let's say those crappy filler paragraphs that get stuck between pictures on meme clickbait pages... I suspect most of those people have already been replaced, but that prose was horrible long before the current LLM boom.
I suspect there is a lot of effort being expended right now on ways to ensure the training data (whether human or AI generated) isn't shite.
Then how can humans often tell that something is generated by AI?
Besides, I wasn't suggesting AI detection as such. I was suggesting crap detection, regardless of how the crap was produced.
You probably don't want crappy human-generated content going into your training set, either.
An optimistic take would be that LLMs will make curation so much more valuable, that it will be done much better. If the wider world gets to use this curation to limit spam, and highlight good work, it would be amazing for the world.
Won't that also mean a return to a world of gatekeepers?
So I'm not nearly as pessimistic.
It's very unlikely for 100% of all intelligent curators to become dishonest either so it doesn't seem like a serious roadblock.
At worse, newer models will get worse and you can just stick to older models.
You could also argue that proprietary models gated by an API are better than anything you can run locally, and yeah maybe those will get worse with time.
They're not going to get any worse than what you can run locally though. If they do, open models will overtake them, and then we'd be in a better position overall.
You can't run an up to date model locally. When I ask Googles models they have knowledge from stuff just a week ago, without using search. You wont get that from a giant local model.
I'd add that majority of LLM-generated stuff put onto the internet isn't pure junk. It's endowed with human-created context and curation. If someone submits some LLM-created code to Github, it's because it works. A small fraction is pure noise (e.g. state-actors spamming Twitter) but that should remain a minority.
Even if the data is 100% synthetic, you can still hill climb to new mountains.
If you don't believe me, look at evolution.
It doesn't matter if we no longer have 100% human art as input. This is the worst these systems will ever look and feel, and they're only going to improve.
I'd be willing to do a longbets on this one.
Likewise SEO incentivizes mass production of low quality LLM text, because they "quality" they are optimizing for is impressions, not actual quality.
My undergrad was in biology, "bro". I was cloning luciferace into plants using agrobacterium-mediated transfection over a decade before this week's news of transgenic petunias.
I was planning to do a PhD in computational metabolomics, but life took a different turn: a couple of Google engineers saw the laser projector I built and programmed to play video games on the side of skyscrapers [1], and they lured me away to work on what became a decacorn. My quality of life and net worth certainly thank me for the pivot.
Take a look at my post history [2]. My analogies are informed and salient.
In any case, I'm working directly in this field now and the results we're achieving don't need affirmation. I tried to communicate this to outsiders in an easy to digest analogy. I'll let our work speak for itself.
[1] https://youtu.be/5XTi-jf-ans
[2] https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Plugging LLM into this analogy would lead to a story where we're at the initial "Permian explosion" phase of evolution. New LLMs are rapidly "hill climbing" as they feed on virgin data scraped from the Internet. But as those LLMs diversify and adapt, they peter out once all niches have been filled and the fuel consumed.
Another analogy is the Petri dish, but that's an even more pessimistic analogy than adaptive radiation.
We have the pressures of growth and novelty. My analogy is perfect.
Art is very heavily tagged, accurately described and filtered. Many galleries don't accept AI art at all, which means to sneak in the AI needs to be pretty much perfect.
Also, there's an enthusiastic scene of LoRAs where the makers work with small enough datasets to do manual curation.
That's status quo. Because right now artificial art is easy to distinguish form "natural art". Besides that I consider this - no offense! - as some kind of "arrogance". Galleries accept what they think what art is. Why can't I, the art consumer, decide for myself, what art is?
Future world leader advisor.
Ahem, I hope not.
All in all LLMs being more gullible than your average 6 year old is really not helping. Humans will have to (continue to) curate the data we train LLMs on.
I think people are substantially worse at figuring out what is “good” data, and I think a huge number of people are/are starting to nominate LLMs as those “authority figures”. And, given that, I think there is a low ceiling for the creators of these tools to clear to make them appealing to users.
When LLMs started to go mainstream I said I was not afraid of AI, but was afraid of (unwise) people with AI. This phenomenon is what I foresaw.
The worst thinkers are the ones most hungry to outsource their decision making.
The fact that LLM output requires good judgement by the recipient to filter the wheat from the chaff, and those with poor judgement skills are most likely to lean on decision tools makes for a potentially volatile situation.
At doing whatever the model is suppose to do.
The key is that you need some kind of external indicator that tells you which generated examples are good and which are bad. In the case of alphazero you get that by simulating games and seeing who wins, in the case of LLMs you will be only taking the generations that are 'successful', e.g. which HN posts get upvoted.
I see it this way, maybe there’s no new ‘knowledge’ but the AI can apply our collective knowledge better than we can. Within that set of knowledge there are surely discoveries never yet realized based on by the fusion of ideas.
Pre-AI we rely on individuals Einstein, Bohr, and Oppenheimer and their associations and studies of each others’ work. With AI, we can fuse the corpus of scientific discoveries into a single entity that we each can communicate with. Maybe the AI lacks the spark of creativity needed to make discovery, but put today’s Einstein in front of it and what would he ask? How much boost would it give him?
Einstein said - “I have no special talent. I am only passionately curious.”
Recently a potential investor asked if we were using AI in our product. I said, “no, not at all, just a neural network.” They seemed satisfied.
And everybody's asking what AI is going to be and can do for them, all the while freely working for it. Don't ask what AI can do for you, just do for AI what is asked of you.
Used to be real people pretended to be girls on the internet. These days I can’t even get an honest real fake person pretending to be an attractive female on LinkedIn.
Parrots can speak, but they cannot reason. Moreover, evolutionary, birds learned to mimic the speech to fool other species. To fool in such way that other species would think that parrots are of the same specie.
We, the humans, are smart enough to recognise that even though parrots can talk, they are not as smart as we are. Unfortunately for us, LLMs are much more advanced things than parrots, and they are capable to fool broad masses pretending that LLMs are of the human specie with all inherent features including the reasoning intelligence (that we cannot easily test externally).
This is unfortunate, because such false believes slow down actual scientific progress towards the natural intelligence researches, and towards creating of the true reasoning artificial intelligence.
Btw, I wouldn't be surprised that if one day the AGI created, it will not be able to speak at all, nor recognise the images.
At least when my pet parrot manages to fool me into whatever it is he wants to get out of me that time, he does it for his own primal benefit and not because it was programmed to adhere to some strict set of ethic and moral boundaries set by legal requirements and someone else's idea of how people should think and behave.
Parrots are definitely smart, and parrots can definitely speak, but they can't learn to speak beyond what they are trained to do. It's not like a parrot has ever spontaneously put together a coherent sentence independently.
LLMs aren't even that smart. Not even close.
Regardless, speech is just another means of communication. In the end it doesn't really matter if I use perfectly articulated Swahili or just scream "aaar" at you for a few seconds as long as I get the cheeseburger with extra cheese I want from you. These parrots, much like children, just push your buttons and are quite good at finding (or negotiating) the right pattern of things to do and sounds to make to get to a certain outcome.
What makes human specie special is the ability of some individuals to create new things that didn't exist before. That's a loose criteria of course, because most individuals just follow educated social constructs for their entire life.
Relatively advanced problem solving abilities are well documented in some species of birds, such as crows.
>LLMs are much more advanced things than parrots
Hardly. Parrots are able to fly, find food, find mates, interact with other parrots and lots of other things than no LLM is even close to being able to do.
Sorry, I didn't mean to offend the parrots. I like parrots too! :)
In this sense if the ML developers would be focused on the inventions automatisation, I would expect that they would choose something more formalised than the natural language.
Anyway, whatever way and methods they chose I think the better external estimation criteria of intelligence should be the ability to make completely new things that clearly didn't exist before. Not just reasoning about existing one. After all, it's not a new thing that computers are able to deduce. Any programming language can do that better than any chat bot.
The model predicts a 99% chance that the next word is "alive" and a 1% chance that it's "dead".
The llm calculates probabilities. How does it actually choose the word? It throws a weighted die. It literally chooses one at random (albeit on a custom probability distribution).
So tell me, what how can you eliminate hallucinations from something that is literally designed to pick stuff at random?
Hallucinations will never be removed from these types of llms. Hallucinations are fundamental to how they work. In the sense that even the "good" outputs are hallucinations picked out at random from a probability distribution.
Any company that says they can control hallucinations, in any way, is flat out lying.
My first thought was that this would create more money in genuine creativity, which would be great. But instead it feels more likely there will be much more telemetry and tracking to determine whether an input is made by you (and thus by a human) or not.
"Only trust stuff that can be traced to a specific human" is a great way weed out non-human stuff. But it requires a pretty effective surveillance system...
The MO for these vendors and the AI companies that buy their data seems to be a race to the bottom in price with little concern for quality. The current industry norm is to outsource the work to developing countries, where the cost of living is more in line with what the annotation agencies are willing to pay. While this isn't necessarily problematic for quality in and of itself, it does seem to make it harder to find candidates with the English skills required to generate high-quality RLHF and SFT training data. Furthermore, the pay offered for coding annotators is not competitive with local pay for software engineers, making it challenging to recruit skilled programmers. A lot of coding annotation is done by beginners and students.
There is certainly a lot of hype surrounding LLMs and their potential to disrupt various industries — even the US DOD has been scoping out the potential use of LLMs to assist military commanders in strategic decisions. However, if we want these LLMs to consistently perform at an expert level, they need expert-level training data. I worry that producing this data at the quality and scale required may be prohibitively expensive, and could cause a major bottleneck in model improvements long-term.
Besides curation from a trusted source, what other solutions are there to prevent drowning in hay?
That Michigan PhD student dump was by an unauthorized third party and was shut down. But was the pricing reasonable? Are admins of old school message boards sitting on now valuable DB dumps of activity on the board in the form of millions of messages?
One could argue that LLMs have some form of intelligence already even though I believe it falls more under “very good intuition”. Even if the internet is filled with trash, we still have (and will continue to do so I hope) a lot of content of high quality available (all the books, podcasts, videos, etc. produced up until now). If you take human intelligence, it doesn’t require the whole internet to start to get smart. A lot of interactions with the real world and some good books/videos should be enough to get to AGI (and start the dreaded feedback loop). We just haven’t found a way to achieve it yet (AFAIK).
Were it to improve by 0.5, then 0.333, then 0.25, then 0.2 and so on would be an entirely different matter!
Saying that models will get attracted to bullshit local maximum is similar fallacy to saying that wikipedia will be full of rubbish when it was created. Forces are set up in a way that creates improvements that accumulate, humans don't represent any ceiling and unlike humans models have near zero replication cost, especially time wise.
Like there is only so much you can do with a single punch card.
For example information communication rate that humans can perform (read or write) compared to what computers can do.
Same with information storage, retrieval, precision, computation rate etc.
If people are worried about a runaway effect, why would you think you can dismiss their concerns by constructing a very specific scaling function that will not result in a runaway effect?
The first versions being trained on "human knowledge" seems kind of like a proof-of-concept. Future iterations will be much, much smarter.
We've spent endless time and money sealing ourselves off from the great unwashed only to expose ourselves to them in full force on the internet (plus Russian paid trolls). It doesn't make much sense.
Also, this will put pressure on human creativity to produce weirder stuff that can't be produced by simply paraphrasing and remixing known ideas. Mass photography was bad news for the portrait painting industry, but forced painters to think outside known concepts and explore crazy abstract art.
I totally look forward to whatever humans come up with next, that can't be easily generated by a computer program.
You know, there's no replacement for knowing how to do something yourself. And no, you cannot be as good creatively if you haven't learned a craft. (And no, prompt editing is not a craft any more than ordering food in a restaurant is cooking. Could you even describe in a useful way what was wrong with a soup you were served if you had never learned to cook?)
For example, a version of popular software is released with new functions. A framework could simulate numerous plausible code snippets that exercise the new release, including edge cases. The code snippets would be amply commented, and proven to work in a test harness.
This method is the opposite of depending upon the internet for training data. And this method is being used to train AI robots, that have no available internet training data.
The argument that LLMs can’t possibly scale because of data contamination falls apart the moment we discover a method to incorporate context-learning into the training loop.
Will high quality training data be needed for incremental improvements?
What is the upper bound of improvements, and what is it contingent on? Compute, training data, etc.
You're arguing that there isn't a trust filter that can do that, but there will be, and it will probably be an AI.
What kind tends to limit cryptography from solving problems is that controlling a secret is actually quite difficult.
Cryptography and human behavior are incompatible.
AI seems to add another dimension of explosive hype
Eg, this article still shows up on the second page. After a while, when the replies slow down, it will move back to the first page (if it's still getting upvotes).
Obviously, the enshittification threat as the author points out is significant. It will be interesting problem to deal with in the future, but until then I feel like no one knows when or what that will look like.
At this rate, no one can predict what AI will bring us 6 months in the future even!
Then, they were shown the diagram that describes its behavior. Even after that, they kept insisting it was sentient, etc.
Some people just want to believe bullshit.
Also, for bonus points define intelligence in a manner that describes simple systems with rudimentary intelligence up to complex systems with higher levels of intelligence than a human. Can you do it, because as of so far no one else has.
"There is no agreed-upon definition of the concept of intelligence neither in psychology nor in philosophy. Experts’ definitions differ widely."
Intelligence itself seems to be a bunch of different behaviors and abilities that when combined have emergent behaviors that are difficult to predict and reduce to simple systems.
ELIZA replies to almost everything with questions. So ELIZA is constantly prompting you to do the work required to continue the conversation.
It’s knowing who to trust.
One of the defining factors under the Stasi in East Germany, it was not that “ordinary people” could not recognise what they heard on the radio was bullshit, but that they could not know who to trust to say “that’s bullshit”. Every family has a Stasi informer, so whilst everyone knew the regieme was lying there was no critical mass event.
There are a dozen technical solutions to “bullshit”. We just need to ensure we have institutions that support us standing up to it.
There is a human choice in a lot of matters, especially when it comes to how and why one perceives quality.
Quality in literature. Food. Arts. Fashion. Blogs. Programming. Acting. And all such things.
What I believe, based on a both cultural and philosophy stance at where the world is and where its going, is that, at best, so-called AI will push an even larger human segment towards finding the truth for oneself in a different and orginal aspect in terms of being human and how one perceives reality.
I think that the intelligent and thoughtful individual will find plenty of ways to nuture one's own abilities towards quality, which has always been the only true real value you can measure yourself against.
No one will stand a chance against the fluff and the obvious stupidity of "my ai will call your ai" and therfore, the only real direction to follow is the one that gives you an edge for yourself based on your fundamental skills you learn through years and years of experience and personality.
The reason great books are great is not that it's written. It's that it sips through the words and sentences, by their very human authors, that you can feel them inside of you.
There will be, is my belief, a larger calling in the world for removing oneself from the fast paced results which has absolutely no nerve or soul.
I am not afraid at all. I only feel sorry for the humans that has nothing to give in themselves to obtain a larger meaning with their time spent, in whatever profession or life they pursue.
You could ask it any question, ”why does my hair look dead if using a hairdryer” and it actually gave an on-point 100% relevant no-bullshit answer. Try googling that, it’s a million seo spam results, none which answer the question
I get it, engineering trains us to look for failure modes, but my god try to have a little amazement at the progress.
FFS.
If I still need to do that with "AI", what purpose does it serve?
It's a lazy argument to imply this 1. invalidates the technological achievement 2. prevents iterative improvement a la the singularity. For one, the Internet itself is not bullshit just because a lot of spammers/hustlers put bad content on it to try to make money. And secondly, you can curate datasets... nothings stopping researchers from training LLMs on shitty SEO now, and if they wanted to they could curate datasets going forward to try to prevent LLM-spam from entering the training sets of future models.
And finally, people already use reputation/identity/branding and proxies for it as quality filters on the internet. For example, this is an unfamiliar blogger to me and so I entered it with skepticism I wouldn't have with people like Gwern or Lynn Alden. Good writing from people like Gwern and Lynn Alden won't disappear just because LLM content exists on the internet - it just makes reputations and identity (eg to a real human) more important.
Its the needle in a haystack problem, where AI is used to increase the size of the haystack, making the needle (i.e. quality content) harder to find.
People choose to use push models for content through meta properties, tiktok, and aggregators like reddit and HN, but nothing is forcing them to. If they push enough bad content, people won't keep using them. Already happened with Facebook and Reddit predecessors, probably happening to Reddit now.
It doesn't matter how big the haystack is when you have the ability to go directly to the needle.
It's on the first page only because it talks about a very widespread fear, namely that LLMs might actually become dumber because they are trained on their own output.
In reality, LLMs are already trained on the output of other LLMs, as specific and well-directed training is much more effective than a disorderly ingestion of text produced by illiterate internet users. Let's not forget the demographic makeup of the typical average user. I think that training on reddit and 4chan messages can be much more harmful than training on text produced by the worst chatbot.
Hacker News editors are trying to build a narrative to instill fear and pessimism.
"What's the fun in writing on the internet anymore?" explains that everyting can be stolen and rewritten by AIs. And today they talk about LLMs and the whole internet becoming dumber. Tomorrow they'll talk about news falsification and LLMs used to replace writers. https://news.ycombinator.com/item?id=39415878
That's a fact, but are there any studies on training a LLM on it's own output and not the output of a different LLM ? For instance, chatgpt gets knowledge updates so I understand it must be retrained somehow. What happens if the retraining data contains large patches of its own output ? Has this scenario been explored ?
Most gpts, including chatGPT, are trained on the chatGPT outputs.
Plus, a lot of chatGPT training material is hand picked by humans.
Oh sweet summer child, you must be new here.