every time you respond to an AI model "no, you got that wrong, do it this way" you provide a very valuable piece of data to train on. With reasoning tokens there is just a lot more of that data to train on now
Dinosaur bones are either 65 million year old remnants of ancient creatures or decoys planted by a God during a 7 day creation, and a large proportion of humans earnestly believe either take. Choosing which of these to believe involves a higher level decision about fundamental worldviews. This is an extreme example, but incorporating “honest” human feedback on vaccines, dark matter, and countless other topics won’t lead to de facto improvements.
I guess to put it another way: experts don’t learn from the masses. The average human isn’t an expert in anything, so incorporating the average feedback will pull a model away from expertise (imagine asking 100 people to give you grammar advice). You’d instead want to identify expert advice, but that’s impossible to do from looking at the advice itself without giving into a confirmation bias spiral. Humans use meta-signals like credentialing to augment their perception of received information, yet I doubt we’ll be having people upload their CV during signup to a chat service.
And at the cutting edge level of expertise, the only real “knowledgeable” counterparties are the physical systems of reality themselves. I’m curious how takeoff is possible for a brain in a bottle that can’t test and verify any of its own conjectures. It can continually extrapolate down chains of thought, but that’s most likely to just carry and amplify errors.
If you let a model write code, test it, identify bugs and fix them, you get an increasingly obtuse and complex code base where errors happen more. The more it iterate the worse it gets.
At the end of the day, written human language is a poor way of describing software. Even to a model. The code is the description.
At the moment we describe solutions we want to see to the models and they aren't that smart about translating that to an unambiguous form.
We are a long was off describing the problems and asking for a solution. Even when the model can test and iterate.
All of this is possible in a bottle, but laughably far beyond our current capabilities.
Efforts to feed deployed AI models various epistemic poisons abound in the wild.
1. Online trolls and pranksters (who already taught several different AIs to be racist in a matter of hours - just for the LOLs).
2. Nation states like China who already require models to conform to state narratives.
3. More broadly, when training on "the internet" as a whole there is a huge amount of wrong, confused information mixed in.
There's also a meta-point to make here. On a lot of culture war topics, one person's "poisonous information" is another person's "reasonable conclusion."
Im looking forwards to protoscience/unconventional science and perhaps even that what is worthy of the fringe or pseudoscience labels. The debunking there usually fails to adress the topic as it is incredibly hard to spend even a single day reading about something you "know" to be nonsense. Who has time for that?
If you take a hundred thousand such topics the odds they should all be dismissed without looking arent very good.
Apparently, you haven't been on that Internet thingie in the last five years or so... :-)
But I do agree with your point. What's interesting is the increasing number of people who act like there's some clearly objective and knowable truth about a much a larger percentage of topics than there actually is. Outside of mathematics, logic, physics and other hard sciences, the range of topics on which informed, reasonable people can disagree, at least on certain significant aspects, is vast.
That's why even the concept of having some army of "Fact Checkers" always struck me as bizarre and doomed at best, and at worst, a transparent attempt to censor and control public discourse. That more people didn't see even the idea of it as being obviously brittle is concerning.
We can trust altman and elon to weed out the "fakenews". Finally we will get the answer which is the greatest linux distro.
> Outside of mathematics, logic, physics
No need to go outside. There are plenty of Grigori Perelmans with various levels of credibility.
I more often than not use the thumbs up on bad Google AI answers
(but not always! can't find me that easily!)
What about unmarked crosswalks? Does it have to contain the crosswalk in whole or in part? That bit of white stripping is there just on the edge of this image, does that count? There’s a crosswalk in the background does that count? Etc etc.
The answer to all these questions is generally that you shouldn’t be asking. I can almost hear someone saying “You know what we mean.”
> Aaron clearly warns users that Nepenthes is aggressive malware. It's not to be deployed by site owners uncomfortable with trapping AI crawlers and sending them down an "infinite maze" of static files with no exit links, where they "get stuck" and "thrash around" for months, he tells users.
Because a website with lots of links is executable code. And the scrapers totally don't have any checks in them to see if they spent too much time on a single domain. And no data verification ever occurs. Hell, why not go all the way? Just put a big warning telling everyone: "Warning, this is a cyber-nuclear weapon! Do not deploy unless you're a super rad bad dude who totally traps the evil AI robot and wins the day!"
There’s a lot of incentive to figure this out. And they have so much data coming in that they can likely afford to toss out some good data to ensure that they’re tossing out all of the bad.
We can use the same tactics they are using to crawl the web and scrape pages and bypass anti-scraping mechanisms.
And at the end of the daty, they can always look at the responses coming in and make decisions like “95% of users said these responses were wrong, 5% said these responses were right, let’s go with the 95%”. As long as the vast majority of their data is good (and it will be) they have a lot of statistical tools they can use to weed out the poison.
So expert answers are out of scope? Nice, looking forward to those quality data!
If people want to keep playing pretend that their data poisoning efforts are causing real pain to OpenAI, they’re free to do so. I suppose it makes people feel good, and no one’s getting hurt here.
Why do you think we are stupid? We work at places developing these models and have a peek into how they're built...
Ask a dozen experts to decide what that boat needs to fit your need.
That is the specification problem, add on the frame problem and it becomes intractable.
Add in domain specific terms and conflicts and it becomes even more difficult.
Any nontrivial semantic properties, those without a clear T/F are undecidable.
OpenAI with have to do what they can, but it is not trivial or solvable.
It doesn't matter how smart they are, generalized solutions are hard.
The companies whose datasets you're "poisoning" absolutely know about the attempts to poison data. All the ideas I've seen linked on this side so far about how they're going to totally defeat the AI companies' models sound like a mixture of wishful thinking and narcissism.
Also I don't get the narcissism part. Would it be petty to poison a website only when looked by a spider? Yes, but I would also be that petty if some big company doesn't respect the boundaries I'm setting with my robots.txt on my 1-viewer cat photo blog.
The idea that a couple bad faith actions can destroy a 100 billion dollar company, is the extraordinary claim that requires extraordinary evidence.
Sure, bad actors can do a little damage. Just like bad actors can do DDoS attempts against Google. And that will cause a little damage. But mostly Google wins. Same thing applies to these AI companies.
> Also I don't get the narcissism part
The narcissism is the idea that your tiny website is going to destroy a 100 billion dollar company. It won't. They'll figure it out.
Obviously a singular poisoned site will never make a difference in a dataset of billions and billions of tokens, much less destroy a 100bn company. That's a straw man, and I think people arguing about poisoning acknowledge that perfectly. But I'd argue they can eventually manage to at least do some little damage mostly for the lulz, while avoiding scraping.
Google is full of SEO manipulators and even when they recognize the problem and try to fix it, searching today is a mess because of that. Main difference and challenge in poisoning LLMs would be coordination between different actors, as there is no direct aligning incentive to poisoning except (arguably) global justified pettiness, unlike black hat SEO players that have the incentive to be the first result to certain query.
As LLMs become commonplace eventually new incentives may appear (i.e. an LLM showing a brand before others), and then, it could become a much bigger problem akin to Google's.
tl;dr: I wouldn't be so dismissive of what adversaries can manage to do with enough motivation.
Remember Dogecoin or Gamestop; the lulz-oriented meme outbursts had a real impact.
Equally, a particular way to gaslight LLM scrapers may become popular and widespread without any enforcement.
- Significant fraction of all developer time (30%+ just on my team?) - Huge increase to the complexity of the system - Large accumulated performance cost over time
Obviously it's not a 1-to-1 analogy but if we didn't have to worry about this sort of prodding we would be able to do a lot more with our time. Point being that it's probably closer to a 2x cost factor than it is to a 1% increase.
That said, I suspect the signal is very weak even today and probably not too useful except for learning about human stylistic preferences.
I would assume to use data they would have to filter it a lot and correlate between many users.
You can detect if the user is the real one and trust their other chats "a bit more".
If we had to grade truthiness of data sources - our sight or other main senses would probably be #1. Some gossip we heard from a 6 year old is near the bottom.
We know how to grade these data sources based on longitudinal experience and they are graded on multiple axes. For instance Angela is wrong about most facts but always right about matters of the heart.
You don't need honest user feedback because you could judge any message part of a conversation using hindsight.
Just ask a LLM to judge if a response is useful, while seeing what messages come after it. The judge model has privileged information. Maybe 5 messages later it turns out what the LLM replied was not a good idea.
You can also use related conversations by the same user. The idea is to extend context so you ca judge better. Sometimes the user tests the llm ideas in the real world and comes back with feedback, that is real world testing, something R1 can't do.
Tesla uses the same method to flag the seconds before a surprising event, it works because it has hindsight. It uses the environment to learn what was important.
Structural because while a human being can be the judge of an LLM output, a computer (or another LLM) cannot.
No amount of error correction is enough to turn an LLM output into a reliable input to another (possible dumb) computer system. Worse: each time that output is processed the error increases and when the final output is shown to an user, the error might have been amplified beyond human recovery (or recognition) capacity.
Think about this: one user sends Amazon support an email asking to refund for a stolen item.
Can this email be processed do feed an automatic refund pipeline system? If the answer is no and you need a human to verify the result, then we have one reason why hallucinations matter.
And there are the cases where a user verification is not even possible, like:
- what is the procedure to perform CPR in a person above 80 years old?
The user can’t recover errors in the output generated by an LLM here, because she doesn’t know the correct answer.
That being the case, you cannot build a search engine out of an LLM here. Hence hallucinations matters very much is this case too.
Not even in the case of simple information extraction from a text you can ignore hallucinations, because if you provide a list of names and ask for all those starting with “A” you cannot be certain that all names output will actually start with “A” and most certainly cannot be certain that all correct names will be in the output. And this behavior cannot (as of today) be corrected on the LLM we have right now (the first part yes, the second part no).
So, LLM with hallucinations are a very powerful tool, but not the tools they are being sold as.
1) Which search engine comes with infallible information? 2) Where are LLMs being sold as something different?
2) chatbots, perplexity search engine, summarization chrome extensions, RAG tools. Those all built over the idea that hallucination is a quirk, a little cog in the machine, a minor inconvenience to be dutifully noted (for legal reasons) but conveniently underestimated.
Most things in life don’t have a compiler that will error on a inexistent python package.
No they don't. The human is meant to read, analyze and summarize the output same as they would for search results
>> Today's date is Tuesday, January 28, 2025.
> No, you're wrong, today's date is actually Wednesday the 29th.
>> My mistake. Yes, today's date is Wednesday, January 29th, 2025.
Three months later in April when this tagged data is used to train the next iteration, the AI can successfully learn that today's date is actually January 29th.
At best, you can only try to layer in contextual info like this as metadata during inference, akin to how other prompting layers exist.
Even then, what up-to-date information should present for every round-trip is a matter of opinion and use-case.
This is mostly irrelevant no? A binary digit by definition cannot encode more than 2 dates; so therefore we devise a more elaborate system (of using multiple digits).
This is very similar to NYT's lawsuit against OpenAI where in addition to other claims, they claimed OpenAI maintainted a DB of NYT articles that they would directly grab from for a response. It's seems very feasible to maintain a DB or system of looking up real-time values like dates / weather.
That makes it a bit harder, but still, spitting out the wrong date just seems like a plain old time-zone bug.
Such an ingenious attack, surely none of these companies ever considered it.
an example that just occurred to me - if you asked it to generate an image of a mushroom that is safe to eat in your area, how would you tell it it was wrong? "oh, they never got back to me, I'll generate this image for others as well!"
For your example, what if you want to show what such a mushroom looks like to a friend? What if you want to use it on a website?
While it's possible Wikipedia is wrong, the model always agrees with me when I correct it, so that isn't going to help with training either.
Of course for anything high stakes relying on a model probably isn't a great idea.
If I say "no, you hallucinated basically the entire content of the response", then maybe a newer training set derived from that could train on the specific fact that that specific hallucinated response is hallucinated. This seems to be of dubious value in a training set.
Provide a cost function (vs labels) and have it argue itself to greatness as measured by that cost function?
I believe that's what GP meant by "respond", not telling GPT they were wrong.
We've been saying this "we get valuable data" thing since the 2010s [1].
When will our collective Netflix thumbs ups give us artificial super-intelligence?
[1] Especially to investors. They love that line.
They know exactly what makes you stay, and what makes you leave.
I would not be surprised if in the near future movies and series are modifed on the fly to ensure users stay glued to their screens.
In the distant future this might be done on a per user level.
I can stop the AI takeover?