AI Is a Terrifying Purveyor of Bullshit. Next Up: Fake Science | Hacker News Reader

AI Is a Terrifying Purveyor of Bullshit. Next Up: Fake Science

lastwordonnothing.com

30 pointsblueridge23 comments

The bullshitting in "writing" is one thing, but these things are indeed scary:

"The researchers showed that GTP-4 ADA “created a seemingly authentic database,” that supported the conclusion that one eye treatment was superior to another. In other words, it had created a fake dataset to support a preordained conclusion. This experiment raises the threat that large language models like GTP-4 ADA could be used to “fabricate data sets specifically designed to quickly produce false scientific evidence.” Which means that AI could be used to produce fake data to support whatever conclusion or product you want to promote."

Note also that faking data is not only about science. I'm pretty sure we're going to see fake datasets for fake market analyses, fake polling data for fake political campaigns, fake data from fake data breaches, etc., etc.

This made me think of Dirk Gently's Holistic Detective Agency, Douglas Adams's 1987 novel, where there's an AI company that takes military and political decisions and comes up with post hoc rationalizations for them.

The postmodernists (so far as I understand them) say that we are post truth - that all claims of truth are actually assertions of power. I always thought that was bunk. Now it looks more like prophecy.

How does society function when nobody can know if anything they think they know is actually the truth? And not just in the "deep epistemological thoughts" way. Even in the way that we used to be able to know things, with less than total epistemic certainty, but still with sufficient certainty to usefully be able to use information... we can't do even that any more.

I've alluded to this before, by I had an unfinished sci-fi novel in the decades past where this was the backdrop, that at one point faking everything that comes over the wire became trivially easy. My prediction -- and I have no idea how true it would be -- is that once this became really prevalent and media outlets just used this to keep the 24/7 news cycle going, election results were faked, dead celebrities were prancing around commercials, seamless insertion of just about anything into your favorite old movies, that we would collectively have a Reality Crisis. Imagine if some percentage of people who talked about the Mandela Effect were right and that number kept growing. Imagine if we could hide a Tuskegee or two for a little while longer before it came out.

You would get a huge amount of digital distrust, and some percentage of people would retreat to "If it wasn't printed in a book before 2020 ..." as a standard. Wild paranoia would emerge and others would retreat into a kind of indifference because "Who knows?" Previous conspiracy theories, suck as a faked moon landing, might become plausible just due to how many other wild fakes existed. How would you know who really won an election?

The good thing about the Internet is that no one knows you're a ~~dog~~ AI.

Every website will require your phone number and captchas to verify your humanity, but somehow none of them will do a good job keeping their content authenticly human.

“Every record has been destroyed or falsified, every book rewritten, every picture has been repainted, every statue and street building has been renamed, every date has been altered. And the process is continuing day by day and minute by minute. History has stopped. Nothing exists except an endless present in which the Party is always right.” ― George Orwell, 1984

“Being in a minority, even in a minority of one, did not make you mad. There was truth and there was untruth, and if you clung to the truth even against the whole world, you were not mad.” ― George Orwell, 1984

So long as you need calories to metabolize in your brain to think about anything, we are in a truth world.

> How does society function when nobody can know if anything they think they know is actually the truth?

It doesn't.

> Which means that AI could be used to produce fake data to support...

There is an interesting dichotomy here. This case is indeed not ideal, but many developers would be happy if an AI could create the fake data for testing that is representative of real data.

It's not the tool, but how it is used, or by whom it is used?

> This case is indeed not ideal, but many developers would be happy if an AI could create the fake data for testing that is representative of real data.

I agree: synthetic data has many well-intentional use cases. It may also remedy privacy issues, among other things. And you can easily enough generate synthetic data without AI, but the issue, I think, is that AI makes the generation easy also for laypeople, and I suppose you can prompt LLMs to generate custom "high-quality" fake data that no data sleuthing can detect.

Of course the issue is the user not the tool. But, there is no shortage of assholes in the world. So if something can be used for low-effort, high reward unethical purposes, it’s very likely to happen and is something that should be considered.

but you don't want to hamper the normal user just because of the asshole potential

We're seeing something proposed in the US around banning contributions to RiscV because it is open source and "our enemies" might benefit and avoid tech sanctions. This seems like another bad take to further isolationism in the US

These things are capable of being wrong and malicious in every way that a human can. To me it doesn’t really seem like news. A human is capable of creating a dataset to support a false conclusion, are we really surprised that an AI model can too? Do we need AI to be 100% perfect before people are allowed to start using it?

It seems easy to create a list of everything bad a human can do and then write a corresponding article about how bad it is that an AI can do it too.

1. Not _every_ human is capable of creating a false dataset. 2. No human is capable of creating 1000 false datasets in a day or whatever.

The same reason it's cool that AI can make a recipe for a cake or draw a picture of a boat is the same reason it's bad when it can do bad things. It's not as if people were not capable of making recipes for cakes or drawing pictures of boats. It's just now _anyone_ can rapidly do those thing on demand +/- some efficacy determined by the sophistication of the AI.

Reminds of a quote from The Magnus Archives

> And I could. More than anyone else on Earth, he said, because of my expertise. That knowledge I had gained in defiance of the dark could finally be put to use. I was to create a focus, a black star, a new centerpoint around which a universe of purest darkness could turn. To take dark matter, dark energy, and harness it, bring it forward into a form that could be held, used… worshipped.

> Scientifically, it was nonsense, of course. Dark energy and the like don’t work like that, not even remotely. But that wasn’t important. What mattered was that it felt like science, and that was all I needed; to do my work, to create the black star, would need a parody, an aping mockery of science. But it would also need the deepest of darknesses.

I think it can only be a good thing if AI breaks all our bullshit detectors and forces us to actually establish roots of trust and make science independently verifiable.

> and forces us to actually establish roots of trust

How would that even be possible in a world where literally nothing can be trusted?

Algorithmic Idiocy, just because it's a algorithm doesn't make it right.

The point on hallucinations is particular on point: when a human makes up stuff it's lying, when a machine does it, it's marketed as hallucinations.

when a human imagines a reality that isn't true, but they don't know it isn't true, it's often called hallucination, that's where they got the term from

Much of what is called 'science' is not, in fact, 'science'.

Reproducibility becomes even more important. We must appreciate redoing scientific experiments, even if it's at the cost of novelty.

Aschwanden is a fairly serious journalist, but the green ChatGPT 3.5 icon in the Nate Silver example isn’t doing her any favors.

> It will become increasingly difficult to determine what is true.

welcome to the party, pal

Reply on news.ycombinator.com