White House releases health report written by LLM, with hallucinated citations | Hacker News Reader

White House releases health report written by LLM, with hallucinated citations

nytimes.com

198 pointsriffraff53 comments

I have this pet theory that this wave of AI won't end with amazing productivity gains.

I think it'll end in mass confusion. Dumbing down of the leadership and elites that now can cosplay as smart and eloquent.

countries that don't embrace AI will have a massive edge over other countries because their population will be smarter and more capable.

It’s all about trust. Which sources of journalism are trustworthy? Is the government administration trustworthy?

You can’t trust an administration that would release a report that cites sources that do not exist. They are either so incompetent that they cannot perform the most basic fact checking possible, or they think we are idiots who can be easily misled.

> they think we are idiots who can be easily misled.

... I mean... Do you want a list of times that the American public has been easily misled?

The error was retracted. This is a clear example for an industry or branch for ensuring accuracy. We could call it the ministry of truth.

It makes me remember a story I'd read, about a certain "Comrade Ogilvy", who recently died a hero whilst serving. Theres no real records of this guy, but a few lines text and a couple of faked photographs seemed easy enough to do.

I don’t think eloquence or the appearance of intelligence is all that valuable for political leadership these days.

Are the countries with limited Internet better off in regards to the issues that Internet has created? Maybe, but they are also missing out on the progress of the society because despite the issues it also created a lot of opportunities and changed how we do things in general.

I have entertained a similar notion when imagining the direction industries that make software might go from here. There's a possible future where some sizeable percentage of companies goes extra hard in the direction of "LLMs in, costly programmers out", and end up getting completely smashed when the LLM systems fall apart.

There might even be a couple of months of "gains" as (pointless) metrics go up, and then we might see a proper crash when stuff stops working. Especially for software which businesses rely on, surely there must be a point where they'll say enough of this crap?

Maybe not, too. Capitalism is a very surprising system, capable of absorbing shocks and morphing itself seemingly endlessly.

We had mass confusion and politician hallucination well before LLMs.

I'm not really sure anything has changed.

Tolerance for error of an average human is higher than one expects. It’s a cultural problem that’s getting more prominent in the West, because people genuinely start looking down on education. I have over-educated family members who are firm set on “my children don’t have to study hard, I’m sure they’ll smart enough to make big money at the end” idea.

Or maybe they will come on-line with AI after it's been sufficiently improved, and not have wasted time and resources on the early versions. Kind of like countries that never had an installed copper landline phone infrastructure, and then leap-frogged the US by going directly to a mobile infrastructure.

https://news.ycombinator.com/item?id=42707238 Nobody cares.

I assure you Mrs. Buttle, the Ministry is very scrupulous about following up and eradicating any error.

If you have any complaints which you'd like to make, I'd be more than happy to send you the appropriate forms.

Braaaaazil......

[where my mind goes on every news from this admin]

Someone should put in an FOIA request for the prompt. Or get a congressional committee to ask for it.

Lol nobody is keeping records in this admin.

The only records we have are in Signal chats and we need people to screenshot those.

It would also be helpful to know which models were consulted, and whether they are certified for use in guiding national healthcare policy.

In a representative democracy, elected leaders need to be able to use their judgement to decide what tools they and their lieutenants should use to come up with and implement the policies in service of what the voters' want.

Who is supposed to do this certification? How is it supposed to be binding?

Leaders are also accountable and should be transparent about how policy is implemented, in the absence of any overriding security considerations.

Sure, that's what elections (and the courts) are for.

Leaders can decide to be even more accountable and transparent, if they think that'll increase their chances at the ballot box (or if they are just nice people).

Your idea seems to be outright incorrect. We need transparency to stop bad things -> Bad things do not help at the ballot box -> We won't be given transparency through the ballot box (unless there are no bad things in which case we didn't really need it). In other words, if what you say is true (and it seems it unfortunately is), we won't ever get transparency and bad things will happen unchecked and unseen.

Instead of elections I think transparency should be determined by rule of law so everyone has to participate, even those that would like to hide their bad actions.

If people want 'bad' leaders, they can elect 'bad' leaders. That's democracy for you. ('Bad' in the sense of people you and me disapprove of.)

It's obvious to anyone that Trump is intransparent at best, and most likely corrupt. People still voted for him. And Trump isn't even hiding his bad actions all that much. I mean, he might be hiding some details, but it's common knowledge that he is doing 'bad actions'.

Technically, this is title editorializing, since WH denied using AI… but they are probably lying.

And when you're lying on that scale, automating it with AI is likely an actual productivity gain.

First the bogus tariff equation, now this. Certainly not the last.

Jokes aside, it's really sad to see seemingly no competent people not wanting to work with this administration.

Your causality is wrong. This adminstration has no interest in working with competent people. They're fired huge numbers of them, gone out of their way to make life hell for those left so they quit, and are moving faster than ever imagined possible to drive competent people in non-government jobs away from this country - by deporting them, taking away their visas, attacking them, and more.

The causality is bidirectional.

I disagree. The first Trump administration showed that not only were some competent people interested in working in the Trump cabinet, plenty of other competent people were content to keep doing their non-political jobs well for the government, running things, fighting for normal people and what they right, pushing back against overreach.

The second Trump administration took aggressive measures against all competent people - selecting a horrifying cabinet with literally no one qualified for the role they fill and working as fast as possible to fire as many as possible, even through illegal means later rejected by the courts, and punishing any who remain. It's a one-direction causation.

This administration is a haven for charalatans and grifters. It's no surprise that they don't have competent people. The goals of competent people are diametrically opposed to what this government wants to do.

No. There's still lots of overlap in goals.

For example, most competent people I know didn't want a nuclear war to start yesterday. Lo and behold, the administration also did not start nuclear war yesterday.

While there are a lot of incompetent idiots in this admin, don't forget that a lot of them are grifters who are only trying to get rich/powerful while they can.

I don't like doing the Hitler comparison, but the similarities are definitely there. A lot of the Nazis thought Hitler was a "useful idiot" that they could use and then get rid of. Trump is very similar.

I would imagine this is true of many other dictators and authoritarians over time too: Putin, Kim Jong Un, etc. If you are looking for non-Hitler comparisons.

https://archive.ph/FBVff

These people even struggle to do a bad job well enough.

Next time, they'll just add "don't include citations" in the prompt. Problem solved.

Its all a big joke to these people, like asking about medical advice from Groks "not a doctor".

Can we all agree to not cite any study that a) doesn't exist, b) cannot be reproduced, or c) includes fake data?

For bonus points, the participants of the b) and c) options should be forced to pay back whoever funded their research.

What do you mean by 'cannot be reproduced'?

If I do a simple study where I flip a coin a million times and write down the result, you are unlikely to be able to reproduce the same result.

I'm pretty sure if we both flipped a million coins, our results would be very similar.

I think you and GP are using different metrics or definitions of "result".

Well, I'm mostly just saying that you have to be careful with your definitions. And you can't just randomly demand refunds from people.

In this context we would be talking about reproducing the experimental conditions and method. Obviously it is unlikely that someone else would get the same sequence you did.

Basically, the whole thing is a bit more complicated.

However, it's useful to require pre-registration and sharing of data etc for studies you plan to fund.

Ideally, you give researchers an in-principle approval, but then hold the funds in escrow and only disburse them after they published their data etc.

(Financial markets can provide the bridge financing between in-principle approval and actual disbursal.)

There's many journals that have open data requirements, but more often than not they are flouted. The above suggestion would give them real teeth.

In case you're not trolling: to reproduce a paper means to achieve the reported result within an acceptable margin through independent replication of the experiment. Since nobody is referring to a paper which reports one million coin flips, reproduction is not in question.

This is a great example of why AI is the perfect tool for politicians who want to do something, then try to find some rational for it. AI will always give you the answer you want, even if it has to make it up.

At this point we can fully replace talking heads with AI. Let them generate reports, write fake citations, review each other’s work, sign it, worship it – all in imitation of what we once called a state. A huge win for humanity.

Reminds me of this “AI study” on climate change

https://factcheck.afp.com/doc.afp.com.39798G2

It is not the “minor citation errors” what are the most worrying, but rather the fact that LLMs aim to please you, even if you really believe your prompt was neutral (which I doubt when it comes to Kennedy and vaccines for example).

The goal here is to normalize the US government doing random shit, in order to reduce transparency. Average US citizen won't be able to tell if another political scandal is just Trump being Trump, or actual power grab.

this is obviously horrific, but they are succeeding at their broader goal - making the entire US Federal apparatus untrustworthy and operated completely by selfish idiots who do whatever they want at any given time, with the only binding force the idiotic whims of the president. (almost all of - AOC and Bernie and a few others are notable exceptions) congress has already completely conceded this fact and continues to do so by not impeaching the president.

to me, it is extremely hard to imagine how this deliberate destruction can be undone in less than decades.

Not really sure what’s better - hallucinating LLM or hallucinating politicians…

Think the title here is a little misleading. The citations do indeed appear to be hallucinations but it is not known if the report was written partially or entirely with an LLM. I would have no trouble believing that it was but at this point there isn't even a leak at MAHA claiming that it is.

Why would you write a report and ask an LLM to make up citations once done?

Anthropic had a slip-up like this recently with a legal filing (totally unrelated to the report, just a similar example of bad citations). After being challenged by a judge, they said:

> A Latham & Watkins associate located that article as potential additional support for Ms. Chen’s testimony using a Google search. The article exists [...]

> [...] I asked Claude.ai to provide a properly formatted legal citation for that source using the link to the correct article. Unfortunately, although providing the correct publication title, publication year, and link to the provided source, the returned citation included an inaccurate title and incorrect authors.

Anthropic could be lying, but apparently the link is indeed correct, so the account seems plausible.

However, the current situation is less understandable. The article says that "some correctly cited papers were inaccurately summarized", which suggests that AI either was used for the report itself, or at least was told to add citations without the author's input, which would be far more irresponsible than what Anthropic did. The apparently completely hallucinated "paper on direct-to-consumer advertising of prescription drugs" also doesn't look good.

The article also mentions that "[a]n early copy of the report shared with reporters did not include citations", which does support the theory that citations were added after the fact (whether or not AI was also used for the report itself).

Source for Anthropic testimony: https://storage.courtlistener.com/recap/gov.uscourts.cand.43...

I'm not saying they didn't. It's just that the title here says "WH releases health report written by LLM" but the article it links to does not claim that. The headline from the article is about the fake citations.

Also, I could imagine that the report, whether drafted by human or not, could have been pasted into an LLM with a prompt like "make this sound more authoritative" and the LLM dutifully added some "citations" because, what's more authoritative looking than citations?

Reply on news.ycombinator.com