Hacker News Reader

> You cannot permit your employees to use LLMs in this manner and then tell them it's entirely their fault when it makes mistakes, because you gave them permission to use something that will make mistakes 100% without fail.

Yes you can. The same way Wikipedia (or, way back when, a paper encyclopedia) can be used for research but you have to verify everything with other sources because it is known there are errors and deficiencies in such sources. Or using outsourced dev resource (meat-based outsourced devs can be as faulty as an LLM, some would argue sometimes more so) without reviewing their code before implemeting it in production.

Should they also ban them from talking to people as sources of information, because people can be misinformed or actively lie, rather than instead insisting that information found from such sources be sense-checked before use in an article?

Personally I barely touch LLMs at all (at some point this is going to wind up DayJob where they think the tech will make me more efficient…) but if someone is properly using them as a different form of search engine, or to pick out related keywords/phrases that are associated with what they are looking for but they might not have thought of themselves, that would be valid IMO. Using them in these ways is very different from doing a direct copy+paste of the LLM output and calling it a day. There is a difference between using a tool to help with your task and using a tool to be lazy.

> it's company policy not to burn everything to the ground!

The flamethrower example is silly hyperbole IMO, and a bad example anyway because everywhere where potentially dangerous equipment is actually made available for someone's job you will find policies exactly like this. Military use: “we gave them flamethrowers for X and specifically trained them not to deploy them near civilians, the relevant people have been court-martialled and duly punished for the burnign down of that school”. Civilian use: “the use of flamethrowers to initiate controlled land-clearance burns must be properly signed-off before work commences, and the work should only be signed of to be performed by those who have been through the full operation and safety training programs or without an environmental risk assessment”.

> The same way Wikipedia can be used for research

Before LLMs, Wikipedia was the greatest source of disinformation in human history. No journalist should ever have been using it for research. At best it's a fun project for satisfying people's idle curiosity where the truth of what they read doesn't really matter, but if your job is to report factual information, reading Wikipedia is doing a disservice to yourself and your readers. Just like people don't properly verify the BS LLMs fabricate, very few people thoroughly read the citations on Wikipedia, which often involves purchasing books and getting access to research papers. If they did read citations, they would realise that Wikipedia citations are all too frequently unsupported by the actual material they're citing, or in some cases, the cited material establishes the exact opposite. This is to say nothing of cherry-picking sources, of course.

> Should they also ban them from talking to people as sources of information, because people can be misinformed or actively lie, rather than instead insisting that information found from such sources be sense-checked before use in an article?

Statements made by people are attributed to those people to account for this. Rather than saying "Company X's product is the safest product ever made", a journalist says "Company X's CEO claims their product is the safest ever made". People do not do this with LLMs or Wikipedia, rather than attributing it to an understood-to-be-unreliable source they just present it as a factual statement. Also, if the journalist has good reason to believe the quoted statement is false, it is in fact journalistic malpractice to cite the quote wholesale without caveats informing the reader of the evidence that the quoted person is trying to mislead them.

> because everywhere where potentially dangerous equipment is actually made available for someone's job you will find policies exactly like this

Which maybe makes sense when the flamethrowers are a necessary part of the job. Flamethrowers are not necessary for journalism, full stop, so the fault rests with the organization introducing a dangerous tool into the work environment unnecessarily.

> Before LLMs, Wikipedia was the greatest source of disinformation in human history. No journalist should ever have been using it for research.

Look up what a bibliography and how it's used.

> Yes you can. The same way Wikipedia (or, way back when, a paper encyclopedia) can be used for research but you have to verify everything with other sources because it is known there are errors and deficiencies in such sources.

I think that if Wikipedia had no recommendations on good sources for their own articles and did not ever ban sources, companies would not be so sanguine about letting people use Wikipedia. There's an entire internal process associated with evaluating sources, and the expectation when using Wikipedia is that nothing written in an article is going to be sourced from the Daily Mail or Conservapedia, as an example. Also, I do think that there are companies that do have policies against talking to known liars. Given the Wikipedia bans sources and news agencies ban human sources once they've been shown to be unreliable, I don't think it's insane to then have such companies or agencies say that AI shouldn't be used because it's been shown to be unreliable. Obviously there's a balancing act of utility versus accuracy, and Ars has (probably incorrectly) decided that the utility of AI outweighs its inaccuracies.

What is frustrating is that AI cannot have a higher accuracy than the median reporter, given a little more time. AI is trained on all digitizable text, including falsehoods and inaccuracies by laypeople. Humans can look up digitizable text using search engines, too. An AI can't follow up on leads or ask anyone questions. There's no world in which synthesizing available data from digitized sources alone ends up with more accurate data than a human with a search engine and the ability to make a phone call. So allowing LLM use at all is a direct admission that seeking out the "truth" is not an important goal because it could never actually improve accuracy and could only worsen it through hallucinated, probable reporting. It's one thing when companies say that they're committed to truth and then secretly their most important overriding concern is their bottom line - it's quite another thing when a company directly says that the bottom line is their most important concern. Imagine the emperor walking through the parade, nude, saying "So what if I am nude? What are you going to do about it?"

> companies would not be so sanguine about letting people use Wikipedia

Are companies sanguine about using Wikipedia without verification. Maybe some, but they darn well shouldn't be. And I say this as someone who uses Wikipedia for many minor things (though for anything important, I verify elsewhere).

> Also, I do think that there are companies that do have policies against talking to known liars.

No doubt most/all. But such policies will always be caviated with exceptions if the information is properly validated afterwards.

> So allowing LLM use at all is a direct admission that seeking out the "truth" is not an important goal because it could never actually improve accuracy and could only worsen it through hallucinated, probable reporting.

I'm generaly anti-LLM, but this is… ad absurdum.

There is a huge difference between lazily accepting what an LLM spews out, and using that along with other sources for further research. No good reporter will trust a single source away from exceptional circumstances, wether that source is a person or an LLM, and what would be considered “exceptional circumstances” for trusting specific meat-sourced information won't apply for an LLM-sourced summary.

If you can trust Wikipedia as a starting point, you can trust a good LLM as a starting point. Both are offering a summary of what a bunch of people on the internet have written, neither should be trusted as a reliable source.

> I don't think it's insane to then have such companies or agencies say that AI shouldn't be used because it's been shown to be unreliable

If taking an absolutist approach. I would be a little more quakified and say that LLM output should never be used without verification of all details, rather than should not be used at all. It may be the case that this verification makes using LLMs no more efficient than doing research from other sources in the first place, and I suspect that this is the case often, if when using LLMs proper time is given to verifying the output.

The problem is people musunderstanding what an LLM is: a summariser, offering access to a compressed version of its sources. If you are using them as sources rather tham summarisers then you are using them wrongly. Unfortunately, that means a great many people are using them wrongly…

The next sentence after your quoted section:

“Even then, AI output is never treated as an authoritative source. Everything must be verified.”

Any verification process thorough enough to catch all LLM fabrications would take more work than simply not using the LLM in the first place. If anything verifying what an LLM wrote is substantially more difficult than just reading the material it's "summarising", because you need to fully read and comprehend the material and then also keep in mind what the LLM generated to contrast and at that point what the fuck are you even doing?

I believe this policy can never result in a positive outcome. The policy implicitly suggests that verification means taking shortcuts and letting fabrications slip through in the name of "efficiency", with the follow-up sentence existing solely so that Ars won't take accountability for enabling such a policy but instead place the blame entirely on the reporters it told to take shortcuts.

The LLM can find material that it would be hard or time-consuming for you to do.

You still need to verify it, but "find the right things to read in the first place" is often a time intensive process in itself.

(You might, at that point, argue that "what if LLM fails to find a key article/paper/whatever", which I think is both a reasonable worry, and an unreasonable standard to apply. "What if your google search doesn't return it" is an obvious counterpoint, and I don't think you can make a reasonable argument that you journalists should be forced to cross-compare SERPs from Google/Bing/DuckDuckGo/AltaVista or whatever.)

I believe what their point is is that if you give people a "extract-needle-from-haystack" machine and then tell them they have to manually find where in the haystack the needle was, it defeats the purpose of having the machine.

With that said, a good RAG solution would come with metadata to point to where it was sourced from.

> I believe what their point is is that if you give people a "extract-needle-from-haystack" machine and then tell them they have to manually find where in the haystack the needle was, it defeats the purpose of having the machine.

We've got to be careful to not let the perfect be the enemy of the good.

I'm not an LLM enthusiast, but I think you have actually compare it against what the alternative would really be. If you give the journalist a haystack but insufficient time to manually search it properly, they're going to have to take some shortcut. And using an LLM to sort through it and verifying it actually found a needle probably better than randomly sampling documents at random or searching for keywords.

I don't want to come off as an AI-maximalist or whatever, but, I mean, at some point, skill issue, right?

You can use Google to find you results reinforcing your belief that the earth is flat too; but we don't condemn Google as a helpful tool during research.

If you trust whatever the LLM spits out unconditionally, that's sorta on you. But they _can_ be helpful when treated as research assistants, not as oracles.

This is a bogus analogy leaidng to a bogus conclusion.

If something points to the needle in the haytack (saying "this haystack has a needle positioned eighteen centimeters from the top and three left of center"), it's much easier to verify that indeed there is a needle there than it would be to find that needle in the first place.

If an LLM spits out a claim that something happened (citing a certain article), it's less work to read the article and verify the claim than it would be to DISCOVER the article in the first place.

In other words, LLMs can be a time-saving search engine, and the idea that it's just as much work to find+verify information as it is to have the LLM find it and then you verify it is hokum.

when you use the extract-needle-from-haystack machine, verify that it actually extracted a needle.

that's much easier than manually extracting the needle yourself

Another interpretation is if you have multiple haystacks, and the machine tells you which haystack likely has a needle in it. You still need to extract the needle yourself,

> Any verification process thorough enough to catch all LLM fabrications would take more work than simply not using the LLM in the first place

Sometimes you have a weak hunch that may take hours to validate. Putting an LLM to doing the preliminary investigation on that can be fruitful. Particularly if, as if often the case, you don't have a weak hunch, but a small basket of them.

You can prompt LLMs to scan thousands of documents to generate text validating your hunches. In some cases those validated hunches may even be correct.

It's easy to get an LLM to make any argument you like based on whatever data is available. Those arguments are going to be trivially bad if that data is bad.

It's more using LLMs like a metal detector, rather than digging through the entire beach by yourself.

You still need to check the junk you dig up using the metal detector.

Disagree. If I’m I’m a reporter and I’m trawling though a mass data dump - say the Epstein files or Wilileaks or statistics on environmental spills or something, using AI to pull out potential patterns in the data, or find specific references can be useful. Obviously you go and then check the particular citations. This will still save a lot of time.

> I believe this policy can never result in a positive outcome.

I get where you're coming from (I'm learning more and more over time that every sentence or line of code I "trust" an AI with, will eventually come back to bite me), but this is too absolutist. Really, no positive result, ever, in any context? We need more nuanced understanding of this technology than "always good" or "always bad."

If you need accuracy, an LLM is not the tool for that use case. LLMs are for when you need plausibility. There are real use cases for that, but journalism is not one of them.

I didn't say in any context. I'm specifically talking about this policy on journalistic research.

Is this a case of Moral crumple zones? where "responsibility for an action may be misattributed to a human actor who had limited control over the behavior" https://www.researchgate.net/publication/351054898_Moral_Cru...

Related concept: Unaccountability machines [0] where the system (electronic or organizational) mainly exists to make things nobody's fault.

There's Discworld bit [1] that often comes to mind for me, where the protagonist is reading a press-release by a corporate communications monopoly

> The Grand Trunk’s problems were clearly the result of some mysterious spasm in the universe and had nothing to do with greed, arrogance, and willful stupidity. Oh, the Grand Trunk management had made mistakes—oops, “well-intentioned judgments which, with the benefit of hindsight, might regrettably have been, in some respects, in error”—but these had mostly occurred, it appeared, while correcting “fundamental systemic errors” committed by the previous management. No one was sorry for anything, because no living creature had done anything wrong; bad things had happened by spontaneous generation in some weird, chilly, geometric otherworld, and “were to be regretted.”

[0] https://press.uchicago.edu/ucp/books/book/chicago/U/bo252799...

[1] Going Postal by Terry Pratchett

Sidebar I like “moral crumple zone” much more than “moral hazard” just because it conjures up a much clearer picture of the problem it depicts.

> permit

I suspect that it's more like "ordered."

Many companies are now requiring staff to use AI.

Of course, if the employee screws up, then The Secretary Will Disavow Any Knowledge of Your Activities.

I'm not a journalist and just for random things I'm interested in, I have no problem using an LLM to point me in a direction and then directly engage with the source rather than treat any of the LLM output as authoritative. It's easy to do. This is not a flamethrower.

> ...this is equivalent to providing all of your employees with a flamethrower, and then saying they bear all responsibility for the fires they start.

This is essentially the policy of most SWE groups but with PRs merged, though right?

You can / should use AI to accelerate SWE workflows and assist in reviews but if you merge something that is bad or breaks production that is on you.

> "Hey, don't blame us for giving them flamethrowers, it's company policy not to burn everything to the ground!".

Flamethrowers are inherently dangerous to the operator and are ~intended to be used burn things to the ground.

I'm no expert on arms but there is probably a simile with a better fit out there.

The way LLMs are being forced upon the workforce in tech is just as bad, actually, yes.

> Flamethrowers are inherently dangerous to the operator and are ~intended to be used burn things to the ground.

I actually think bringing up this point reinforces the analogy rather than undercutting it. LLMs are ~intended to spread disinformation, eg. Deepseek on 1989, Grok going full Mecha-Hitler, ChatGPT selling out prompts to advertisers. One of the biggest impacts LLMs will have on human society is as a propaganda tool with a reach of billions.

These sentences are directly contradictory. How does a user of Ars' AI tools use summaries in their research and at the same time not use them?

> summarizing background documents

> AI tools must not be used to generate, extract, or summarize material that is then attributed to a named source,

> the author they fired for the fabricated reporting

Didn't one of the magazine's editors share the byline?

Yes but editors don't take the fall, they take the credit.

Everything occurred exactly as predicted.

I have a little bit of a bias here, as I am building forth.news, which is an AI-powered news platform -- but I am also a former journalist.

It's not necessarily contradictory. I see this more like giving your employees cars, but telling them they are responsible if they get into accidents.

All of this is entirely predicated on expectation and responsibility. First, mark something as being AI if it cannot be verified, and verify everything that you can.

Forth is using AI so we can detect and push out stories as quickly as possible, getting breaking news out there as soon as it breaks. Our summaries are AI, but marked as AI. Our underlying source information is right there and cited. We try to be as transparent as possible about the tools we are using, and the tradeoffs.

Every journalist should instinctively and reflexively double check everything, regardless of the source. There's an old maxim, "if your mom tells you she loves you, check it out." Being from an LLM doesn't change that.

They're also allowed to use wikipedia to use research. It has similar sorts of problems.

> If I wasn't clear, I consider this to be part of what makes the policy self-contradictory.

Yeah, it's not. And this is a bad take. Adults can easily do this. That you consider this to be burning everything to the ground says more about you than anything.

>Adults can easily do this.

no they can't, as showcased by the countless of people who don't. This isn't how human beings work. If you give someone a tool that facilitates taking mental shortcuts they will abuse it.

You put a code generation tool or even just stackoverflow in front of programmers, they will mindlessly copy-paste. You put the slop generator in front of journalists, they'll write slop. You put the tasty snack and the healthy snack at equal distance in front of someone, guess what they'll eat.

The vast majority of people will pick laziness and speed over correctness, always have. So we need tools that if anything slow down work and encourage thinking, not the opposite. You want a better programmer you hand them a debugger and not a chatbot because then they'll be forced to think.

I've never seen a summarizing mistake from any modern LLM. What are you even talking about?

LLMs hallucinate when they don't have enough context. Not when they're just cutting down the message in their context.

> LLMs are terrible at accurately summarizing anything.

I think you are perhaps stuck in 2023?

And yet we are discussing this in the context of a reporter having been fired from Ars Technica for publishing an article which included inaccurate LLM-generated summaries in 2026. How come?

https://news.ycombinator.com/item?id=47226608

Maybe you should read the article? :)

What failed was extracting verbatim quotes, not summarizing.

If you want an LLM to do verbatim anything, it has to be a tool call. So I’m not surprised.

> LLMs are terrible at accurately summarizing anything. They very randomly latch on to certain keywords and construct a narrative from them, with the result being something that is plausibly correct but in which the details are incorrect, usually subtly so, or important information is omitted because it wasn't part of the random selection of attention.

I don't know what you've been doing, but the summaries I get from my LLMs have been rather accurate.

And in any event, summaries are just that - summaries.

They don't need to be 100% accurate. Demanding that is unreasonable.

The LLM meeting-summary bot in Teams seems accurate… unless you were in the meeting, and also closely read the summary afterward. It misrepresents what people actually said all the time.

Depends on topic, often what they consider important isn't what is important and details that are essential get out of view. I'm having good success with youtube video, not as much with technical docs.

Yes, search and summarization is where LLMs shine. I use them all the time for that, and much less for code generation. I would say search > summarization > debugging > code gen/image gen

>They don't need to be 100% accurate. Demanding that is unreasonable.

If an intern was routinely making up stuff in the summaries they provided to their bosses, they'd be let go.