AI summaries turn real news into nonsense, BBC finds | Hacker News Reader

AI summaries turn real news into nonsense, BBC finds

theregister.com

23 pointsisaacfrond7 comments

The link to the BBC research document is https://www.bbc.co.uk/aboutthebbc/documents/bbc-research-int...

"In December 2024, the BBC carried out research into the accuracy of four prominent AI assistants that can search the internet – OpenAI’s ChatGPT; Microsoft’s Copilot; Google’s Gemini; and Perplexity. We did this by reviewing responses from the AI assistants to 100 questions about the news, asking AI assistants to use BBC News sources where possible. Ordinarily the BBC ‘blocks’ these AI assistants from accessing the BBC’s websites. These blocks were lifted for the duration of the research and have since been reinstated. AI answers were reviewed by BBC journalists, all experts in the question topics. Journalists rated each AI answer against seven criteria – (i) accuracy; (ii) attribution of sources; (iii) impartiality; (iv) distinguishing opinions from facts; (v) editorialisation (inserting comments and descriptions not backed by the facts presented in the source); (vi) context; (vii) the representation of BBC content in the response. For each of these criteria, journalists could rate each response as having no issues; some issues; significant issues or don’t know."

Associated blog post: https://www.bbc.co.uk/mediacentre/2025/articles/how-distorti...

I checked the original research by BBC but the underlying data was not published - no way to replicate the study, and no way to check its validity.

They didn’t provide the answers models provided, so we can’t even say if the reviews of the answers were correct. And since the experiment relied on BBC giving timed access to their archive to the models, we couldn’t even type in the same prompts into the models to see the model outputs.

I think what little we do know about the research also exposes flaws. For example, although the BBC prompted the AI to only use BBC sources, it admits that other sources were used to compile responses. But the BBC only judged the accuracy of the answers against its own reporting. So, the AI summaries might well have contained info that wasn't in the BBC reports, but it may have gleaned those from other sources.

It would be interesting to see the experiment repeated, but with the BBC feeding the AI PDFs/text of the initial reports. I suspect it would be much more accurate.

Witty isn't it. It seems to have all the hallmarks of the same "total cost of ownership" study of Linux that Microsoft issued during Ballmer's years...

Have a look at the appendix of https://www.bbc.co.uk/aboutthebbc/documents/bbc-research-int...

There you will find at least some examples of questions and answers. But it is true, the whole raw dataset has not been published, yet.

Ah the AI naysayers. Down with all kings but King Ludd!

Pretty good though, for consumer-grade, off-the-shelf AI.

Reply on news.ycombinator.com