Hacker News Reader

Comment by kolinko | Hacker News Reader

I checked the original research by BBC but the underlying data was not published - no way to replicate the study, and no way to check its validity.

They didn’t provide the answers models provided, so we can’t even say if the reviews of the answers were correct. And since the experiment relied on BBC giving timed access to their archive to the models, we couldn’t even type in the same prompts into the models to see the model outputs.

View on HN

I think what little we do know about the research also exposes flaws. For example, although the BBC prompted the AI to only use BBC sources, it admits that other sources were used to compile responses. But the BBC only judged the accuracy of the answers against its own reporting. So, the AI summaries might well have contained info that wasn't in the BBC reports, but it may have gleaned those from other sources.

It would be interesting to see the experiment repeated, but with the BBC feeding the AI PDFs/text of the initial reports. I suspect it would be much more accurate.

Have a look at the appendix of https://www.bbc.co.uk/aboutthebbc/documents/bbc-research-int...

There you will find at least some examples of questions and answers. But it is true, the whole raw dataset has not been published, yet.

Witty isn't it. It seems to have all the hallmarks of the same "total cost of ownership" study of Linux that Microsoft issued during Ballmer's years...

Reply on news.ycombinator.com