How to use generative AI for historical research: Four real-world case studies

For somebody extolling the uses of generative AI, they missed several glaringly factually incorrect errors in the chart generation from the cigarette advertisement in the very first visualization.

This is actually my biggest fear – that people are going to assume the infallibility of these agents and never fact check them.

I tried this and… half of what chat got says is a lie. Using it to help your research is dangerous if you trust it at all. I’ve stopped using it completely by now (some of the lies are extremely subtle and hard to catch, some implied background detail it made up that totally skews your perception of a situation)

Not to define how LLMs can be used but for me the best case for History is different. Basically the idea is that since we're training LLMs on available data from a time period, historians from the future will be able to understand nuances about culture in a way better way if they boot up an LLM trained in 2023 than any other way of querying records.

Things like "what did people in 2023 think about X", "what were some 2023 fads", etc. Some of this can be deduced nowadays from historical records but I think this has the potential of allowing way deeper understanding. Also better than asking that from the 2623 LLM which will be polluted by whatever is the current culture then.

Unfortunately it doesn't help us, only the future historians.

Very fascinating. I love hearing perspectives like this. This is more evidence for me that the medium term future will be extremely narrow agents that do one thing and do it well.

> When asked to transcribe and translate those three lines at the bottom, the AI got the general idea, but also made significant errors

> GPT-4 is quite useful for “getting the gist” of written sources in unfamiliar languages.

People don't like to deal with tedious details and issues like accuracy (perhaps separating the good historians from the bad). They seem to wish - and seeing general agreement from fellow dreamers and dismissing tiresome objectors they seem to believe - that somehow misinformation won't be a problem. I don't know how, but somehow, because everyone does it, everyone believes it. As if the zeitgeist of the post-truth era will save them and us from the consequences. As if accuracy (completeness, correctness, consistency) of information is like cooking - for most dishes, overcooked or undercooked isn't great but it's fine.

Information doesn't work that way: Unless you can read the original yourself - and if so, you don't need the GPT to translate - you have no idea if the GPT's 'gist' is correct - any element of it could be wrong, and likely some are. And a small mechanical error can cause devestating misunderstandings, such as a 'not' being attached to the wrong word (and remember different languages have very different rules about word order). If the GPT translates something as 'to our surprise, Jones, not Smith, was the Soviet spy', what does the original really say? Was it Smith? Jones? Maybe they were 'not' surprised?

The true, rational answer is that you have no idea what it says (without some outside information). Information where any element can be inaccurate is useless, and worse, dangerous: believing a falsehood is worse than thinking, 'I don't know'.

Honestly, the use cases presented here are probably cautionary tales about what not to do. Asking questions about the distant past, from which the LLM likely has very little data in its dataset and we have no "ground truth", is just begging for hallucinations, which you can see happening all over in this post.

we urgently need a "by ai" tag for text and images etc. this is one place where maybe even regulation would make sense: software should require all generated content to add a (hidden) "made by ai" tag. any medium which does not allow such meta tags?

Horrible popup on the blog, cant cancel it pn mobile

The issue is always going to be verification and reproducibility with LLMs

"I am starting to think that this sort of automated data visualization may end up being among the best use cases for AI in historical research."

If you actually look at the plots, there are all kinds of horrors.

"We just do it in a targeted way, because it’s time consuming."

Please.