benbreen

Gwern and others who have dug into it this far might be interested by this footnote in the Crespo article: "I have tried to lay my hands on the original version of the conversation, as I am sure Simon did, too. I contacted Gabriel Zadunaisky, who, as the article explains, participated in the meeting. He is a professional translator. I asked him for the original version, and he replied on WhatsApp: 'Mr. Crespo: I am very sick. Unfortunately, I am unable to provide you with the information requested.' My hypothesis is that Zadunaisky translated the conversation directly from the recorded version and that this original version has been lost."

My read is that most likely, it was recorded on an old school reel-to-reel tape recorder. It's entirely possible that the tapes are still sitting on a shelf somewhere in Argentina, though the chances of actually tracking them down are pretty low. I worked with some reel-to-reel tapes that Alan Ginsberg made (now held at Stanford) in the mid-60s (including one where he is talking to Bob Dylan!) and they held up pretty well. Had to use audio editing software to remove tape hiss, but they were not as badly preserved as I expected.

I've been developing a more elaborate variation on the "chat with a pdf" idea for my own use as a researcher. It's mostly designed for a historian's workflow but it works pretty well for science and engineering papers too. Currently Flash 2.0 is the default but you can select other models to use to analyze pdfs and other text through various "lenses" ranging from a simple summary to text highlighting to extracting organized data as a .csv file:

https://source-lens.vercel.app

(Note: this is not at all a production ready app, it's just something I've been making for myself, though I'm also now sharing it with my students to see how they use it. If anyone reads this and is interested in collaborating, let me know).

Such a confusing comment, because when I enter the text from case study #1 into Deepl, it's very clearly much worse than what Claude or GPT4o can come up with (the first few lines from Deepl are: "With his expositions to all the Tables, particularly of the quality of the countries, et of the most notable things, to be found in them. Which Tables, can be, and t are taught to reduce' together" and so on).

Likewise with using Google translate on both case studies #1 and #2 - the results are self-evidently far worse. In both cases there were multiple errors in each line and in case study #2 it was entirely unable to transcribe or translate the title line. If you see this, please email me at bebreen [at] ucsc dot edu to share the better results you are seeing because I genuinely am interested and open to using alternative tools - I just am not seeing what you are seeing, apparently.

In terms of typos not changing the meaning, yes naturally a real human needs to double check absolutely everything if it's being used in research. We agree on that - the point is simply that this significantly speeds up the initial research process, not that it replaces the expertise necessary to, for instance, double check that a name or year is transcribed correctly. A huge amount of historical research is simply about skimming through documents looking for relevent info to zero in on - this is where LLMs can really help.

I agree, I probably should've gone into more detail on the actual case studies and implications. I may write this up as a more academic article at some point so I have space to do that.

To your point about OCR: I think you'll find that the existing OCR tools will not know where to begin with the 18th century Mexican medical text in the second case study. If you can find one that is able to transcribe that lettering, please do let me know because it would be incredibly useful.

Speaking entirely for myself here, a pretty significant part of what professional historians do is to take a ton of photos of hard-to-read archival documents, then slowly puzzle them out after the fact - not by using any OCR tool (because none of them that I'm aware of are good enough to deal with difficult paleography) but the old fashioned way, by printing them out, finding individual letters or words that are readable, and then going from there. It's tedious work and it requires at least a few days of training to get the hang of.

If anyone wants to get a sense of what this paleography actually looks like, this is something I wrote about back in 2013 when I was in grad school - https://resobscura.blogspot.com/2013/07/why-does-s-look-like...

For those looking for a specific example of an intermediate-difficulty level manuscript in English, that post shows a manuscript of the John Donne poem "A Triple Fool" which gives a sense of a typical 17th century paleography challenge that GPT-4o is able to transcribe (and which, as far as I know, OCR tools can't handle - though please correct me if I'm wrong). The "Sea surgeon" manuscript below it is what I would consider advanced-intermediate and is around the point where GPT-4o, and probably most PhD students in history, gets completely lost.

re: basically perfect, the errors I see are entirely typos which don't change the meaning (descritto instead of descritta, and the like). So yes, not perfect, but not anything which would impact a historical researcher. In terms of existing tools for translation, the state of the art that I was aware of before LLMs is Google Translate, and I think anyone who tries both on the same text can see which works better there.

re: "irrelevant books," there's really no way to make an objective statement about what's relevant and what's not until you actually read something rather than an AI summary. For that reason, in my own work, this is very much about augmenting rather than replacing human labor. The main work begins after this sort of LLM-augmented research. It isn't replaced by it in any way.

Thank you! Have been a big fan of your writing on LLMs over the past couple years. One thing I have been encouraged by over this period is that there are some interesting interdisciplinary conversations starting to happen. Ethan Mollick has been doing a good job as a bridge between people working in different academic fields, IMO.

Author here, I agree that my read may not be correct either. It’s tough to make out. Although keep in mind that “ph” is used in Latin and Greek (or at least transliterations of Greek into the Roman alphabet) so in an early modern medical context (I.e. one in which it is assumed the reader knows Latin, regardless of the language being used) “ph” is still a plausible start to a word. Early modern spelling in general is famously variable - common to see an author spell the same word two different ways in the same text.

Author here, I had the same question and looked into it. The author of that comment seems to be onto something because hygrine is indeed found in nightshades as well as in coca. Interesting.

Reading this now and really enjoying it. Here's a brief review from Stewart Brand: https://twitter.com/stewartbrand/status/1800941614287946003

Same. I emailed him about whether he'd ever met Margaret Mead, John C. Lilly, or Gregory Bateson in the 1960s while researching my book. I got this reply within hours:

"Afraid I never met any of those you mention, though I’ve followed their work for many years.

I’ve never been close to intellectual elite circles, including people I very much admire."

The time stamp for my email was Tuesday, Nov 26, 2019 at 2:19 PM. It was answered by chomsky@mit.edu at Tuesday, Nov 26, 2019 at 9:29 PM. Pretty remarkable.

Author here - I love Dune and mostly just used the phrase for the title. But I did write this about Herbert and the drug/spice trade which might be of interest: https://lareviewofbooks.org/article/pharmacology/

He's in there! In fact I decided to cut several pages on him, since a lot has already been written. Michael Pollan's How to Change Your Mind, among other books, has a lot of detail that I couldn't add much to.

Hubbard was important, to an extent, but one of the things I'm trying to do is draw out the larger social and intellectual circle that surrounded the more famous names. So to just to take on example, Betty Eisner was (I would argue) significantly more important than Hubbard both in developing the concept of set and setting and in developing psychedelic therapy in general.

She actually released a memoir online before her death, which is fascinating reading and has a lot of quotes from original letters (some of which I consulted at Stanford, which has her archive): https://erowid.org/culture/characters/eisner_betty/remembran...

Yes, Canada (and especially Saskatchewan) is very important in psychedelic history. I didn't look into it much personally because it's one part of the story that's been documented very well already via the work of Ericka Dyck, which is excellent. Specifically her book Psychedelic Psychiatry: LSD from Clinic to Campus (Johns Hopkins, 2008) but also other articles by her students and collaborators.

I wouldn't recommend my process to anyone, to be honest! I probably wrote a total of around 200k words over a four year period, of which I ended up cutting around 110k. I find it to be very true that you don't know what you actually want to say until you start writing things that aren't what you want to say. Then it's an iterative process of critiquing, rethinking, and starting over. I'm sure some people are able to start with a clear outline and then just plow through to the end, but I'm not one of them.

In terms of research, I used to keep all my photographs of archival documents, PDFs of sources, etc in DEVONThink, but I switched over to using the standard Photos app on Macs. It has automatic OCR now so I'm able to search text that appears in photographs quite easily. I did a lot of oral history interviews to supplement the archival research. This book wouldn't have been possible up until recently because it's only in the past decade that so many historical archives have been digitizing their collections. I was able to visit the key archives in person, but with others, archivists were nice enough to send me scans of key documents. Super grateful to them.

Author of the book under review here. AMA!

One thing I thought would be helpful is to link to the YouTube video described in the opening paragraph: https://www.youtube.com/watch?v=UMF-cyHAaSs

It's from a 1957 CBS documentary series called "Focus on Sanity" that featured interviews with Aldous Huxley and Gerald Heard, among others. I found it fascinating and my questions about it were actually one of the motive forces for why I wrote the book.

I believe the recording was first brought to public attention by Don Lattin, whose books The Harvard Psychedelic Club (2010) and Distilled Spirits (2012) are both great.

Our toddler is now asleep so I am also here to do a brief AMA about the book and psychedelic science in the 20th century if anyone has questions. :)

Forgetting context is a problem, for sure. One thing I've found that works fairly well is to include a request for a "status bar" in your prompt. I.e. you ask it to remind itself with each response 1) who it is pretending to be 2) what the date is 3) what the setting is 4) what is in their NPCs "inventory" (which it intuitively understands because LLMs seem to have a natural affinity with MUDs). You can even have it track its mood and variables like weather.

As the context windows of Claude/GPT-4 etc increase, I think this will be less of an issue, but for now it's a pretty effective workaround.

Here's an example of the prompts I'm using (from an activity I just did with my world history class): https://docs.google.com/document/d/1sLRsUVJ_KSPtjrO83ko2MSFf...

And my writeup of an earlier version: https://resobscura.substack.com/p/simulating-history-with-ch...

Not factually wrong at all - the dosages, ingredients, diagnosis and even the language are all strikingly accurate. Naturally, the "fake" 1680s doctor didn't write the same prescription as the real one (Sydenham). But a different real life doctor would've disagreed with Sydenham, too. In other words, if you had 100 physicians in the 1680s write out a prescription for hysteria and "hypochondriacal passion," this would be (IMO) indistinguishable from the real ones. What's different is that this is interactive, so you can change elements at will. Again, this isn't reflective of historical fact. It also isn't simply fan fiction. This text isn't an end point, but a starting point for jumpstarting your own thinking about the affordances of a past world.

In other words, I think this is a new method for thinking creatively about history -- one among many that already existed, like historical fiction, historical re-enactment, various forms of experiential learning like debates and roleplaying, etc. But it's cool that there's a new one!

I think we agree, just define terms differently -- video games are art! In other words, gamers are consumers of artwork, and that consumer demand for a new kind of art drove demand for the hardware to go with it. (Naturally that wasn't the only source of demand - engineering and research applications were there from the beginning too).

Edit: this discussion is interesting because I have always just taken it for granted that video games are a form of art. Clearly others don't see it that way, which is fair! Nevertheless, I think a strong case can be made: https://en.wikipedia.org/wiki/Video_games_as_an_art_form

Author here - thank you for this. I definitely don't claim to be an expert on the history of 3d graphics, and you clearly know a lot more than me about the detailed history of NVIDIA.

That said, starting in the early 1990s is missing the whole first half of the story, no? Searching Google Books with a 1980-1990 date range for things like "3d graphics" "art" or "3d graphics" "special effects" yields a lot of primary sources that indicate that creative applications were driving demand for chips and workstations that focused on graphics. For instance this is from a trade journal for TV producers in 1987: "Perhaps the greatest dilemma facing the industrial producer today is what to do about digital graphics... because special effects, 2d painting, and 3d animation all rely on basically the same kind of hardware, it should be possible to design a 'graphics computer' that can handle several different kinds of functions." [https://www.google.com/books/edition/E_ITV/0JRYAAAAYAAJ?hl=e...]

It's not hard to find more examples like this from the 1985-1989 period.

Thanks for doing this. We should talk! I am a historian and have been exploring using LLMs in teaching (https://resobscura.substack.com/p/simulating-history-with-ch...). I love this idea. I’m fairly sure at this point that history education will be one of the fields most positively impacted by LLMs, just because there is so much potential for redeploying primary sources in creative ways like this.

Etymonline.com is one of my favorite websites. I had no idea it had a blog though - thanks for posting. I love the description of English as “Built from half-Frenchified Roman marble and local wattle-and-daub.”

Author here. Thank you, I thought this was an interesting point. Researching the book this post is based on, I was really struck by who was actually attending the Macy cybernetic conferences — it was incredibly eclectic. Anthropologists, physiologists, psychiatrists... and, of course, also Claude Shannon and von Neumann.

Carving off computer science as a separate realm with less interdisciplinary input was definitely a fork in the road for the history of science.

By the way, if anyone is interested, you can find the list of cybernetics conference attendees here: https://www.asc-cybernetics.org/foundations/history/MacyPeop...

And if anyone is really interested, here's my book, which is coming out in January: https://www.hachettebookgroup.com/titles/benjamin-breen/trip...

Not quite smoking (in the sense of using pipes), as he describes them as throwing cannabis seeds on hot stones and inhaling the smoke. I mention it in my Lapham's Quarterly article:

“The Scythians then take the seed of this hemp [kannabis] and, crawling in under the mats, throw it on the red-hot stones, where it smolders and sends forth such fumes that no Greek vapor bath could surpass it. The Scythians howl in their joy at the vapor bath.”

One thing that's interesting about this is that "Scythian" was a pretty broad category in Herodotus's time, in some senses meaning a specific tribe around present-day Bulgaria, in others broadly referring to something like "Indo-European nomadic horse people." So it could be that Herodotus is actually passing along an account of Indo-Aryan or Indo-Scythian practice that is part of the larger South Asian culture of cannabis use (see: https://en.wikipedia.org/wiki/Indo-Scythians).

Yes, this is right on (and great book rec - Mike Jay is a friend and a brilliant writer on drug history).

The concept of smoking cannabis is not quite so surprisingly new as that of smoking opium (which apparently emerged in Southeast Asia as recently as ~300 years ago). However although there is archaeological evidence of cannabis smoking in South Asia and sub-Saharan Africa stretching back quite a long time, there is virtually no evidence that it reached Europe before the 19th century.

I wrote an article about this for Lapham's Quarterly (https://www.laphamsquarterly.org/roundtable/our-strange-addi...) that gets into one intriguing piece of inconclusive evidence from earlier:

>In 1689 the natural philosopher Robert Hooke gave a firsthand report on the effects of Indian cannabis to the Royal Society of London: “The patient understands not, nor remembers anything that he sees, hears, or does in that ecstasy, but becomes, as it were, a mere natural, being unable to speak a word of sense; yet is he very merry, and laughs, and sings, and speaks words without any coherence.” Hooke described cannabis as being “chewed or swallowed” but specified the dose only ambiguously, as “about as much as may fill a common tobacco-pipe.” It is still not clear to what extent cannabis smoking, as opposed to edible or drinkable preparations, was practiced in Hooke’s time outside of the cannabis-smoking regions of South Asia and Africa.

I made a pretty thorough search of the textual mentions of cannabis in the 16th and 17th centuries and couldn't find any evidence that Europeans were aware you could smoke it. In other words, it's entirely possible that someone who sat for a portrait in the Renaissance had consumed cannabis orally -- in the form of either hashish or mixed in an "electuary" with things like nutmeg and sugar -- but I would be very, very surprised to see any visual evidence of it being smoked, because the textual records don't support it.

"If this knowledge is valuable, the best way to show a support for it is hiring somebody to work full time on this problem."

Why not have both? I.e. academics and students dedicated to the fairly monastic career of being a professional historian (which I am), but also prizes which seek to bring in experts from other fields with different skillsets who may approach the problem in an original way.

Obviously there is a limited amount of funding for anything. But surely some of the billions being donated every year to the general funds of elite universities like Harvard would be better spent setting up specific, goal-oriented prizes modeled on the Vesuvius Challenge.

This is awesome, thank you! Or should I say gratias tibi :)

This is the result when I asked GPT-4 to write "a new, original passage in latin which is inspired by Del Rio's style. Make the Latin elegant and lucid, with witty turns of phrase," then asked Claude to translate it back to English. I am no Latin expert, but I know enough to say that this is clearly lucid, makes sense -- though overwrought -- and has no glaring errors. I will ask my classicist friends to weigh in on the matter of style though.

GPT-4 improvising: "Quemadmodum divini philosophi docent, mysteria naturae introrsus latitant, neque enim sine causa Democritus 'naturam rerum' dixit esse clausam in occulto. Verum, si Angelorum bonorum opera ut in Psalmis celebratur, 'Benedicite, omnes Angeli Domini, Domino,' quo pacto maleficae entitates—daemones scilicet—invisibiles nec tamen inactiones sunt? Mihi consentaneum videtur quod Tertullianus et Ambrosius prodiderunt, angeli esse nuntios divinos, sed daemonas esse nuntios corruptionis. Ne confundamus! Nam verbum 'daemon' Graecis non semper malum significat, sed Romanis—o mores!—semper sinistram portendit.

In huius argumenti speculo, miror qua de causa daemones semper in tenebris vel in locis desertis, vel in nocturna silentia se abscondant. An non, ut Seneca loquitur, 'magna vis veritatis est, quae se per se defendet'? Atqui daemones a veritate abhorrent, quasi nox a lucem fugiat.

Ergo, si volumus a daemonum insidiis nosmetipsos protegere, debemus, ut Virgilius canit, 'forsan et haec olim meminisse juvabit,' non oblivisci quod lux aeterna sit telum acerrimum contra tenebras. Haec ita dicam, et iam concludo: sicut angeli ad caelum pertinent, sic daemones ad abyssum—et utrique suum locum noverunt. Caveamus igitur ne, dum caelum quaerimus, abyssum inveniamus."

Claude translation: "As the divine philosophers teach, the mysteries of nature lie hidden within, for not without reason did Democritus call nature 'enclosed in the dark'. Yet, if by the work of good Angels as celebrated in the Psalms, 'Bless the Lord, all you Angels of the Lord,' in what manner are evil entities—namely demons—invisible yet not inactive? It seems fitting to me what Tertullian and Ambrose handed down, that angels are divine messengers, but demons are messengers of corruption. Let us not confuse the two! For the word 'daemon' does not always signify something evil for the Greeks, but for the Romans—oh the times!—it always portends something sinister. In the mirror of this argument, I wonder for what reason demons always hide themselves in shadows or deserted places or the silences of the night. Does not, as Seneca says, 'truth have great power, which will defend itself'? Yet demons shrink from the truth, as the night flees from the light. Therefore, if we wish to protect ourselves from the snares of demons, we must, as Virgil sings, 'perhaps one day it will please us to remember these things,' not forget that eternal light is the sharpest weapon against the darkness. I will say this much, and now conclude: just as angels belong to heaven, so demons belong to the abyss—and both know their proper places. Let us beware, therefore, lest while seeking heaven we find the abyss."

Well, that's just it - I use Google Translate all the time to translate historical texts, and for whatever reason, GPT-4 and Claude both work better. Since I deal with texts that feature archaic orthography like the long s (ſ) the main advantage over Google Translate is that LLMs can make educated guesses about what a word should be. But even in terms of pure translation ability — assuming all orthographic issues have been corrected — the LLMs do a better job in the languages I've tested and which I can read (early modern Portuguese, Spanish and French, plus Latin).

The post by David Bell which I linked to gets into this for French - I agree with him that ChatGPT (I guess he was using GPT 3.5) has a tendency to "overtranslate." But it is super impressive as a translator overall IMO: https://davidabell.substack.com/p/playing-around-with-machin...

As I move into my second decade as an educator (started as a teaching assistant around 2012, now a history professor) it's been really interesting seeing what "sticks" and what doesn't. A couple years back I made a Simpsons reference that I know would've landed with my students from circa 2015, but it was crickets. However I feel like certain movies and other cultural productions from as far back as the 70s are still on students' radar -- like Scorsese films, for instance.

I've got a two year old daughter and as she gets older, I'm tempted to show her old episodes of the PBS show Nova and the original Cosmos so we're on the same cultural wavelength, but I probably shouldn't try to force it. She was born with an illustrated box set of the Lord of the Rings already sitting in her room though ;)

Author here - yes, this is what I was getting at. The false promise of a rapid victory over the French via encirclement from the right flank (in the style of Hannibal) really was a major factor in justifying the risks of war for the German General Staff. And because Schlieffen was preoccupied with the idea that an ancient battle could be a model for that approach, it distracted from the now-obvious technological changes of the period (machine guns=defensive warfare) that made the approach impossible.

https://en.wikipedia.org/wiki/Schlieffen_Plan