Hacker News Reader

Comment by Dorialexander | Hacker News Reader

Hi! Model creator here. I happen to be a cultural historian and that's a main use case that I see. It's not complicated to learn about past events but having a general idea of the culture of the time (and its alieness from our perspective) is way harder. Monad is just prototype but there is a real potential for immersing experience through historical LLMs.

View on HN

Thanks for doing this. We should talk! I am a historian and have been exploring using LLMs in teaching (https://resobscura.substack.com/p/simulating-history-with-ch...). I love this idea. I’m fairly sure at this point that history education will be one of the fields most positively impacted by LLMs, just because there is so much potential for redeploying primary sources in creative ways like this.

Certainly. In fact I see we already follow each other on Twitter :D

And yes totally. The other massive impact could be in source analysis. I have started using Mistral-Hermes for text annotation and it is both impressive and very fast.

Would be interesting to check with a suitably trained LLM if our understand of historical wording is actually correct. Whole parts of history might be written down but lost in translation if you see what I'm saying...

I don't see what you're saying. How do you propose to determine whether the current understanding is incorrect or the model is unsuitably trained?

That would be a question for the LLM. It could explain where exactly our understanding is incorrect and provide examples.

A poorly trained llm would do that just fine.

A poorly trained LLM would do that poorly, by definition.

What I'm saying is you review its findings and verify it. Use your own intelligence to determine if its bullshit or not.

Great project! Do you have a list of the training/fine tuning data that went into it?

A great use would be to enable one to have conversations with Pascal or Leibnitz, etc.

For instance, I published online the complete text of the Mémoires de Saint-Simon (written in 1745-1755, but describing the second part of the reign of Louis XIV and the Régence, 1695-1721).

Saint-Simon was described by his contemporaries to be one of the greats conversationalists of his time. It would be so cool to chat with him.

I published the completely dataset here: https://huggingface.co/datasets/Pclanglais/MonadGPT

While I don't think Saint-Simon is included, a French colleague did a few try with it that turned out better than ChatGPT.

I'm currently working on an extended historical model for French (from 1000-2000) and maybe Saint-Simon memoirs will be included as well.

Thanks for the info.

> the completely dataset here: https://huggingface.co/datasets/Pclanglais/MonadGPT

Classic French transcription seems to be lacking. In particular, "s" used to be printed in a manner very similar to "f", but they're really s.

For example this:

> ce qui augmentoit ſes craintesc'eſt que certe innocente Vierge ne parloit iamais d'autre choſe aux Domeſtiques que du lcge d'Orl'cans donnant à connoitre à la façon dont elle en difcouroit que fon inclination eſtoit toute aux armes

should be spelled like this:

> ce qui augmentoit ses craintes c'est que cette innocente Vierge ne parloit jamais d'autre chose aux Domestiques que du ?? d'Orléans donnant à connoître à la façon dont elle en discouroit que son inclination étoit (or estoit) toute aux armes

Maybe there should be some kind of dictionary step before fine-tuning?

Ah it's completely volontary on my part: I want to keep the historical spelling as much a possible. That's why I used the google books OCR which does a better work at it than Gallica. That's still a bit erased in the current model (I don't think the tokenizer likes this so much).

Ok -- "avoit" instead of "avait" is indeed a different spelling -- but "f" in original text is not a different spelling, it's a different way of writing the same letter s (a different shape, but the same letter).

A roman era GPT would be fantastic

Already in the work. Just had a meeting today with two latinists about it.

Awesome!!

Why don't you restrict the training just to old text? That would be awesome

I'm not sure you'd be able to scrape together enough data for that to be viable...

https://en.wikipedia.org/wiki/Corpus_Inscriptionum_Latinarum

approximately 180,000 inscriptions

That only contains a few million tokens. Useless for pre-training an LLM from scratch. You would need to find billions of tokens.

It should be similar to this 1700s English model, probably trained on modern data to start and then at the end fitted to the smaller data set

Yes it requires an extremely large diverse training set for the first unsupervised stage (pre-training). Then fine tune it on the smaller data set. But we may need to wait for the next generation of LLMs that incorporate planning algorithms so that it can better stay focused on its goal for whatever tasks we are asking it to do for research purposes. Otherwise we end up with this https://news.ycombinator.com/item?id=38418974

Not feasible to go with pretraining only.

What is possible is to use a larger learning rate but this will be a hard trade-off with conversational capacities. Fine tuning is currently based on original texts with a synthetic prompt. The issues that people have noticed (repetitions, not remembering what was in the prompt) will be more significant if the learning rate is higher.

Maybe a solution will be to provide two different variant of the same model, one less immersive and more workable, and the other more immersive and buggy.

What did you use to train this? I'm thinking about making a fine tune of my own and if you have some tips to train it easier that would be much appreciated!

Any chance anyone is working on one for "the 80s"?

This seems like a fine way to steel man a controversy by going directly to the sources. I'd like to be able to, e.g., dial up a debate between Karl Marx and Milton Friedman. Even better if it's possible for Karl to convince Milton or visa versa.

Reply on news.ycombinator.com