First Principles on AI Progress

I don't think this takes into account sparse models and or fine tuning.

Also, storing data is better suited to databases than large language models I think so it's probably better to store "facts" in databases and let the language model access the database.