Time is encoded in the weights of finetuned language models
arxiv.org
arxiv.org
Feels like a click bait title. Of course language model weights encode different writing styles. The fact that you can lift out a vector to stylize writing is also more interesting, but that’s also nothing newly discovered here. It should be obvious that this is possible given that you can prompt ChatGPT to change its writing style.
Maybe this could be useful for singling out post-LLM text and generating output that excludes it.
Like, "only use the the 2000 most common words of the English language" or "the response should be 500 words long".
It’s faster than it’s ever been, and seemingly without 85% of its staff. Says a lot
Nitter: I get all 8 posts in the thread in 18 requests, 207 kB, 169 kB transferred.
X: I only get the first post, 128 requests, 11.45 MB, 2.11 MB transferred.
Allowing those who would otherwise avoid twitter to view the content.
Obviously this isn’t time travel, but more of a zeitgeist extrapolation.
I would expect that if an AI was made to answer like it’s from December 2024 it would talk a lot about the US election but it wouldn’t know who won — just that a “race is on.”
This could have actual utility: predicting trends, fads, new market opportunities, etc…
LLMs aren’t a silver bullet for everything.
Please explain this to my Product org.
So, the LLM still needs to know about 2020 from somewhere. In a way, you teach it about the task, then separately you teach it about 2020, and this method can combine that to make it solve the task for year 2020.
LLMs treat language as combinations of vectors of a very high dimension -- (x, y, z, a, b, c, d, ...). The neat thing is that we can combine these just like the 2D vectors and get meaningful results. If we have the vectors for the concepts "King" and "Woman", adding them gives a vector close to the one for "Queen"!
Once you know this, you can extrapolate and look for ways to categorize groups of vectors and combine them in new ways. As I read it, this research is about finding the vector weights for text from specific time periods -- i.e. January of 2021 -- and comparing them to the vectors for text from a different period -- i.e. March of 2021. It seems that all the operations are still meaningful, you can even do something like averaging vectors in January and March and getting ones that look like vectors in February!
To be honest, it is relatively obvious idea, to make vectors from timestamps and feed them to LLMs, but for some strange reason, nobody made this before and looks like, this is mostly unnoticed in NN community.