Learnings from fine-tuning LLM on my Telegram messages

"Using a half-precision FSDP full shard with a 1024 sequence length and a micro batch size of 2 required 63GB of VRAM on each of the eight A100 80 GB GPUs. The training, lasting three epochs, took just 20 minutes. The total cost for the VM was $8.88 per hour, resulting in $3, not including the time for experiments and bug fixes."

I wondered where you could rent cycles on a machine like that, a quick Google found that p4d.24xlarge on AWS is available, while the on-demand cost is $20.1755 per hour, the Spot is only $8.99 (I guess it's gone up?)

Cool to know I could fine-tune for only ~$3.

Back when the "I forced a bot to watch 1000 hours..." memes were popular (https://knowyourmeme.com/memes/i-forced-a-bot), ages ago in AI/ML time, I tried to do something similar by fine-tuning GPT-2 on messages from a group chat of my friends. Since there were years of chat data, it seemed like a really good opportunity to test whether the language model would capture everyone's personality and generate funny, uncanny-valley versions of our banter.

Turns out that the group chat was used nearly exclusively for sending funny pictures and videos (that the language model obviously couldn't see), and for making plans to meet up. The generated conversations almost exclusively consisted of a random group chat member starting with "there is a party tonight, who wants to go?" and others saying "I'm down" or "when?" or "where?" It was 0% banter, and 100% logistics.

It was pretty hilarious in its own way, but not for the reasons anyone expected! I didn't learn very much about language models with that experiment, but I did learn that my friends' group chat is actually pretty boring.

I guess the best banter happens in real life. Glad to see it worked out somewhat more interestingly for this person, even if they did allude to some similar results in their closing thoughts section.

If it's so easy, then you don't need to remove it. The model will solve it easily and focus on everything else. At best, you save some parameters and compute, at worst, you are damaging its ability to learn important things like conversational skills or modeling people. When it comes to LLMs, more is more, and trying to hand-engineer the dataset or think for the LLM can backfire in very subtle and difficult to diagnose ways.

> Ok, it is capable of forming coherent sentences. The most noticeable problem is its lack of awareness regarding the context of the conversations which leads to bland and generic replies. The messages lacked any distinct style, feeling quite basic... > > Conversations have become more interesting and engaging, although there’s still a risk of losing context. Russian language performance has improved, but errors still occur. I believe that before fine-tuning for a specific task with limited data, like mine, it would be beneficial to first fine-tune the model unsupervised on a large corpus of Russian texts. Additionally, incorporating common conversation partners’ names as separate tokens might enhance the quality. I wouldn’t say it has turned out to be significantly better than LoRA. It might be more effective to focus solely on a single person and calculate the loss based only on my responses (or someone else’s), instead of trying to learn about each and every conversational partner.

[0]https://en.wikipedia.org/wiki/Be_Right_Back

Except for encrypted chats (which have bad UI and only work on one device) your messages are stored unencrypted on their servers (handed over to authorities, etc.).

It's also interesting that бля was translated to 'damn'. :)

Mark: wassup

Andy: just chilling

It simulated our conversational style and topics quite well, though GPT-2 reads like a glorified Markov chain. Sometimes the outputs were absolutely hilarious and inappropriate. GPT-2 was peak comedy.

My friend described GPT-2 as "like watching a toddler learning how to walk. When it stumbles it's cute and funny." GPT-3, not so much...

Also, it was oddly (painfully) accurate as far as personality goes... like looking into a mirror. For one thing, I talk way more, and this was reflected in the model's output. For another, I am constantly trying to turn my life around and failing, but ever optimistic... and talking about creative plans endlessly without much execution. (So GPT-2-andai ended up the same way...)

I've wondered since fine-tuning started being a thing how long it'd be before somebody makes a utility where you can dump a giant chat export into it and an API key and then it fine tunes a Telegram bot that can imitate any of your friends - would be fun to play with and even create a group chat with multiple friend-bots talking to each other to see how long until it goes off the rails.