A simulation of me: fine-tuning an LLM on 240k text messages

Yakamoz S-245

Technology is doing things that many people a century ago thought was impossible, in some cases practically, and in other cases even theoretically.

For example, that the chips inside your phone have features significantly smaller than the wavelengths of light used to etch them, which are also in the size range where some of the people involved in the development were sure that quantum mechanical effects would lead to so many electrons tunnelling through insulators as to prohibit reliable logic gates.

Also above liquid nitrogen temperature superconductors, from what I've heard there is no known complete model of that class of superconductors as BCS theory (developed from 1948 to 1957, getting a Nobel Prize in 1972) doesn't explain anything over 30 K… but most people only interact with those superconductors in the form of YouTube videos about them, not even physical toys.

My dad, back when he was alive, gave anecdotes about how much and fast computers changed things over his career. He was born in 1939, and for him "storing all of Shakespeare on a ball bearing" and "have as many digits of pi as you want" were remarkable; and a long-distance phone call meant from Hampshire (in the UK) to Sussex (a county which shares a border with Hampshire), not international, and local phone numbers being typically four digits (although at least one relative just had ten buttons on their phone and each one would call a specific named person, and anything else you'd need the operator). TV (if you could afford it) was black and white (as were the newspapers), and nobody could record anything (though by my time he had managed to pick the third horse of the two-horse race in home taping, as he had a Video 2000 system instead of VHS or Betamax).

He missed the discovery of liquid nitrogen temperature superconductors, and didn't understand public-key encryption (which would've been useful given he worked on IFF systems for Plessey/GEC Marconi).

Radar existed a century ago (under a different name), but magnetrons didn't and RF electronics were terrible. The magnetrons went on to enable microwave ovens.

Nukes didn't exist until he was nearly 6, and the first thermonuclear weapon was detonated just after he turned 13. Artificial satellites were sci-fi until he was 18, and manned spaceflight was sci-fi until he was 21. There are no analogies to these things in the world of astrology or acupuncture.

> It’s not curtailing human social problems that have existed since forever.

Apart from all the diseases we've cured. I finally visited the Mary Rose museum over the holidays, you know what those sailors' bones showed? Rickets, scurvy, broken bones that weren't set right and healed wrong.

Know what else we've done? There are half as many people in abject poverty today than there were a century ago, despite the population going up about 3.5-fold since 1920. "Tech" isn't just your phone and your video games, it's the M-PESA pseudo-banking system letting people use an old Nokia-1100 instead of cash[0].

It's the expansion of rights for women, and more education for everyone, where the pressure came from organisation and coordination which itself was enabled by communication technology.

It's likewise the reporting of My Lai that forced introspection in the USA.

It's thermal remote sensing[1] both in satellites (giving us better weather forecasts and warnings of forest fires) and personal devices (making it cheaper to find where houses need more insulation); it's the microcontrollers in medical devices giving people cybernetic implants for supporting patients with diabetes and for pacemakers, it's the neuroprosthetics in the form of cochlear implants and epilepsy management devices; it's the MRI, CT, and ultrasound scanners; it's the DNA tests, the IVF treatments, the hormonal contraceptives and hormonal supplements for menopause.

And one of the core issues of the human condition is communication: long-distance any-time communication (which is the point of a mobile phone) has gone from a sci-fi fantasy in 1924 to such cheap ubiquity that even in the poorest communities their absence is considered noteworthy.

And then you have things like Google Translate. I'm still surprising and delighting people by showing them a feature first demonstrated over a decade ago — augmented reality live visual translation (although back in 2010 Word Lens was from a separate company, but Google bought them). No more gullible westerners getting "外国变态" as a tattoo thinking it says "asian beauty".

> Trade offs in society just like engineering; arguments it saves people from exploitation at home but airplanes and modern tracking enable large scale human trafficking.

Airplanes? No, and you sound historically (and economically) uninformed for even suggesting that. One of the other places I went in this holiday period was Bristol, where I saw the plinth from which locals have recently toppled and dragged away an unwanted statue commemorating the local philanthropy of a historical slave trader[2]. Planes are about the dumbest possible way to attempt this in the modern world, precisely because of ID requirements on basically all airports.

> The yearly resource consumption for yearly phone updates is insane.

Also no, but that's less nuts than saying airplanes enable large scale human trafficking, there's billions of us and it is normal for people to be very bad with balancing the scale of big numbers.

[0] https://en.wikipedia.org/wiki/File:M-Pesa_on_Nokia_1100_2012...

[1] https://en.wikipedia.org/wiki/Thermal_remote_sensing

[2] https://en.wikipedia.org/wiki/Statue_of_Edward_Colston#:~:te....

You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b parameter model in perplexity and qualitative testing. No one I know does fp32, there's no benefit.

People train LoRAs at low precision regularly. The folks on /r/localllama are really pushing the boundaries of what can be done on consumer hardware, and the boundaries are further out than you'd think.

Also, if you want to really blow-up your intuition, check out model merges...

No man, it's not ridiculous. If I write a program that copies someone's book and try to sell it I'm infringing on that copyright. I cannot sell a zipped version of the Harry Potter books. I feel like there's so many people weighing in on this discussion who haven't actually done any real world copyright related stuff.