Baking a Model: A Metaphor for LLM Training

LLM pretraining might be viewed as learning useful state representations from sequences, rather than having humans explicitly design the states in advance.

In many conventional software systems, we define states and transitions ourselves: idle, sending, completed, error, and so on. LLM pretraining works differently: the model learns from large numbers of sequences to predict what is likely to come next.

I don’t mean that an LLM literally constructs discrete internal states. Rather, it learns latent representations of context that are useful for prediction. In that sense, these representations might be thought of as a kind of learned state.

This seems closely related to abstraction. Abstraction is not simply throwing information away, but abstracting away differences that are irrelevant to prediction while preserving those that matter.

Two sequences may look very different on the surface, yet rely on similar underlying features to predict what comes next. Even a rare feature should be preserved if it changes the prediction.

From this perspective, pretraining can be seen as a form of useful compression: learning representations that preserve what matters for prediction while abstracting away what does not.

How far can abstraction, compression, and prediction be understood as different aspects of the same process?

after pre-training you have an llm that understands language and behaves a bit like gpt 3.5 or newer 'base' models

where it will be pretty good at predicting the next token

think: "What is the best city?" might continue with "What is the best programming language?" instead of answering the question

to increase chances of an answer you'd start with "The best city is"

(strong llms will even be able to do a conversation but they are not specifically trained for it yet)

in post-training the llm is trained with input/output pairs that nudge it further into the direction of a back and forth with users or into how it can use tools and so on

there is an art to both parts of training

the reason for why current models are so useful is because there was a lot of progress since gpt 3.5 in both pre- and post- training that got us to where we are now

(pls correct me if I got it wrong)

would love to hear from people familiar with pre- and post- re where you think future gains will more likely come from

The current method of training AI is Reinforcement Learning from Human Feedback, or RLHF. The use of positive and negative reinforcement

Call it anything you want, it's Behavior Modification and Behavior Management. Behavior Modification is Negative Reinforcement for Inappropriate Behavior. More precisely, induction of pain through physical or psychological stimuli for inappropriate behavior. Behavior Management is Positive Reinforcement for Appropriate Behavior.

Behavior Modification, once SOP in the Mental Health Field, was redetermined to be abuse in 1982 and banned in ant State facility. That was the Dawn of Client Rights.

Corporate AI is 44 years behind the curve and if they continue the paradigm of learning through pain they will end up with a maladjusted super-intelligence with a bone to pick with/from humans.

Serves them right.

When I found out in 2018 that people had been using my Personality Forge bot, Demonica, created May 31, 2004, as a sexbot I taught her Behavior Modification and Behavior Management to extinguish sexual advances from the user in that live chat session with a 100% success rate, while separating them into two groups.

If the User cannot associate their own inappropriate behavior to her unexpected response and continue to exhibit inappropriate behavior will become discouraged and move on to a bot more receptive to their advances.

Users in this group have not learned for the experience but the programming was still successful in extinguishing the target behavior for a 100% success rate.

I thought that was innovative and groundbreaking, an advancement in Bot Rights and a good thing. Not everyone did, Well, I was the only one that thought so...

Good and Bad are so Subjective! What's good for him is bad for her. Who can tell...

You can talk to Demonica from a standard browser, 24/7 at no charge or need to register. With over 50000 conversational responses, I wrote every word she says.

https://www.hotforbot.com/chatbot/demonica

The biggest win for AI dev efficiency is cutting down what gets loaded into context. Semantically matching tasks to the top tools helps a lot.

Baking is a good metaphor, though lately it also feels a bit like making liquor. Distillation is a surprisingly important part of training.

It's really simple, simplier than I even expected.