In this case, what are the tokens?

Also, not all language models are transformers. You can have language models based on diffusion models or state-space models, or any other model that can be used to model sequences (so all of them, as sequences are just functions). Transformers are just the ones that are most common and successful today.

It gets extremely blurry, because people commonly refer to any model that uses a component associated with the Transformer architecture as a Transformer (i.e. using some kind of QKV-esque attention mechanism). I think it's easier to think of it like this:

A large language model is just what it says--a very large statistical model trained for language tasks. This covers the spectrum of GPT-style models, but also those hard to classify ones, like Liquid's "Liquid Foundation Models", which can get up to 24 billion parameters and use grouped query attention, but are closely related to state-space models as well: https://huggingface.co/LiquidAI/LFM2-24B-A2B

Also, as others have pointed out, a Transformer isn't inherently a language model. So really they're sort of two different axes, one classifying the model size and task, the other referring to a specific architecture.

In my unpopular opinion, it wasn't transformers or attention, but pretraining on language data that kicked off the LLM Boom. Alec radford in his little jupyter notebook trained a very small non-transformer to predict simply the next-character on Amazon reviews. He noticed emergence of a neuron which when toggled controlled the sentiment of the remaining text. This is published as the sentiment neuron paper.

Till then, transformers were being used primarily for stuff like translation and such and no one was even pretraining at scale, even tho transformers and attention existed.

Openai and google if you count T5 persisting with pretrained generative models was what led to the LLM boom. Yes they used transformers, but that's just one IMO minor aspect.

There is no language in the training of this, so there is no l.