So how is it decoding b64 then ? We have no idea.
We don't built Neural Networks. Not really. We build architectures and then train them. Whatever they learn is outside the scope of human action beyond supplying the training data.
What they learn is largely unknown beyond trivial toy examples.
We know connections form, we can see the weights, we can even see the matrices multiplying. We don't know what any of those calculations are doing. We don't know what they mean.
Would an alien understand C Code just because he could see it executing ?
So, this is very similar to an LLM having to deal with tokenized input, but instead of sequences of tokens representing words you've got sequences of Base64 characters representing words.
Maybe "no idea" was a bit much for this example but any idea certainly didn't come from seeing the matrices themselves fly.
This isn't speculation - I've implemented Base64 decode/encode myself, and you can google for the definition if you don't believe I've accurately described it!
Edit: Basically, For all anyone knows, it treats b64 as another language entirely and decoding it is akin in the network to translating French rather than the very simple swapping you've just described.
Complexity builds upon simplicity, and the LLM will begin by noticing the direct (and repeated without variation) predictive relationship between Base64 encoded text and corresponding plain text in the training set. Having learnt this simple way to predict Base64 decoding/encoding, there is simply no mechanism whereby it could change to a more complex "like translating French" way of doing it. Once the training process has discovered that Base64 text decoding can be PERFECTLY predicted by a simple mapping, then the training error will be zero and no more changes (unnecessary complexification) will take place.
It's no secret the order of data has an impact on what the network learns and how quickly, it's just not feasible to police for these giant trillion token datasets.
If a NN learns a more complex solution that works perfectly for a less complex subset it meets later on, there is little pressure to meet the simpler solution. Especially when we're talking about instances where the more complex solution might be more robust to any weird permutations it might meet on the internet. e.g there is probably a simpler way to translate text that never has typos and a LLM will never converge on it.
Decoding/Encoding b64 is not the first thing it will learn. It will learn to predict it first as it predicts any other language carrying sequence. Then, it will learn to translate it, mostly like long after learning how to translate other languages. All that will have some impact on the exact process it carries out with b64.
And like i said, we already know for a fact it's not just doing naive substitution because it can recover corrupted b64 text wholesale that our substitutions cannot.
What examples do you have in mind?
Normally it's the opposite, where one hopes for the neural net to learn something complex, and it picks up on a far simpler pattern and uses that instead (e.g. all your enemy tanks are on a desert background, vs the others on a grass background, so it learns to discriminate based on sand vs grass).
You're anthmorphizing by saying that corrupted b64 text can be recovered. There is no "recovery process", but rather conflicting prediction patterns of b64 encoding predicting the corresponding plain text, and the plain text predicting it's own continuation.
e.g.
"the cat sat on the mat" encodes as dGhlIGNhdCBzYXQgb24gdGhlIG1hdA==, but say we've instead got a corrupted dGhlIGNhdCBzYXQgb24gdGhlIHh4dA== that decodes to "the cat sat on the xxt", so if you ask ChatGPT to decode this, it might start generating as:
dGhlIGNhdCBzYXQgb24gdGhlIHh4dA== decodes to "the cat sat on the" ...
At this point the LLM has two conflicting predictions - the b64 encoding predicting "xxt", and the plain text that it has generated so far predicting "mat". Which of these will prevail is going to depend on the specifics. I haven't tried it, but presumably this "recovery" only works where the encoded text is itself predictable ... it won't happen if you encode a random string of characters.
Also, the base model when responding to base64 text, most of the time the next token is also part of the base64 text, right? So presumably the first thing to learn would be like, predicting how some base64 text continues, which, when the base64 text is an encoding of some ascii text, seems like it would involve picking up on the patterns for that?
I would think that there would be both those cases, and cases where the plaintext is present before or after.
However, Transformer self-attention is based on key-based lookup rather than adjacency, although embeddings do include positional encoding so it can also use position where useful.
At the end of the day though, this is one of the easiest types of prediction for a transformer/LLM to learn, since (notwithstanding that we're dealing with blocks), we've just got B64 directly followed by the corresponding plain text, so it's a direct 1:1 correspondence of "when you see X, predict Y", as opposed to most other language use where what follows what is far harder to predict.
It doesn't explain higher level generalizations like being a transpiler between different programming languages that didn't have any side-by-side examples in the training data. Or giving an answer in the voice of some celebrity. Or being able to find entire rhyming word sequences across languages. These are probably more like the kind of unexplainable generalizations that you were referring to.
I think it may be better to frame it in terms of accuracy vs precision. Many people can explain accurately what an LLM is doing under all those matrix multiplies, both during training and inference. But, precisely why an input leads to the resulting output is not explainable. Being able to do that would involve "seeing" the shape of the hypersurface of the entire language model, which as sibling commenters have mentioned is quite difficult even when aided by probing tools.
Where does “we can’t tell whether a person possesses the skill of ‘skateboarding’?” fit in with, DNA not encoding anything specific to skateboarding? It isn’t as if we designed our genome and therefore if our genome did hard-code skateboarding skill that we would therefore (as designers of our genome) have full understanding of how skateboarding skill works at the neuron level.
I recognize that a metaphor/analogy/whatever does not have to extend to all parts of something, and indeed most metaphors/analogies/whatever fail at some point if pushed too far. But, I don’t understand how the commonalities you are pointing to between [NN architecture : full NN network with the specific weights] and [human genome : the whole behavior of a person’s brain including all the facts, behaviors, etc. that they’ve learned throughout their life] is supposed to apply to the example of _knowing_that_ a person knows how to skateboard?
It is quite possible that I’m being dense.
Could you please elaborate on the analogy / the point you are making with the analogy?
The sheer number and variety and networking of synapses involved in the skill 'skateboarding' is irreducibly, unintelligibly complex for an intelligence on the scale of a conscious human mind to describe, fully comprehend, or even recognize with a great deal of analysis. Even if you decided all the functional pathworks through the network in one example, you would not be able to decode another because every skateboarder has trained their neural network in a unique manner.
Well said. You've reminded me of a beautiful sci-fi short story almost about this exact "mystery"
Compare this to dictionaey, where it's obvious what information is on each page and each line.
Interpretability research has resulted in many useful results and pretty visualizations[1][2], and there are many efforts to understand Transformers[3][4] but we're far from being able to completely explain the large models currently in use.
[1] - https://distill.pub/2018/building-blocks/
[2] - https://distill.pub/2019/activation-atlas/
Even this characterization is not strictly valid anymore, there is a great deal of research into what's going on inside the black box. The problem was never that it was a black box(we can look inside at any time), but that it was hard to understand. KANs help some of that be placed into mathematical formulation. Generating mappings of activations over data similarly grants insight.
* Why are there are a set of f each with 0-loss that work?
* Given the weight space, and an f within it, why/when is a task/skill defined as a subset of that space covered by f?
I think a major reasons why these are hard to answer is that it's assumed that NNs are operating within an inferential statistical context (ie., reversing some latent structure in the data). But they're really bad at that. In my view, they are just representation-builders that find proxy representations in a proxy "task" space (def, aprox, proxy = "shadow of some real structure, as captured in an unrelated space").
We can use the economy as an analogy. No single person really understands the whole supply chain. But we know that each person in the supply chain is trying to maximize their own profit, and that ultimately delivers goods and services to a consumer.