What does this mean? When I think about "model architecture", I think about the number of weights in each layer, the organization of the layers, etc. And AFAIK, it's untenable to "port" a model from one to the other without effectively retraining it. So what does it actually mean to "convert to Llama's architecture"?
Phi-3 use to use sliding window attention, but they got rid of that in Phi-4.
So, you can "Mistral-fy" Phi-3 and convert it to Mistral arch (by unmerging the merges), and now you can "Llama-fy" Phi-4 to Llama arch.
The reason why accuracy increases in finetuning is because during LoRA finetuning, you learn only 1 A matrix for merged QKV, whilst unmerging it creates 3 A matrices - this allows the model to have more freedom to learn new features.
Most books are either too low level or too high level.
Low level Technicals of LLMs: https://www.youtube.com/watch?v=pRM_P6UfdIc
CUDA / GPU Mode talk about it here: https://www.youtube.com/watch?v=hfb_AIhDYnA
Chat with PyTorch team here: https://www.youtube.com/watch?v=MQwryfkydc0
PyTorch Conference talk here: https://www.youtube.com/watch?v=PdtKkc5jB4g