Hacker News Reader

Comment by danielhanchen | Hacker News Reader

Hey HN family! I found a few bugs for Phi-4 - Microsoft's latest MIT licensed LLM to be on par with GPT-4o mini

1. End of sentence should be <|im_end|> not <|endoftext|>

2. Chat template should not auto add an assistant prompt

3. Padding token should not be EOS but <|dummy_87|>

I also converted Phi-4 to Llama-arch. I uploaded GGUFs, 4bit quants, dynamic quants and all fixes to https://huggingface.co/unsloth

I also made a Colab notebook to finetune Phi-4 on a free GPU: https://colab.research.google.com/github/unslothai/notebooks...

View on HN

> We converted Phi-4 to Llama’s architecture for better accuracy and easier use.

What does this mean? When I think about "model architecture", I think about the number of weights in each layer, the organization of the layers, etc. And AFAIK, it's untenable to "port" a model from one to the other without effectively retraining it. So what does it actually mean to "convert to Llama's architecture"?

Oh Phi-4's architecture is inspired from Llama itself, except they merged the attention matrices into 1 large matrix for better FLOP utilization, and the gate/up matrices in the MLP.

Phi-3 use to use sliding window attention, but they got rid of that in Phi-4.

So, you can "Mistral-fy" Phi-3 and convert it to Mistral arch (by unmerging the merges), and now you can "Llama-fy" Phi-4 to Llama arch.

The reason why accuracy increases in finetuning is because during LoRA finetuning, you learn only 1 A matrix for merged QKV, whilst unmerging it creates 3 A matrices - this allows the model to have more freedom to learn new features.

I know some of those words... Man, do you recommend any blog/book/etc. that teaches me how to know this stuff?

Most books are either too low level or too high level.

If it helps there a few YouTube recordings of conferences and workshops I did about stuff!

Low level Technicals of LLMs: https://www.youtube.com/watch?v=pRM_P6UfdIc

CUDA / GPU Mode talk about it here: https://www.youtube.com/watch?v=hfb_AIhDYnA

Chat with PyTorch team here: https://www.youtube.com/watch?v=MQwryfkydc0

PyTorch Conference talk here: https://www.youtube.com/watch?v=PdtKkc5jB4g

Would guess GGUF so you can run on llama.cpp, LM Studio, etc..., but OP can hopefully clarity further for you.

Yep converting to Llama arch definitely makes accessibility much better - also many fast LLM serving libraries normally support Llama, so it makes it easier to port and use!

Wasn't Phi-3 also bugged/is still bugged? Seems like Microsoft just doesn't care.

>to be on par with GPT-4o mini

Phi is known to overfit benchmarks. It's way, way worse then that.

Phi-3 should be fixed as well - but yes there were bugs as well! https://x.com/danielhanchen/status/1782853167572832650

Phi-3's sliding window should be 2048 and not 2047, and they also had chat template issues - I uploaded correct versions to https://huggingface.co/unsloth/Phi-3.5-mini-instruct

Anecdotally, I've been experimenting with Phi-4 the past hour or so (so, yeah, not very comprehensive) and it's certainly a strong model. Definitely better than the previous Phi models.

Yep Phi-4 definitely is better than Phi-3.5!

Huh! That may explain why I kept on getting visible <|im_end|> output when I tried running a Phi-4 GGUF file using llama.cpp.

Oh yes exactly! I trimmed it out now :)

The better chat template should be:

{% for message in messages %}{% if (message['role'] == 'system') %}{{'<|im_start|>system<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'user') %}{{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'assistant') %}{{'<|im_start|>assistant<|im_sep|>' + message['content'] + '<|im_end|>'}}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant<|im_sep|>' }}{% endif %}

> I uploaded GGUFs, 4bit quants, dynamic quants

The dynamic quantization[1] looks really interesting. Now, I've just been dabbling, but did I understand correctly that this dynamic quantization is compatible with GUFF? If so, how do you convert it? Just the standard way or?

I was really curious to try the dynamic 4-bit version of the Llama-3.2 11B Vision model as I found the Q8 variant much better than the standard Q4_K_M variant in certain cases, but it doesn't fully fit my GPU so is significantly slower.

[1]: https://unsloth.ai/blog/dynamic-4bit

Oh the dynamic 4bit quants sadly are not GGUF compatible yet - it currently works through Hugging Face transformers, Unsloth and other trainers.

My goal was to make a dynamic quant for GGUF as well - it's just a tad bit complicated to select which layers to quantize and not with GGUF - I might have to manually edit the llama.cpp quantize C file

Also I'm unsure yet if llama.cpp as of 11th Jan 2025 supports Llama Vision (or maybe it's new?) I do remember Qwen / Llava type models are working

> Oh the dynamic 4bit quants sadly are not GGUF compatible yet

Ah bummer. Is this a GGUF file-format issue or mostly "just" a code-doesn't-exist issue?

> Also I'm unsure yet if llama.cpp as of 11th Jan 2025 supports Llama Vision (or maybe it's new?)

Ah, I totally forgot Ollama did that on their own and didn't merge upstream.

I'm using Ollama because was so easy to get running on my main Windows rig so I can take advantage of my GPU there, I still do a bit of gaming, while all the stuff which uses Ollama for inference I run on my server.

Anyway thanks for the response.

> Ah bummer. Is this a GGUF file-format issue or mostly "just" a code-doesn't-exist issue?

Just code! Technically I was working on dynamic quants for DeepSeek V3 (200GB in size), which will increase accuracy by a lot for a 2bit model (if you leave attention in 4bit), and just use 20GB more. -> But I'm still working on it!

> Ah, I totally forgot Ollama did that on their own and didn't merge upstream.

Yep they have Llama vision support! Llama.cpp has Qwen, Llava support - I think Llama V support is coming, but it'll take much more time - the arch is vastly different than normal transformers due to cross attention

Can you convert to ONNX so I can try in web browser?

Would like to update this:

https://huggingface.co/spaces/webml-community/phi-3.5-webgpu

Oh I can probs try doing this!

Reply on news.ycombinator.com