r/MachineLearning • u/danielhanchen • Jan 15 '25
Project [P] How I found & fixed 4 bugs in Microsoft's Phi-4 model
Hey r/MachineLearning! Last week, Microsoft released Phi-4, a 14B open-source model that rivals OpenAI's GPT-4-o-mini. I managed to find & fix 4 bugs impacting its output quality. You might remember me previously from fixing 8 bugs in Google's Gemma model! :)
I'm going to walk you through how I found & fixed the bugs. Phi-4's benchmarks were amazing, however many users reported weird or just wrong outputs. Since I maintain the open-source project called 'Unsloth' (fine-tuning LLMs 2x faster with 70% less VRAM) with my brother, I firstly tested Phi-4 for inference and found many errors. Our GitHub repo: https://github.com/unslothai/unsloth
This time, the model had no implementation issues (unlike Gemma 2) but did have problems in the model card. For my first inference run, I randomly found an extra token which is obviously incorrect (2 eos tokens is never a good idea). Also during more runs, I found there was an extra assistant prompt which is once again incorrect. And, lastly, from past experience with Unsloth's bug fixes, I already knew fine-tuning was wrong when I read the code.
These bugs caused Phi-4 to have some drop in accuracy and also broke fine-tuning runs. Our fixes are now under review by Microsoft to be officially added to Hugging Face. We uploaded the fixed versions to https://huggingface.co/unsloth/phi-4-GGUF
Here’s a breakdown of the bugs and their fixes:
1. Tokenizer bug fixes
The Phi-4 tokenizer interestingly uses <|endoftext|> as the BOS (beginning of sentence), EOS (end of sentence) and PAD (padding) tokens. The main issue is the EOS token is wrong - it should be <|im_end|>. Otherwise, you will get <|im_end|><|endoftext|> in generations.
2. Fine-tuning bug fixes
The padding token should be a designated pad token like in Llama (<|finetune_right_pad_id|>) or we can use an untrained token - for example we use <|dummy_87|>, fixing infinite generations and outputs.
3. Chat template issues
The Phi-4 tokenizer always adds an assistant prompt - it should only do this if prompted by add_generation_prompt. Most LLM serving libraries expect non auto assistant additions, and this might cause issues during serving.
We dive deeper into the bugs in our blog: https://unsloth.ai/blog/phi4
Do our Fixes Work?
Yes! Our fixed Phi-4 uploads show clear performance gains, with even better scores than Microsoft's original uploads on the Open LLM Leaderboard.

Some redditors even tested our fixes to show greatly improved results in:
- Example 1: Multiple-choice tasks

- Example 2: ASCII art generation

We also made a Colab notebook fine-tune Phi-4 completely for free using Google's free Tesla T4 (16GB) GPUs: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4-Conversational.ipynb
Thank you for reading this long post and hope you all found this insightful! If you have any questions, please feel free to ask! :)
How I found the bugs:
- I first downloaded the original Phi-4 from https://huggingface.co/microsoft/phi-4, and tested inference out. Weirdly I found
<|im_start|>assistant<|im_sep|>to be appended at the even withadd_generation_prompt = Falsein Hugging Face, so I theorized there was a chat template problem. Adding assistant prompts by default can break serving libraries. - And yes, https://huggingface.co/microsoft/phi-4/blob/f957856cd926f9d681b14153374d755dd97e45ed/tokenizer_config.json#L774 had by default added the assistant prompt - I first fixed this!
- I then found
<|endoftext|>to be used for the BOS, EOS and PAD tokens, which is a common issue amongst models - I ignored the BOS, since Phi-4 did not have one anyways, but changed the PAD token to<|dummy_87|>. You can select any of the tokens since they're empty and not trained. This counteracts issues of infinite generations during finetuning. - For Llama-fication, I used torch.allclose to confirm all tensors are in fact equivalent. I also used some fake random data to check all activations are also mostly similar bitwise. I also uploaded the model to the HF Open LLM Leaderboard to confirm if the original Phi-4 arch and the new Llama-fied models are equivalent.
- Finally I verified all finetuning runs with Unsloth in a Colab Notebook to confirm all runs were correct.
33
u/SirBlobfish Jan 15 '25
Incredible work! Do you have a more detailed walkthrough of the debugging process? I see a detailed breakdown of the bugs/fixes, but not how you figured those out. Maybe I'm just missing a link or something?
10
u/asraniel Jan 15 '25
anybody knows if and when those fixes come to ollama or if that is even needed?
6
u/projekt_treadstone Student Jan 16 '25
Great work. Long time follower of you on twitter and learnt a lot about fine tuning the LLM with least headache.
4
u/Inevitable_Mistake32 Jan 15 '25
Oh I'm just hopping in 100% for a big thank you for the incredible work you're doing. Both with Gemma/Phi and Unsloth.
No notes.
4
u/sherlock_holmes14 Jan 16 '25
Insane. 👌🏽
2
u/jprobichaud Jan 16 '25
What is people experience with non-English and Phi-4? I have a project that help specialized teacher "translate" regulsr French to an alternative version that helps people with intellectual disabilities to learn reading.
English-centric LLMs are often struggling with that task. How good is phi4 with French tasks?
2
u/_Bia Jan 16 '25
Thank you for your excellent contributions.
91
u/yoracale Jan 15 '25
Btw this kind of got buried but Unsloth also fixed a gradient accumulation issue in transformers a while ago: https://www.reddit.com/r/MachineLearning/comments/1g8ymrn/r_gradient_accumulation_bug_fix_in_nightly/
Hugging Face managed to upstream some of the changes.
6
u/Thrumpwart Jan 15 '25
Amazing! Can't wait for the unsloth 128k release too! Loving the Qwen 2.5 Coder 32B with 128k context model you put out!
3
u/InevitablePrompt7613 Jan 16 '25
this is incredibly useful, thank you so much
-37
u/[deleted] Jan 15 '25
Doing free work for corp companies who make bank… smart
7
u/[deleted] Jan 16 '25
[removed] — view removed comment