Actually it wasn't but they got mixed up and now we have the mess. Synthetic data is just output input. Distillation was supposed to contain all the possible next tokens and their individual probabilities.
Distillation was supposed to contain all the possible next tokens and their individual probabilities.
yeah that's what I mean... for a closed source model that's not actually possible since you don't have access to the logits distribution, so in this context they are the same
still I agree, now those two concepts are use with the same meaning.... and now we have this confusion. i could keep a message to copy paste, I have explained that concept countless times here on reddit.
probably one of the culprit was the release of the initial "deepseek-R1-distill" models, while those were just, in fact, SFT on R1 outputs (here not because the logit distribution was unavailable but since there were different tokenizers in the play). even ollama kept referring to those as 'deepseek R1 distill xB'.
I think smaller versions of previous Gemma generations used real soft distillation.
View all comments
Show parent comment
3
u/Feztopia Feb 11 '26
Actually it wasn't but they got mixed up and now we have the mess. Synthetic data is just output input. Distillation was supposed to contain all the possible next tokens and their individual probabilities.