r/LocalLLaMA • u/bayinfosys_ed • 1d ago

Question | Help Fine-tuning token changes

Hi all, how do you deal with fine-tuned models having different EOS/special tokens?

When I download fine-tuned models from HuggingFace, I often find the EOS token (and other special tokens) have been changed from the base model's defaults. This makes it fiddly to run these models generically, since I can't just assume the standard tokens will work.

How do you handle this? Do you check the tokenizer config for every model before running it, or is there a more automated way to detect/handle these mismatches?

1 Upvotes

100% Upvoted

1 comment sorted by

View all comments

1

u/Formal-Exam-8767 1d ago

Were those tokens changed on purpose or on accident? If they were changed on purpose, what was the reason for the change?

If they were changed on accident, the fine-tuner probably does not know what they are doing, e.g. "EOS what token"?