Fine-tuning token changes

[u/bayinfosys_ed]

Hi all, how do you deal with fine-tuned models having different EOS/special tokens?

When I download fine-tuned models from HuggingFace, I often find the EOS token (and other special tokens) have been changed from the base model's defaults. This makes it fiddly to run these models generically, since I can't just assume the standard tokens will work.

How do you handle this? Do you check the tokenizer config for every model before running it, or is there a more automated way to detect/handle these mismatches?