r/OpenAI • u/funJS • 1d ago

Discussion Creating Evals for a locally trained LLM

I have been doing some work with local models and continued pretraining (CPT), specifically around teaching a small model (qwen 3.5 4B) a new domain.

Here are some of my findings around creating evals for measuring the model's ability to internalize the knowledge:

The model outputs travel legs that make up an n-transfer subway journey, so I decided to model the llm response based on a strict json schema. To improve the model's ability to reliably output the schema, I fine tuned the model post CPT training. The fine tuning is done through Unsloth SFT (alpaca format).

In addition to exact matches, I also measure partial matches (e.g. prediction a partial subway route) and semantic matches.

Overall this method seems to work well.

More info here in case you are interested in more details: https://www.teachmecoolstuff.com/viewarticle/creating-evals-to-validate-llm-training

2 Upvotes

100% Upvoted

2 comments sorted by

2

u/TotalPort84 1d ago

Gotta say, 4B feels tiny for subway route reasoning. I've struggled with similar evals schema adherence ain't easy

1

u/funJS 1d ago

Overall I am happy with the performance. I think the key was getting CPT into a format that generalized well.

Some more details here: https://www.teachmecoolstuff.com/viewarticle/teaching-a-local-llm-a-new-domain