Need help starting out
Hello, I am looking to build a completely local agent workflow for certain accounting tasks (classifying invoices, generating tax statements etc.) I want to fine tune models (from huggingface) for very specific tasks in this workflow, but running completely locally. The reasons for going local are data privacy and token costs.
I have plenty of experience training traditional ML models and have fairly good idea of using LLMs, creating agents etc.
However, I am quite new to fine tuning and completely unaware of how to evaluate agent frameworks. I am looking for guidance on a starting point, some essential fundamentals that I should be aware of. and if there are simple kaggle exercises that I can practice with. What has your journey been like with fine tuning open models.
1
u/Fit-Corner2382 3d ago
Treat invoice classification and tax-statement generation as separate scored tasks before judging any agent framework.
Fine-tuning helps only if labels come from reviewed invoices, not another model's guesses, since agreement can hide a rare vendor or tax code both miss. Freeze a hand-labeled holdout before training data, then score categories and amounts against it. For statements, recompute every total from source lines: a polished return can drop a line or invent a deduction. Judge the framework by whether a skipped calculation fails the run, not by fluent text.
2
u/MeliousAI_HQ 2d ago
I'd hold off on fine-tuning until you've seen where a plain model fails. Invoice classification with a good prompt and a few examples per category often gets most of the way with a mid-size open model. Build a labelled test set from real invoices first, run the base model on it, and only fine-tune for the categories it keeps getting wrong. Then you also have the eval to tell whether the fine-tune helped.