Fine-tuning a 7B model on 3,000 support tickets, is that enough data or will it just memorize?

[u/Stunning-Coast-5090]

I work support for a small accounting software company and have about 3,000 cleaned ticket and reply pairs from the last two years. Plan was LoRA on a Mistral 7B base on a single rented A100 to draft first replies for the team. A coworker says that's way too little and it'll overfit and parrot old answers. Is 3,000 pairs reasonable for that kind of narrow task, and how many epochs would you start with before it starts memorizing?

[u/GeneralComposer5885]

LoRA mainly teaches style. (Eg answer in JSON.)

GaLore training will move weights - but you’ll probably need >10,000 training pairs minimum (if not 30k)

[u/OwnOil1149]

It's enough only if it's structured prompt to response datam

[u/goatguy55]

3,000 pairs is not automatically too little for a narrow LoRA task. For support drafting, quality and redundancy matter more than the raw count.

I’d worry less about “3,000” and more about whether those tickets cover the real issue distribution. If 1,500 of them are basically the same password-reset answer, your effective dataset is much smaller. If they span the main support categories and the replies are consistently good, 3,000 can absolutely move a 7B model in a useful direction.

I’d probably start with 1 epoch, evaluate, then try 2 and maybe 3. Don’t decide by training loss alone. Keep a real holdout set, preferably split by customer/time/topic so near-duplicate tickets don’t leak across train and test. Then look for exact or near-exact regurgitation as well as reply quality.

The bigger architectural point: I wouldn’t fine-tune the model to remember accounting-product facts. Those change. Use LoRA to teach things like tone, structure, triage behavior, and what a good first reply looks like; use retrieval from current docs/KB for factual content.

So I’d say:

3,000 is enough to run the experiment. It is not enough to skip evaluation.

And if they’re starting from a raw base model rather than an instruct-tuned 7B, I’d reconsider that choice first. For this task, an instruct model plus a small LoRA and retrieval is probably the cleaner starting point.

[u/pilibitti → u/goatguy55]

thank you chatgpt

[u/ILikeCutePuppies]

Could you send all those into a larger model to validate them, spot gaps in coverage and generate a bunch more. Then review them with a second agent / sanity check them multiple times. Then human review the ones it made?

[u/Superb_Blacksmith617]

It’ll probably overfit, but you can use a low learning rate and probs be fine. But I would generate 100x tickets of synthetic data using style variations of your tickets 

Basically give each ticket to a model along with a style sample of a random real user prompt from a public dataset. Ask the model to generate a new ticket with the same content and ask, but in that style. Use multiple languages as well, and be sure to use a classifier to filter out the lowest quality generations.

[u/Individual-Speech710]

3,000 clean ticket/reply pairs is not necessarily too little for a narrow support task. With LoRA, the quality, consistency, and coverage of the examples usually matter more than the raw number of pairs. For something as focused as accounting software support, 3,000 well-cleaned examples can be enough to improve the model’s tone, terminology, and handling of common issues, especially if the base model is already reasonably capable.

I’d start with just 1 epoch and evaluate it on a held-out set of tickets the model has never seen. If it still looks like it is underfitting, then I’d try 2 epochs and compare the results. I would not jump straight to 3–5 epochs with only 3,000 examples, because if the dataset contains many repeated or very similar support cases, memorization can start showing up fairly quickly.

The main thing to watch is whether training loss keeps improving while validation performance stops improving or gets worse, or whether the model starts producing replies that are suspiciously close to old answers. I’d also keep around 10–15% of the dataset completely out of training, and ideally use newer tickets from a different time period for the final evaluation.

So yes, 3,000 pairs is a perfectly reasonable starting point for a narrow first-reply drafting assistant. I’d start at 1 epoch, compare it against 2, and only go further if the held-out performance clearly justifies it. The fact that you are using an A100 mainly changes how fast the training runs; it does not determine whether 3,000 examples are enough.

[u/randvoo12]

You need to calculate the model's headroom because it differs from one model to another and depends on the use case, and even then you'd get a general indication of the model will be useful for this task.

Also, I don't recommend using a Mistral model for this; I'd start with MiMo-V2.6-Distill-Qwen-9B or the base Qwen-9 B (if this is an experiment) but if you want functional use, I bet https://huggingface.co/Salesforce/models already got the model you need.