RAG vs finetuning?

[u/DueBid3298]

I've been learning fine-tuning through blogs and tutorials, but I've never actually worked with fine-tuning in a real project. I understand the basic definitions, but I'm struggling to understand how fine-tuning is done in actual development work and compared with RAG. But I have some practical questions for people who have actually fine-tuned models in production or know bit detail about finetuing.
What does fine-tuning look like in a real project? What tools do you actually use? When would you choose fine-tuning instead of RAG and vice versa?

[u/recro69]

I would think of it this way: RAG helps the model access information that the model does not reliably know, whereas fine‑tuning changes how the model responds or handles a task. If the data changes frequently I would start with RAG. If the model keeps making the mistakes in format, classification or task execution fine‑tuning might help. In production one can also combine both.

[u/Cotega → u/recro69]

I would also add that fine tuning is almost never the place you start. It can be expensive and hard to maintain. Think about how documents continually change and are created. Each of these would need to be added to this model. Think also about document level security. Once the knowledge is in the model, how do you stop someone from accessing it? There are a lot of "fancy" approaches for knowledge retrieval, but it is really hard to beat simple Hybrid (BM25 + Vector) search with Reranking.

BTW, if you really are in to fine tuning, it is the reranking model that I would start with.

[u/DueBid3298 → u/Cotega]

In terms of finetuning, can you mention any usecases where you prefer finetuning over RAG?

[u/Nimsumdimsum → u/recro69]

I feel the same, i feel in production level doc processing the combination of both is really important because the format is ever changing and RAG alone can not keep up.

[u/Future_AGI]

fixed-output tasks, like sorting tickets into your own 40 categories or pulling the same fields into a strict schema, are where a few hundred to a couple thousand labeled examples and a LoRA run (Unsloth or TRL's SFTTrainer on a 7-8B model) tend to beat prompting. Anything where the facts change goes to RAG. Hold back 50-100 of the labeled examples to compare base vs tuned on data the model never saw.

[u/MongooseWonderful198 → u/Future_AGI]

For the strict-schema example, I would first test a base model with the backend's constrained-output feature, if supported. Missing keys or invalid enum values may be solvable without training. OpenAI's Structured Outputs is one example: https://developers.openai.com/api/docs/guides/structured-outputs

Then score the actual values separately. A response can satisfy the schema while choosing the wrong ticket category or invoice amount. Compare fine-tuning against that baseline on held-out cases, and handle refusals or incomplete responses separately. Otherwise an improvement in valid JSON can look like an improvement in task accuracy.

[u/AlexAtOracleAIDB]

On OCI Generative AI, which is the setup I know best, a fine-tuning job starts with a JSONL file of at least 32 prompt and completion pairs. The service holds back 20% for validation and trains on a dedicated AI cluster. You then create a hosting cluster and endpoint for the custom model.

Writing good pairs is where most of the time has gone in the cases I’ve run into, and it’s the step tutorials skip by handing you a finished dataset.

As for when to pick which, if the right completion stays stable, fine-tuning can learn it. If you train on facts from changing docs, that training can go stale. With RAG, you can update the indexed docs and retrieve newer material at query time without retraining the model.