RAG Is Not Enough: When to Fine-Tune vs. Retrieve

Retrieval gives a model facts it never saw. Fine-tuning gives it behaviour it never learned. Teams reach for RAG by default and then wonder why the model still writes in the wrong tone, ignores their output schema, or rambles.

The decision rule

• –Knowledge changes weekly, answers must cite sources → retrieval.
• –Format, tone, or a domain-specific taxonomy must be obeyed every time → fine-tune.
• –Both → retrieve the facts, fine-tune the behaviour. This is the common production answer.
• –Latency budget under ~300 ms and a small closed domain → fine-tune a small model and skip the retrieval hop.

Retrieval that actually retrieves

Most disappointing RAG is a chunking problem, not a model problem. Hybrid search plus a re-rank pass moves recall more than swapping embedding models.

def retrieve(query: str, k: int = 8) -> list[Chunk]:
    dense = vector_store.similarity_search(query, k=40)
    sparse = bm25.search(query, k=40)

    # Reciprocal rank fusion: robust, no score normalisation needed.
    scores: dict[str, float] = {}
    for ranking in (dense, sparse):
        for rank, chunk in enumerate(ranking):
            scores[chunk.id] = scores.get(chunk.id, 0.0) + 1.0 / (60 + rank)

    top = sorted(scores, key=scores.get, reverse=True)[:20]
    return reranker.rank(query, [by_id[i] for i in top])[:k]

Fine-tuning the behaviour, not the facts

A few hundred well-curated examples of the exact output shape you want beats thousands of scraped pairs. Keep the retrieved context in the training examples so the model learns to use it.

{"messages":[
  {"role":"system","content":"Answer only from CONTEXT. Reply with JSON {answer, citations}."},
  {"role":"user","content":"CONTEXT:\n[1] Refunds are processed in 5 business days.\n\nQ: How long do refunds take?"},
  {"role":"assistant","content":"{\"answer\":\"Five business days.\",\"citations\":[1]}"}
]}

Measure before you choose

Split evaluation in two: a retrieval metric (was the right chunk in the top-k?) and a generation metric (given the right chunk, was the answer correct and well-formed?). If retrieval recall is 0.6, fine-tuning will not save you — and if recall is 0.95 while answers still fail schema validation, no amount of chunk tuning will.