RAG vs Fine-Tuning (2026): How to Decide for Your AI MVP in an Afternoon

Teams lose weeks to this decision. They read that fine-tuning makes a model "learn your business", budget for it, and discover four months later that the model still can't answer a question about a policy written last Tuesday.

That's not a tooling failure. It's a category error. The two techniques solve different problems, and only one of them is about knowledge.

We hear this question on roughly half our scoping calls, usually phrased as "we're deciding between RAG and fine-tuning." Almost every time, the honest response is that they haven't got a decision yet. They've got twenty test cases they haven't written and a prompt nobody has taken seriously. That sounds dismissive. It isn't. It's the fastest path to shipping, and it costs nothing.

This guide uses published API rates checked on 25 August 2026, peer-reviewed results on what fine-tuning can and can't teach, and a protocol you can finish between lunch and dinner. If your product needs retrieval systems built from your own documents, the same protocol is how we scope it.

What Do RAG and Fine-Tuning Actually Do?

Strip away the marketing and there are two different interventions.

That distinction does the work. A model with your handbook in its context can quote your refund window. A model fine-tuned on your handbook has, at best, absorbed a statistical impression of it.

Here's the version we use on calls. Fine-tuning is like sending someone on a training course. They come back with better habits, a house style, and a feel for how you do things. They don't come back knowing what's in the email you sent this morning. RAG is handing them the email before they answer. You can do both, and most mature products eventually do, but if what you actually need is for them to read today's email, no amount of training will substitute.

The other thing worth knowing early: your data doesn't move. Fine-tuning bakes it into a model checkpoint, so a policy change means retraining and re-evaluating. Retrieval reads the current document at query time, which means correcting a mistake is an edit, not a project. For a product still finding its shape, that difference compounds fast.

RAG vs Fine-Tuning: The Honest Comparison

Here is the comparison with the rows that usually get left out.

That last row surprises people. Anthropic's published pricing lists no fine-tuning tier for Claude at all. If your stack is on Claude, fine-tuning isn't a decision you get to make without changing provider.

What Does Each One Cost in 2026?

Published list rates, read from each vendor's own pricing page on 25 August 2026. Recheck before you budget.

Read the premium column twice. Fine-tuning does not charge you once. It charges you 1.5x to 2x on every token for the life of the product, and it applies to output tokens, which already cost four times what input costs. A chatty product pays that tax harder than a terse one.

RAG's cost works the other way. It adds input tokens per call, and input is the cheap side. It is also the side that caching discounts: OpenAI prices cached input at 75% off on the gpt-4.1 family, and Anthropic prices a cache hit at 0.1x base input.

A worked example at 20,000 tasks a month

Assume a support-answer feature on gpt-4.1-mini. Each task sends 800 prompt tokens and returns 300. RAG adds 2,500 retrieved tokens per call.

What does diverge is the retraining bill. Every behaviour change means another training run and another evaluation pass, and when the provider retires that model version you pay it again. RAG's equivalent is re-indexing one document, which costs cents. Our breakdown of what an AI MVP actually costs to run separates these ongoing ledgers properly.

What the Research Says About Teaching a Model Facts

This is where the decision usually gets made, and it isn't close.

The FineTuneBench number is the one to keep. Those researchers tested the actual commercial APIs a founder would reach for, and found that fine-tuning updated existing knowledge correctly 19% of the time. Their Gemini results were worse: the paper concludes those APIs were unable to learn new knowledge at all.

Fine-tuning is not a memory. Stop treating it like one.

The Afternoon Protocol: Four Hours to a Decision

You do not need a spike, a vendor call, or a week. You need twenty real cases and four hours. Block them out, in this order.

Hour four is the whole point. You're not asking "did it get better". You're asking "what kind of wrong was it", because the kind of wrong maps directly onto the fix.

Why twenty cases? Because it's enough to see a pattern and few enough that you'll actually finish. Five hides the pattern. A hundred turns hour one into a week and the project dies there. Twenty is the number we use internally, and we've never needed more to make this particular call.

One rule for hour three: don't build anything. The temptation to stand up a vector database before you know whether retrieval helps is strong, and it's where most of these afternoons turn into fortnights. Copy and paste the document. If the answers get better with the information sitting in the prompt, retrieval is worth engineering. If they don't, you've saved yourself a sprint and learned something more valuable than any benchmark would have told you. Our step-by-step guide to building an AI MVP puts this same test before the architecture decision for the same reason.

What Your Failure Modes Are Telling You

The seven failure points below come from Barnett et al.'s study of three production RAG systems. The right-hand column is ours.

Count your pile. Five of the seven are not fine-tuning problems. If most of your failures land in rows one to four or row seven, buying a training run will change nothing except your monthly bill.

If your pile is genuinely stacked in rows five and six, you have found the real case for fine-tuning, and you should read on.

When Do You Actually Need Fine-Tuning?

There is a real case. It is narrower than the marketing, and it looks like this.

Notice what's absent from that list. None of it is about knowledge. Every item is about form, and form is what training teaches well.

You also need the data. A few hundred clean, consistent examples minimum, and consistency matters more than volume. Contradictory examples teach a model to be inconsistent, reliably. We've seen a team assemble twelve hundred examples from three different authors who each had their own idea of a good answer, then wonder why the fine-tuned model couldn't pick a lane. It had learned exactly what it was shown.

There's a timing argument too, and it cuts against fine-tuning early. Base models keep improving, and every improvement is free if you're prompting and expensive if you've trained. A fine-tune pins you to one checkpoint. When something better ships six months later, you don't just switch: you rebuild your training set, retrain, and re-evaluate before you can take the upgrade. For an MVP that's still changing weekly, that's a bad trade. For a stable, narrow, high-volume task, it's often worth it.

When Is Neither One Your Problem?

The unprofitable part of the answer, and the one we give most often on scoping calls.

That last one is worth money to hear early. Our guide to choosing an AI MVP tech stack covers when a model swap beats an architecture change.

Can You Use Both?

Yes, and for mature products it's often the right end state. The Microsoft agriculture study found the gains were cumulative: fine-tuning contributed over six points of accuracy, retrieval added five more on top.

Read those numbers carefully before you plan around them. Six points and five points are real improvements, and they are not the order of magnitude that the word "learn your business" suggests. If your evaluation set is failing half the time, an eleven-point combined lift doesn't get you to a shippable product. Something more basic is wrong, and you'll find it in hour four.

The sequencing matters too.

When we built Find The Plan, a plan-comparison MVP, retrieval was the whole architecture: every recommendation had to show the evidence behind it and hand off to a licensed advisor. A fine-tuned model could have produced fluent recommendations. It could not have shown its sources, and in that domain a recommendation without sources is not a product.

Which Should You Pick?

The verdict, by what you are building.

A document question-answering MVP

RAG, without hesitation. Your users will ask about things written after the model's training cut-off, they will expect citations, and access control has to be enforceable per user. All three are retrieval properties and none of them survive fine-tuning.

An internal tool over a small, stable handbook

Probably neither. Run the break-even above. Under ten thousand tokens, cache the corpus in the prompt and ship this week. You can add retrieval later when the corpus grows, and that migration is easy.

A structured extraction or classification feature

Fine-tuning is a genuine contender here, and it may be the cheaper answer because it lets you drop to a smaller model. Baseline with a prompt first, because current models are better at this than they were when the fine-tuning advice was written.

A support or operations agent

Retrieval first, then behaviour. Get the agent grounded in real records, add permissions and approval points, and only then consider training the response style. Our AI agent development work sequences it that way for a reason: an agent that acts on a wrong retrieval is worse than one that answers slowly.

Frequently Asked Questions