Working with businesses in the US, UK, UAE, Canada and Australia info@vrisic.com

RAG vs Fine Tuning: How to Choose for Business AI, With Costs and Failure Modes

RAG vs fine tuning for business AI: what each changes, where each fails, worked cost examples, privacy trade-offs and a decision ladder to use before tuning.

Updated 14 min read
RAG vs Fine Tuning: How to Choose for Business AI, With Costs and Failure Modes

Short answer: Use RAG (retrieval augmented generation) when the AI must answer from your documents, those documents change, or people need to see sources. Use fine tuning when you need consistent format, tone or a narrow classification skill, not new facts. Try prompting and prompt caching first: in our illustrative triage example they cut the cost of a prompted model to about $70 a month against about $50 for a tuned one.

“Should we fine tune a model on our data?” is the question we hear most often at the start of a project, and the answer is usually no, not first. The two techniques solve different problems, and most of the money wasted on them comes from using one to do the job of the other. This guide explains what each technique changes, where each fails, what each costs with worked arithmetic, and a ladder of cheaper options to try before either. It is written for a buyer or an engineering lead choosing an approach, and it uses published research and vendor documentation where numbers are involved. For the retrieval side of the work see our enterprise RAG and AI search service; for the model side, generative AI development.

What each one changes

The cleanest way to separate them is by what they change.

  • RAG changes the input. At question time the system searches your documents, pulls out the relevant passages and puts them in front of the model with the question. The model itself is unchanged. The original RAG paper described this as pairing a pre trained model, its parametric memory, with a dense vector index of documents, its non parametric memory, accessed by a retriever (Lewis et al.).
  • Fine tuning changes the model. You train an existing model further on examples of inputs and desired outputs, which adjusts its weights. The result is a new, private version of the model that behaves differently on every request, with no documents attached.

Everything else follows from that. Because RAG changes the input, you can update knowledge by updating documents, show the passages used, and restrict which passages a given user can retrieve. Because fine tuning changes the weights, you can change style, format and skills, but you cannot point to where an answer came from, you cannot update one fact without retraining, and you cannot hide part of the model from one user.

Five rung decision ladder: better prompt, knowledge in the prompt, RAG, fine tuning, then both
Climb the ladder: cheapest fix first. Vrisic framework; 200,000 token guidance from Anthropic, Contextual Retrieval

The decision ladder: try cheaper options first

Treat the choice as a ladder and stop at the first rung that meets your accuracy target.

  1. A better prompt. Clear instructions, a defined output format and a few examples. It costs hours, and modern models follow instructions well enough that it solves many “style” and “format” problems.
  2. Put the knowledge in the prompt. Anthropic’s guide to contextual retrieval notes that if your knowledge base is under 200,000 tokens, you can include the whole thing in the prompt and skip RAG, and prompt caching keeps repeat calls cheap (see the worked example below) (Anthropic). That was written for the models of its day, so check current context limits and prices. Long inputs also carry a risk: research on long contexts found that performance is often highest when relevant information is at the beginning or end of the input and drops when it is in the middle, even for models built for long contexts (Liu et al.).
  3. RAG. When the knowledge is too large, changes often or needs access control and citations.
  4. Fine tuning. When tests show a style, format or narrow skill problem that prompting cannot fix, or when volume makes a smaller tuned model cheaper.
  5. Both. A tuned model that is also fed retrieved passages, once you have measured that each part earns its place.

Why fine tuning is a poor way to add facts

The most common misunderstanding is that you can “teach the model our documents” by fine tuning on them. Research does not support it as a method for adding knowledge. A study that compared unsupervised fine tuning with RAG for knowledge injection across several tasks found that fine tuning gave some improvement, but RAG consistently outperformed it, both for knowledge the model had seen in training and for entirely new knowledge. The authors also found that models struggle to learn new facts through unsupervised fine tuning (Ovadia et al., EMNLP 2024).

There are practical reasons beyond the research:

  • Facts go stale. A price list or policy changes next month. With RAG you replace a document. With tuning you retrain and re-test.
  • No citations. A tuned model gives an answer with no source. Staff cannot check it, and a regulator or customer cannot be shown where it came from.
  • Confident errors. A model tuned on your documents will still produce fluent wrong answers about them, and nothing flags which ones.
  • No per user permissions. If HR documents were in the training data, the model can leak them to anyone who can ask the right question.

The right use of tuning is to teach behaviour, such as how to write a claims letter or which of 40 categories a ticket belongs to, and let retrieval supply the facts.

Bar chart: failed retrievals fall from 5.7 percent with standard embeddings to 3.7 with contextual embeddings, 2.9 adding keyword search and 1.9 adding reranking
Most RAG errors are retrieval errors. Anthropic, Contextual Retrieval, on Anthropic's own test sets; your results will differ

Where RAG actually fails

RAG is not a switch you turn on. In our experience most RAG disappointments are retrieval failures, where the right passage never reached the model, not model failures. Anthropic measured this: in its tests, the top 20 retrieved chunks missed the relevant passage in 5.7% of queries with standard embeddings. Adding context to each chunk before embedding cut failures by 35% to 3.7%, combining that with keyword search (BM25) cut them by 49% to 2.9%, and adding a reranking step cut them by 67% to 1.9% (Anthropic). Those are results on Anthropic’s own test sets, not a guarantee for yours, but they show how much engineering sits between “we added a vector database” and a system that finds the right passage.

The failure modes we test for, in rough order of frequency:

  1. Chunks that lose their meaning. A paragraph that says “the limit is 5,000” is useless without knowing which product, plan and year. Add the document title and section path to every chunk.
  2. Keyword blind spots. Part numbers, error codes, names and clause numbers are poorly matched by embeddings alone. Combine vector and keyword search.
  3. Too few or too many passages. Too few misses the answer. Too many dilute it and, as the long context research shows, the middle gets ignored.
  4. Stale or duplicate documents. Three versions of the same policy produce three answers. Index the current version, keep dates, and retire old files.
  5. Tables and scans. Parsing a pricing table or a scanned PDF badly gives the model garbage. Test the parser on your worst documents first.
  6. The model ignores the passages. It answers from memory when the sources are silent. Require citations and a refusal when no passage supports an answer.
  7. Permissions applied after retrieval. Filter by user rights inside the search, not after, or restricted text can leak into the answer.

What a RAG system costs to run

Illustrative example: an internal policy assistant for 2,000 documents averaging 5,000 tokens each, answering 5,000 questions a month. Prices below use a third party summary of Anthropic’s API rates updated 14 September 2026, where Sonnet 5 is $2 per million input tokens and $10 per million output tokens (CloudZero).

Item Calculation Cost
Add context to each chunk, one time 2,000 documents x 5,000 tokens = 10 million tokens, at about $1.02 per million with caching per Anthropic’s example About $10
Model cost per answer 8 passages x 400 tokens + 500 token question and instructions = 3,700 input tokens; 250 output tokens $0.0074 + $0.0025 = about $0.01
Model cost per month 5,000 answers x $0.0099 About $50
Vector store A managed vector database has a floor; for example, the default Amazon OpenSearch Serverless setup behind Bedrock Knowledge Bases requires 2 compute units at $0.24 an hour About $350 a month

The vector store floor of about $350 a month comes from a CloudZero summary of Bedrock pricing, which also notes cheaper dev and scale to zero options (CloudZero). At this scale a vector extension on a database you already run can cost far less, in our experience, so the infrastructure choice matters more than the model bill. The build is the larger cost: ingestion pipelines, parsing, search tuning, permissions, an evaluation set and a feedback loop. That is why RAG projects are priced like software, not like subscriptions. See the AI agent cost guide for how we scope work in this range.

What fine tuning is good for

  • Consistent format and style. A specific report layout, a house voice, a strict JSON schema, where prompts keep drifting.
  • Narrow classification and extraction. Routing tickets into 40 categories or pulling fields from a recurring document type, where you have thousands of labelled examples.
  • Distillation. Using a large model’s outputs to train a smaller, faster, cheaper model for one task at very high volume.
  • Latency and deployment limits. A small tuned model on your own hardware where calling a large hosted model is not allowed or too slow.
  • Domain language. Specialised terminology or non English dialects the base model handles poorly, shown by tests, not assumed.

If the problem is document extraction, test an off the shelf model first. Our guide to OCR vs LLM for invoice extraction shows how far a prompted model gets before tuning is worth considering.

What fine tuning costs, with a worked example

The training bill is usually the smallest part. Published training prices per million tokens as of 18 September 2026 include $0.34 for a 9 billion parameter open model on Together, $1.50 to $5.00 for OpenAI’s smaller GPT models, and $25 for GPT-4.1 and GPT-4o, with LoRA training on Fireworks from $0.50 for models up to 16 billion parameters (Gupta). The cost that matters is data and serving.

Illustrative example: ticket triage at 100,000 tickets a month

A company routes support tickets into 40 categories. The prompted version sends each ticket with 20 worked examples to a Sonnet class model.

Option Calculation per ticket Monthly cost for 100,000 tickets
Prompted, 20 examples, no caching 2,500 input tokens x $2 per million = $0.0050; 20 output tokens x $10 per million = $0.0002 About $520
Prompted, 20 examples, cached Cache reads cost 10% of the input price, so 2,500 tokens at $0.20 per million = $0.0005, plus $0.0002 output About $70
Tuned small model, short prompt 400 input and 20 output tokens at Haiku class rates ($1 and $5): $0.0004 + $0.0001 About $50

Cache read pricing at 10% of the base input price and the model rates come from the CloudZero summary above. Whether a tuned model is served at base prices depends on the provider: Fireworks states that tuned models are served at base model prices, while custom models on Amazon Bedrock require provisioned throughput, and a dedicated H100 on Together lists at $5.49 per GPU hour, or about $4,000 a month if left running (Gupta). Our assumption of Haiku class rates for the tuned model is therefore the best case.

The result is the key insight. Before caching, tuning looked like a saving of about $470 a month. With caching, the saving is about $20 a month, which would never repay the work. Add the one time costs, around 3,000 labelled examples at 40 seconds of review each, or about 33 hours, plus evaluation and engineering, and the project pays back only at far higher volume, or if tuning fixes an accuracy or latency problem that prompting could not. That is the right way to justify tuning: by a measured quality gap or a hard latency or deployment constraint, not by a hoped for token saving.

Side by side

RAG Fine tuning
What it changes The input to the model The model’s weights
Best for Facts from your documents, policies, products, records Format, tone, narrow skills, high volume tasks
Updating knowledge Replace or add a document Retrain and re-test
Shows sources Yes, by design No
Per user permissions Yes, filter at retrieval No, one model for everyone
Data you need Clean, current documents Hundreds to thousands of labelled input and output pairs (Bedrock’s Claude 3 Haiku tuning accepts 32 to 10,000 examples)
Main cost Build and search quality, plus a vector store Data labelling and evaluation, plus serving
Typical failure Right passage never retrieved Confident wrong facts; drift from base model updates
Time to first result Days to weeks Weeks, after data is ready
Reversibility High: change the index Low: retrain from scratch

The Bedrock figure comes from AWS’s announcement of fine tuning for Claude 3 Haiku, which lists 32 to 10,000 training lines and says Bedrock trains a private copy of the base model that only the customer can access (AWS).

Privacy, access control and deletion

The two approaches create different privacy problems. With RAG your documents sit in an index you control, and each query sends only the relevant passages to the model. You can run the index and the model in your own cloud account, apply per user permissions at search time, and delete a document and have it disappear from answers immediately. With fine tuning, your examples become part of a model copy. That copy is private to you on the major platforms, but it contains what it was trained on for every user of that model, and removing one person’s data later means retraining.

Two practical rules follow. Do not fine tune on personal, health or contractual data unless you have a lawful basis, a retention plan and a way to retrain when someone asks for deletion. And where staff have different access rights, use RAG with permission filtering. For the strictest requirements, see our guide to private and self hosted LLMs.

Platform reality: who still offers tuning

Managed fine tuning is less available than it was. Reports dated 2026 say OpenAI announced in May that it was winding down its self serve fine tuning platform: new organisations were blocked first, with job creation ending for all customers by January 2027 and inference on existing tuned models continuing until their base models are deprecated. OpenAI’s stated reason was that newer models follow instructions and formats well enough to make much of the work unnecessary. The same reports say Google’s Gemini on Vertex AI and Anthropic’s Claude through Amazon Bedrock still offer managed tuning, and that open weight models can be tuned with parameter efficient methods such as LoRA (Tessl). Check the vendor’s own notice before you plan around any of these dates.

The planning lesson is lock in. A tuned model is tied to its base model. When the base is deprecated you must retrain on a new one, which is easy if you kept your training data, test set and scripts and painful if you did not. We insist that clients own all three.

Using both

Combining the two makes sense when each solves a different measured problem. Typical cases:

  • Regulated writing. RAG supplies the clause or guideline text; a tuned model writes it in the required structure and tone.
  • Specialised retrieval. A tuned reranker or embedding model improves search on your jargon, while a general model writes the answer.
  • Cost at scale. A distilled small model handles routine questions, and hard ones escalate to a larger model.

Add the second technique only after you have measured the first. A common mistake is to build both at once and then be unable to tell which one is responsible for an error.

How to evaluate before you commit

Whichever route you choose, build the test set first. It takes a few days and prevents months of argument.

  1. Collect 100 to 200 real questions or tasks with the correct answer and, for RAG, the document passage that supports it.
  2. Measure retrieval separately. For each question, is the supporting passage in the top 5 or top 20 results? This one number tells you whether to fix search or the model.
  3. Measure answers for faithfulness. Does each claim appear in the retrieved text? Mark unsupported claims as failures even if they happen to be true.
  4. Test the refusal. Include 20 questions your documents cannot answer. The correct response is “I could not find this”, not a plausible guess.
  5. For tuning, hold out data. Keep 15 to 20% of your examples unseen during training and compare against the best prompted baseline on exactly those.
  6. Re-run on every change to prompts, chunking, models or training data.

Mistakes we see

  • Fine tuning to add knowledge. It is the wrong tool; see the research above.
  • Skipping the prompt and caching baseline. Teams tune a model to save money that a cache would have saved for free.
  • Judging RAG by the demo question. Test on the ugly questions, the scanned PDFs and the contradictory documents.
  • Indexing everything. Old drafts and duplicates cause wrong answers. Curate the corpus.
  • Ignoring permissions until the end. Retro fitting access control into a finished index is costly.
  • No owner for the content. RAG quality decays when nobody keeps documents current.
  • Not keeping training data and tests. You will need them when the base model is retired.
Two by two matrix: prompting when both needs are low, RAG when facts matter, fine tuning when style matters, and both when both matter
Which technique fits your problem. Vrisic analysis

Decision rules

  • Answers must come from documents that change, with sources: RAG.
  • Knowledge base under about 200,000 tokens and one team using it: put it in the prompt and cache it.
  • Strict output format or house style that prompts cannot hold: try structured outputs and examples first, then tune.
  • Thousands of labelled examples and a high volume narrow task: tune a small model, after pricing the cached prompt baseline.
  • Different users may see different documents: RAG with permission filters, never tuning on the restricted text.
  • Unsure: build the 100 question test set and let the numbers choose.

How Vrisic can help

We build RAG systems and, where tests justify it, tuned models, and we start by building your test set so the choice is measured. After a free scoping call we quote a fixed price. The system runs in your cloud with your own model keys, you own the code, prompts, index pipeline, training data and test data, and there are no per seat fees.

To see how your documents behave before committing, the AI Pilot costs $1,500, runs for 10 business days and is credited against a build started within 60 days. You can also read how we work or book the scoping call.

Sources

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
  2. Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (Ovadia et al., EMNLP 2024)
  3. Lost in the Middle: How Language Models Use Long Contexts (Liu et al., arXiv)
  4. Contextual Retrieval in AI Systems (Anthropic)
  5. Anthropic Claude API Pricing: Every Model, Token Rate, and Cost Lever (CloudZero, updated 14 September 2026)
  6. Amazon Bedrock pricing: every model, hidden costs (CloudZero)
  7. Top 8 Fine-Tuning and Model Customization Platforms 2026 (Deepak Gupta, checked 18 September 2026)
  8. Fine-tuning for Anthropic’s Claude 3 Haiku model in Amazon Bedrock is now generally available (AWS News Blog)
  9. OpenAI is shutting down self-serve fine-tuning (Tessl)

Questions we hear every week

Still unsure about something? Ask us on a call. We will give you a straight answer, even if it means we are not the right partner.

What is the difference between RAG and fine tuning?

RAG searches your documents at question time and gives the relevant passages to the model, so the model is unchanged. Fine tuning trains a model further on example inputs and outputs, which changes its weights. RAG suits facts that change and need sources; fine tuning suits format, tone and narrow skills such as classification.

Is RAG cheaper than fine tuning?

Usually to start, and certainly to keep current. In our illustrative example, a policy assistant answering 5,000 questions a month cost about $50 in model usage, plus a vector store that can run from near zero to about $350 a month. Fine tuning has cheap training but costs data labelling, evaluation and serving, and any fact change means retraining.

Can I fine tune a model on my company documents so it knows them?

It is the wrong tool for that. A 2024 EMNLP study found RAG consistently outperformed unsupervised fine tuning for both old and new knowledge, and that models struggle to learn new facts that way. Tuned models also give no sources and cannot respect per user permissions. Use RAG for knowledge and tuning for behaviour.

Does RAG stop AI making things up?

It reduces invented answers when responses are grounded in retrieved text and the system refuses when nothing supports an answer, but it does not eliminate them. Most RAG errors are retrieval failures: Anthropic measured 5.7% of queries missing the right passage in the top 20 with standard embeddings, falling to 1.9% with contextual embeddings, keyword search and reranking.

How much data do I need to fine tune?

It depends on the task. Amazon Bedrock’s Claude 3 Haiku tuning accepts 32 to 10,000 training examples, but quality depends on label accuracy more than count. For classification we would plan for 1,000 to 5,000 reviewed examples and hold out 15 to 20% for testing. Always compare the tuned result against a well prompted and cached baseline on the same held out test cases before committing to the project.

Can I use RAG and fine tuning together?

Yes, once each has earned its place in tests. Common patterns are RAG supplying clause text while a tuned model writes it in a required structure, a tuned reranker for specialised jargon, or a small distilled model for routine queries with hard cases escalated to a larger one. Add the second technique only after measuring the first.

Is my data safe if I use RAG or fine tuning?

Both can run in your own cloud account with your own model keys. RAG lets you filter by user permissions and delete a document with immediate effect. Fine tuning bakes examples into a model copy for all its users, so deleting one person’s data means retraining. Avoid tuning on personal or restricted data without a lawful basis and a retraining plan.

Is OpenAI discontinuing fine tuning?

Reports dated 2026 say OpenAI announced in May 2026 that it is winding down its self serve fine tuning platform, with new organisations blocked first and new jobs ending for everyone by January 2027, and existing tuned models running until their base models are deprecated. Verify dates in OpenAI’s notice, and keep your training data and tests so you can retrain elsewhere.

Find the process where AI will pay off first

Message us on WhatsApp for the fastest reply, or book a free thirty minute call. You will leave with the processes most worth automating, whether to build new or upgrade what you have, a realistic timeline and a clear idea of cost, whether or not you work with us.

No obligation. NDA available on request. WhatsApp replies are usually fast; email within one business day. Or start with a 10 day AI Pilot for $1,500, credited in full
Chat with us on WhatsApp Chat on WhatsApp