Fine-Tuning vs. RAG vs. Prompting: Stop Asking Which Is Best

Every vendor selling fine-tuning will tell you fine-tuning is the answer. Every vendor selling a vector database will tell you RAG is the answer. Every API provider will tell you prompting is the answer. None of them are wrong — they’re just not talking about your problem.

The honest version of this comparison requires a different starting question. Not “which is best?” but “what are you actually trying to change about the model’s behavior?” Once you’re precise about that, the right choice is usually obvious. When it isn’t obvious, you’re probably dealing with a problem that needs two of the three.

Thanks for reading Under The Hood! Subscribe for free to receive new posts and support my work.

Let’s go through each approach on its actual mechanics, then build a decision framework that works for both the indie hacker shipping in two weeks and the enterprise team that has a compliance officer in the room.

What Each Approach Actually Does

The single most useful mental model for this whole decision:

Prompting changes what the model is asked to do right now.
RAG changes what information the model has access to right now.
Fine-tuning changes how the model behaves by default, permanently.

These three target entirely different levers. Conflating them is why most comparisons go wrong.

Prompting

Prompting is runtime instruction. You’re not modifying the model — you’re telling it what role to play, what constraints to operate under, what format to output, and what context to consider, all within a single request. The model resets to factory defaults the moment that context window closes.

The ceiling on prompting is higher than most developers realize. Modern frontier models — Claude Sonnet/Opus, GPT-5.3, Gemini Pro — are remarkably instruction-following. With a well-constructed system prompt, few-shot examples, output format constraints, and chain-of-thought instructions, you can get behavior that would have required fine-tuning two years ago. The effort is low, the iteration cycle is hours not weeks, and you can change behavior by editing a text file.

The floor drops out in three scenarios: when the task requires consistent behavior across thousands of requests and prompt drift is a real cost, when you’re dealing with a domain so specialized that the base model has weak priors and few-shot examples can’t compensate, and when token budget is tight and you can’t afford to include extensive instructions every call.

RAG

RAG — Retrieval-Augmented Generation — gives the model access to information it wasn’t trained on, at query time. The architecture is: user query → semantic search over a knowledge base → top-k relevant chunks retrieved → chunks injected into the prompt alongside the query → model generates a response grounded in those chunks.

What RAG changes is the model’s knowledge, not its behavior. A RAG system backed by your internal policy documents will answer questions about those policies accurately. The same RAG system cannot make the model respond in a specific communication style, follow a specific output schema consistently, or perform a narrow classification task reliably. Those are behavior problems, not knowledge problems.

The mechanics also carry real constraints. Everything retrieved has to fit inside the context window along with the original query, system prompt, and expected response. If your knowledge base has relevant information spread across 50 documents, you need sophisticated retrieval to surface the right 5. Chunking strategy, embedding model quality, reranking, and retrieval precision are all unsolved problems in most RAG implementations — which is why “RAG” as a concept sounds simple and RAG in production is where projects quietly die.

Fine-tuning

Fine-tuning continues training a model on your dataset, updating the model’s weights so the new behavior is baked in. The model’s default responses change. It doesn’t need to be told every time — the behavior is now intrinsic.

What fine-tuning actually buys you:

Behavioral consistency at scale. Every call follows the fine-tuned behavior without needing extensive system prompt instructions.
Format and schema adherence. If your output is always a JSON object with 12 specific fields, a fine-tuned model follows this reliably. Prompting does too, but with variance.
Domain-specific language. A model fine-tuned on your industry’s terminology, your company’s voice, or your codebase’s conventions will produce output that feels native to that context.
Reduced inference token cost. If you’re currently spending 800 tokens per call on system prompt instructions that could be baked into the model’s weights, fine-tuning eliminates that overhead permanently.

What fine-tuning does not buy you: up-to-date knowledge. A fine-tuned model knows what it knew before, plus the patterns in your training data. If your training data doesn’t include the latest information, the model will still hallucinate about things it doesn’t know. Fine-tuning is not a replacement for RAG when the problem is knowledge currency.

The Real Cost Picture

This is where most articles give you useless abstractions. Here are actual numbers.

Prompting: Your only cost is inference — tokens in, tokens out. For GPT-4o, that’s $2.50/M input tokens and $10/M output tokens. For Claude Sonnet 4.5, it’s in a similar range. A well-constructed 800-token system prompt on 1M calls costs $2,000 in system prompt tokens alone before you’ve even sent a user query. At scale, that’s not trivial.

RAG infrastructure: You need an embedding pipeline, a vector store, and a retrieval service. Minimal setup: a managed vector DB (Pinecone, Weaviate, Qdrant) runs $70–$300/month at entry level. Embedding models cost roughly $0.02–$0.13/M tokens. The hidden cost is engineering time — a real production RAG pipeline with proper chunking, reranking, and fallback strategies is 2–6 weeks of senior engineering work to do correctly. Most teams underestimate this by half.

Fine-tuning: The training run is often cheap now. LoRA (Low Rank Adaptation) fine-tuning on Llama 3.2 8B with 1,000 examples runs $5–$15 in cloud GPU time. QLoRA on a Mac M2 with 32GB+ is effectively free beyond electricity. OpenAI’s fine-tuning API charges per token and handles all infrastructure — a practical job on a few thousand examples costs $50–$200.

The part teams consistently miss: inference deployment for self-hosted fine-tuned models. If you fine-tune an open-source model and need to self-host it, a 7B model running 24/7 on a cloud GPU costs $2,000–$4,000/month. A 13B model runs $4,000–$7,000/month. The training run was cheap. The infrastructure to serve the result is where the real cost lives. If you’re fine-tuning via OpenAI’s API, this isn’t your problem — they serve it. If you’re fine-tuning open-weight models for data sovereignty or cost reasons, build that hosting cost into your ROI calculation from day one.

The Decision Framework

Start here: what is the problem type?

Knowledge problem (the model doesn’t know something it needs to know):
→ RAG if the knowledge changes frequently, is large volume, or needs auditability.
→ Fine-tuning if the knowledge is stable, relatively small, and deeply domain-specific.
→ Prompting alone won’t fix this beyond what few-shot examples can teach.

Behavior problem (the model doesn’t do what you need by default):
→ Prompting first — always. The ceiling is higher than you think.
→ Fine-tuning if prompting can achieve it but the cost of prompting at scale is too high, or if consistency is critical and variance in prompted behavior is unacceptable.
→ RAG won’t help here. You can’t retrieve your way to consistent output format.

Both (the model doesn’t know your domain and doesn’t behave like your domain’s expert):
→ Fine-tuning + RAG. The benchmark data backs this up: a 2024 Microsoft/academic paper showed +6 percentage points accuracy improvement from fine-tuning alone on a domain-specific task, with an additional +5 points when RAG was layered on top. They’re additive, not competing.

The indie dev filter

If you’re building a side project or an early-stage product with unknown PMF:

Don’t fine-tune yet. The iteration cost is too high. Fine-tuning commits you to a dataset and a model version. Your product will pivot before you’ve exhausted what prompting can do. You’ll also need evals before fine-tuning is meaningful — and building good evals for a product that’s still finding its shape is premature optimization.

Start with prompting. Use RAG if you genuinely have a knowledge retrieval problem. The right sequence is:

Prompt engineering until you hit a real ceiling
Add RAG if the ceiling is knowledge-related
Add fine-tuning when behavior consistency becomes a business requirement, not a preference

The point at which fine-tuning makes sense for an indie product is usually when you have enough users to tell you what the consistent failure mode is, enough data to build a training set from, and enough volume to justify the infrastructure. Most projects never reach that point before some other problem kills them.

The enterprise/fintech filter

If you’re in a regulated industry or building internal tooling at scale, the calculus shifts significantly:

Data privacy changes everything. If your training data contains customer PII, trade data, or anything that can’t leave your infrastructure, you’re not fine-tuning via OpenAI’s API. You’re fine-tuning open-weight models on your own infrastructure or through a vetted vendor with proper DPA agreements. Factor in the infrastructure cost accordingly.
Auditability favors RAG. When a compliance officer asks “why did the model say that?”, a RAG system can show the retrieved source documents. A fine-tuned model cannot — the knowledge is baked into weights that can’t be inspected. For use cases like compliance Q&A, policy lookup, or document-grounded advice, RAG is the correct architecture specifically because of this traceability property.
Consistency requirements favor fine-tuning. If you’re generating structured reports, classification outputs, or extractions that go into downstream systems, the variance in prompted behavior becomes an engineering liability. At 10M calls a month, a 0.5% inconsistency rate in output format is 50,000 broken records. Fine-tuning on format compliance is a legitimate engineering choice at that scale.
Update frequency is a real axis. Regulations change. Internal policies change. Product documentation changes. A fine-tuned model is frozen at its training data timestamp. Every time your knowledge base changes, you need a retraining cycle — which is a human-process cost, not just a compute cost. RAG updates by updating documents in the vector store. For domains with high knowledge volatility, the operational flexibility of RAG is a competitive advantage.

What Fine-tuning Is Actually Good For (That Most Articles Get Wrong)

The marketing answer for fine-tuning is “domain adaptation.” The practical answer is more specific.

Fine-tuning excels at teaching format, not facts. The use cases where fine-tuning produces outsized gains are structured output, classification, and task-specific generation — not factual recall. The model already knows things. Fine-tuning is best at changing how it expresses what it knows.
LoRA changed the game. LoRA (Low-Rank Adaptation) fine-tunes only 0.1–1% of the total model parameters by injecting small trainable rank-decomposition matrices into the transformer layers. The frozen base weights stay unchanged. At inference time, the adapter merges into the base weights — zero latency penalty, zero additional memory. QLoRA adds 4-bit quantization to the frozen base weights, which means a 7B model that normally requires 100+ GB of VRAM to fully fine-tune can be QLoRA fine-tuned on a $1,500 RTX 4090. Full fine-tuning of a 7B model costs ~$50,000 in H100 compute. QLoRA on an RTX 4090 costs $1,500 and gets within 80–90% of full fine-tune quality on most benchmarks.
For almost everyone reading this, LoRA or QLoRA is the only fine-tuning approach worth considering. Full fine-tuning is for organizations with GPU clusters and model teams.
Direct Preference Optimization (DPO) is worth understanding separately. DPO is a variant of fine-tuning that trains on preference pairs — “given this prompt, response A is better than response B.” It’s how you teach style, tone, and judgment rather than just format. If your problem is “the model’s outputs are technically correct but don’t sound right for our brand or context,” DPO is the tool, not standard supervised fine-tuning.

The Trap Nobody Warns You About

Here’s the decision most teams make: “Prompting isn’t good enough, let’s fine-tune.”

Here’s what they actually had: a prompting problem they hadn’t solved, and they thought fine-tuning would solve it for them.

Fine-tuning amplifies what the model already tends to do — it doesn’t fix fundamentally broken reasoning or knowledge gaps from the base model. If your prompting isn’t working because the instructions are ambiguous, the output format is under-specified, or you haven’t provided the right few-shot examples, fine-tuning on those same ambiguous examples will bake the ambiguity into the model’s weights permanently.

The prerequisite for fine-tuning that almost no one follows: you need working evals first. An eval is a systematic way to measure whether the model is doing what you want — a test suite with inputs, expected outputs, and a scoring function. Without evals, you cannot tell if your fine-tuning improved anything. You’re running training runs blind, iterating on vibes.

Before you fine-tune anything, you should be able to answer: “What metric improves? By how much? How do I measure it?” If you can’t answer that, you’re not ready to fine-tune.

The Actual Summary

Prompting RAG Fine-tuning What it changes Task instructions Knowledge access Model behavior Time to first result Hours Days–weeks Weeks Knowledge currency Static (in prompt) Real-time Frozen at training Auditability High High Low Inference cost Token overhead per call Token overhead per call + retrieval Lower per-call token cost Update cost Edit a text file Update documents Retrain + redeploy Privacy fit API fine Depends on vector DB On-prem possible Right for Most things, start here Knowledge retrieval problems Behavior consistency at scale

The ordering is also the sequence. Almost every production AI system that uses fine-tuning or RAG should have started with prompting, validated the problem, and escalated deliberately. The teams that skip straight to fine-tuning because it sounds more serious usually spend six weeks and significant compute to end up with a model that performs worse than a well-prompted baseline they never tried.

Start with prompting. Add RAG when knowledge is the gap. Fine-tune when behavior consistency is the business constraint. Run evals throughout — without them, every architecture debate is just vibes.

Under The Hood covers the mechanics that the AI product ecosystem hand-waves. If this was useful, forward it to whoever on your team is about to spin up a fine-tuning job without an eval in sight.

Thanks for reading Under The Hood! This post is public so feel free to share it.