RAG, Fine Tuning, or a Better Prompt: How to Choose

The short answer on RAG vs fine tuning vs a better prompt

If you are weighing RAG vs fine tuning for an AI feature, here is the answer I give most people: start with a better prompt, add retrieval when the model lacks information, and fine tune only when behaviour will not hold consistently. Fine tuning changes how a model behaves. RAG changes what it knows. Prompting shapes both, costs an afternoon and needs no infrastructure.

That sounds too simple, and the reason it feels that way is that every article you read recommends whatever that article happens to be about. A vector database vendor tells you to embed everything. A fine tuning platform tells you to train. The real decision is more boring: match the problem to the cheapest technique that solves it, and the cheapest technique is nearly always further down the list than people assume.

I build LLM features into production software, and I have made most of the expensive mistakes in this post at least once. What follows is the order I now work through, and the specific signal that tells me to move to the next step.

RAG vs fine tuning vs prompt engineering at a glance

OpenAI's own guide to optimising LLM accuracy frames it the same way. It says you optimise for context (RAG) when the model lacks knowledge that was not in training, is out of date, or is proprietary, and you optimise the model itself (fine tuning) when outputs are inconsistent in format, tone or reasoning. It also says to squeeze as much as you can out of basic methods first, with an evaluation set as your baseline. That last point matters, so hold on to it.

Before anything else: build a small eval

I used to put this last. It belongs second, straight after your first prompt, because every decision on this page is a guess without it.

An eval is a set of test inputs, criteria for a good answer, and a score. Twenty to fifty real examples in a spreadsheet counts. Read the failures, group them, and you will usually find the problem is obvious: the model did not have the information, or it had it and ignored the format, or retrieval handed it the wrong page.

Without an eval you will swap a model, feel that it improved, and have no way to know. With one, you can answer whether reranking helped, whether the prompt edit was worth it, and whether the new model version broke something that used to work. AI Evals: Everything You Need to Know is the best free guide to doing this properly, and it is stage 4 of the AI engineering roadmap for exactly this reason.

Step 1: start with the prompt, always

Write a better instruction before anything else.

• Be specific about the task. "Summarise" is vague. "Summarise for a customer support agent in three bullet points, including the order number" is not.
• State the output format. Show the exact shape you want back.
• Give two or three examples. Few shot examples change results more than clever wording.
• Say what to do when the input is unclear. Otherwise the model guesses.

This costs an afternoon and fixes a surprising proportion of problems. The Anthropic prompt engineering documentation is a good free reference, and Grokking Prompt Engineering for Software Engineers is a structured course if you want to go deeper from an engineer's point of view.

Move on when: you have iterated properly and the failures are about information the model does not have, or behaviour it will not hold consistently. Not when you are bored of iterating.

Step 2: when to use RAG, because the model does not know your data

Your policies, your documentation, your product details, anything that happened after training. The model has never seen any of it, and no prompt will conjure it.

Retrieval augmented generation has a simple shape: find the relevant documents, put them in the context, ask the question against them. It is where most real applications live.

Choose RAG when: answers must be based on information you control, and that information changes. Updating a document changes the answers immediately, with no retraining and no deployment. You also get citations for free, which matters a great deal for trust.

Check this first: if your whole knowledge base is small, you may not need retrieval at all. Anthropic's contextual retrieval write-up notes that below roughly 200,000 tokens, about 500 pages, you can often put the entire knowledge base in the prompt, and prompt caching makes that affordable.

The trap: when a RAG system answers badly, the instinct is to rewrite the prompt. Check retrieval first. Print the documents that were actually fetched. In my experience most of these problems are the retrieval step handing over the wrong material, and no prompt fixes that.

Step 3: when keyword search misses, use embeddings

Someone asks how to cancel, and your document is titled "ending your subscription". Keyword search finds nothing. Embeddings place the two near each other because they mean the same thing, and search by meaning starts working.

Choose embeddings when: the words users type do not match the words your documents use, which is most of the time.

Keep in mind: keyword search is still better for exact things like product codes, error numbers and names. The strongest retrieval systems run both and merge the results. In Anthropic's tests, adding contextual embeddings plus keyword (BM25) search cut retrieval failures by 49% compared with plain embeddings.

Chunking matters more than people expect. Split a document mid table or mid argument and the right chunk never matches the question. Try a few chunk sizes and measure them against your eval.

Step 4: once you have many vectors, pick somewhere to store them

A vector database stores embeddings and searches them quickly, using approximate nearest neighbour search that is fast and occasionally misses a good match. That trade off is tunable.

Choose a vector store when: you have enough documents that comparing against all of them is too slow. Below a few thousand chunks, a plain library and a loop is genuinely fine. When you do need one, pgvector lets you keep vectors inside the Postgres database you probably already run, which is one less system to operate.

Step 5: if the right answer is retrieved but buried, rerank

Retrieval gives you twenty candidates. The best one is eleventh. The model only reads the first five.

A reranker is a slower, more accurate model that reorders those candidates before they reach the context window. Position matters too: the Lost in the Middle paper found models use information at the start and end of a long context better than information in the middle.

Choose reranking when: you inspect failures and find the correct document was retrieved but ranked too low. It is one of the highest value additions per hour in the whole stack. In the same Anthropic tests, adding a reranker on top took the reduction in retrieval failures to 67%. It is routinely skipped in favour of upgrading the model.

Step 6: when code reads the answer, constrain the output

If your code has to parse the answer, define a schema and make the model return that shape using structured output.

Choose this when: the output feeds anything other than a human reading it. It removes a whole category of parsing failures and costs almost nothing.

Step 7: when the model needs to act, give it tools

Look up an order, query a database, send an email. The model requests a function, your code runs it, and the result goes back.

Choose tools when: the task needs current information or a real action, not just text.

The trap: tool failures are usually your fault, not the model's. A parameter named q with no description produces wrong calls all day. Write tool descriptions for a new colleague who cannot ask follow up questions, because that is exactly the situation.

Step 8: when the steps depend on what you find, consider an agent

A fixed pipeline runs the same steps every time. An agent decides: look at this, and based on what it says, do that.

Choose an agent when: you genuinely cannot write the sequence in advance.

Be honest here. This is the most over reached for tool in the field. Anthropic's Building Effective Agents advises starting with simple prompts, optimising them with evaluation, and adding agentic systems only when simpler solutions fall short. Agents are slower, cost more and fail in ways that are hard to reproduce, because errors compound across steps. A boring pipeline that works beats an agent that works most of the time. I cover the shapes they take in six ways to structure work for an AI agent.

Step 9: when to fine tune an LLM, because behaviour will not hold

You need a specific format or tone every single time, and prompting gets you to ninety per cent but not consistently.

Choose fine tuning when: the requirement is about how the model responds, not what it knows. Consistent output structure, a house style, a narrow classification task done the same way at high volume, or getting a smaller, cheaper model to match a larger one on one specific job.

The most common mistake in the field: fine tuning to teach the model facts. It is expensive, it does not reliably work, and every fact update means retraining. If the goal is knowledge, the answer is retrieval.

Can you use RAG and fine tuning together?

Yes, and mature systems often do: a fine tuned model that reliably produces your format and tone, fed with retrieved documents for the facts, wrapped in a carefully written system prompt. Each technique does the job it is good at.

But combine them because your eval shows each one earning its place, not by default. OpenAI's accuracy guide includes a worked example where adding RAG on top of an already fine tuned model actually made results worse, because the extra context added noise to a problem that was really about consistency. More components is not the same as more accuracy.

Step 10: route and cache when you have numbers

Two ordinary engineering tools finish the list.

• Route when you have a real cost or latency problem with actual numbers attached. Classify requests and send simple ones to a small, fast model and hard ones to a capable one. Not before, because the router is one more thing that can be wrong.
• Cache when you can see repetition in your traffic. Identical or near identical requests do not need recomputing, and most providers now also offer prompt caching for long, repeated context.

A decision checklist for RAG vs fine tuning

When a system is not good enough, ask these questions in order:

1. Is my instruction specific, with a format and examples? If not, fix the prompt.
2. Is the failure about missing or changing information? Use RAG.
3. Is retrieval returning the right material? If not, fix chunking, add keyword search or rerank before touching the model.
4. Is the output going into code? Constrain it with a schema.
5. Does it need live data or actions? Add tools.
6. Is the failure about inconsistent behaviour after all of the above? Now consider fine tuning.
7. Did the change improve my eval score? If you cannot answer, you do not know yet.

If the vocabulary here is new, the AI engineering words you keep hearing, explained properly defines each term, and an AI engineering roadmap that puts things in the right order shows the order to learn them in. For the bigger picture of how these pieces fit into a full system, Grokking the AI System Design Interview is a good structured walkthrough.

Learn it on Sefism

• AI engineering roadmap, with stages on RAG, evals and tools
• AI engineering cheat sheet
• RAG Techniques, an open source collection of retrieval patterns
• Which AI topics are worth learning: agents, RAG, MCP, LLMs
• Grokking Prompt Engineering for Software Engineers
• AI resource library

FAQ

Is RAG better than fine tuning?

For knowledge, usually yes: RAG is cheaper, updates instantly and can cite its sources. For consistent behaviour like format, tone or a narrow task, fine tuning can be better. They solve different problems.

When should you fine tune an LLM instead of using RAG?

When the model already has the information it needs but will not respond consistently in the way you require, even after careful prompting with examples. Fine tuning is for how it responds, not what it knows.

Is prompt engineering enough for most applications?

For a surprising number, yes, especially when combined with a small eval set. Many teams jump to RAG or fine tuning before they have written a specific prompt with a format and examples.

Do I need a vector database for RAG?

Not always. Small knowledge bases can go straight into the prompt, and a few thousand chunks can be searched with a simple library. When you do need one, a Postgres extension like pgvector is often enough.

Can RAG and fine tuning be combined?

Yes. A fine tuned model can handle format and style while RAG supplies current facts. Add each piece only when your evaluation shows it improves results.

Ask your own question

If your situation is different from the ones above, the career questions library has honest answers to the questions students and engineers ask me most.

Want a learning roadmap built around you, written answers to your own questions and a place to discuss your FYP? Create your free Sefism account. Members can also book a one to one session when they want to talk it through.

Stay in touch

• LinkedIn: where I share roadmaps, career advice and honest takes on hiring. Follow, or message me.
• X (Twitter): short practical thoughts on engineering and careers.
• Instagram: quick tips and a more personal side.
• Sefism Discord: meet other students and engineers working through the same job market and share what is working.
• Sefism on YouTube: walkthroughs and talks.