RAG, Fine Tuning, or a Better Prompt: How to Choose

The question people actually have

You have an idea that involves an AI model. Now you have to decide how to build it, and every article you read recommends whatever that article is about.

The decision is more boring than the discourse suggests. Most of it comes down to matching the problem to the cheapest technique that solves it, and the cheapest technique is nearly always further down the list than people assume.

Here is the order to work through, and the signal that tells you to move to the next step.

Start with the prompt, always

Before anything else, write a better instruction. Be specific about the task. State the output format. Give two or three examples of what good looks like. Say what to do when the input is unclear.

This costs an afternoon and no infrastructure, and it fixes a genuinely surprising proportion of problems.

Move on when: you have iterated properly and the failures are about information the model does not have, or behaviour it will not hold consistently. Not when you are bored of iterating.

The model does not know your data, so retrieve it

Your policies, your documentation, your product details, anything that happened after training. The model has never seen any of it and no amount of prompting will conjure it.

The shape is: find the relevant documents, put them in the context, ask the question against them. That is RAG, and it is where most real applications live.

Choose this when: answers must be based on information you control, and that information changes. Updating a document changes the answers immediately, with no retraining and no deployment.

The trap: when a RAG system answers badly, the instinct is to rewrite the prompt. Check retrieval first. Print the documents that were actually fetched. In my experience most of these problems are the retrieval step handing over the wrong material, and no prompt fixes that.

When keyword search is not finding it, use embeddings

Someone asks how to cancel, and your document is titled "ending your subscription". Keyword search finds nothing. Embeddings place the two near each other because they mean the same thing, and search by meaning starts working.

Choose this when: the words users type do not match the words your documents use, which is most of the time.

Keep in mind: keyword search is still better for exact things. Product codes, error numbers, names. The strongest retrieval systems run both and combine the results rather than picking a side.

Once you have many vectors, you need somewhere to put them

A vector database stores embeddings and searches them quickly, using approximate search that is fast and occasionally misses a good match.

Choose this when: you have enough documents that comparing against all of them is too slow. Below a few thousand, a plain library and a loop is genuinely fine, and reaching for infrastructure early is a common way to add operational work you did not need.

If the right answer is in the results but not at the top, rerank

Retrieval gives you twenty candidates. The best one is eleventh. The model only reads the first five.

A reranker is a slower, more accurate model that reorders those twenty before they reach the context window.

Choose this when: you inspect failures and find the correct document was retrieved but ranked too low. This is one of the highest value additions per hour of work in the whole stack, and it is routinely skipped in favour of upgrading the model.

When you need data rather than prose, constrain the output

If your code has to parse the answer, define a schema and make the model return that shape.

Choose this when: the output feeds anything other than a human reading it. It removes a whole category of parsing failures and it costs almost nothing.

When the model needs to do something, give it tools

Look up an order, query a database, send an email, call an API. The model requests the function, your code runs it, the result goes back.

Choose this when: the task needs current information or a real action, not just text.

The trap: tool failures are usually your fault, not the model's. If a parameter is named q with no description, expect wrong calls. Write tool descriptions as though for a new colleague who cannot ask follow up questions, because that is exactly the situation.

When the steps depend on what you find, use an agent

A fixed pipeline runs the same steps every time. An agent decides: look at this, then based on what it says, do that.

Choose this when: you genuinely cannot write the sequence in advance.

Be honest here. This is the most over reached for tool in the field. Agents are slower, cost more, and fail in ways that are hard to reproduce, because errors compound across steps. If you can write the steps down, write the steps down. A boring pipeline that works beats an agent that works most of the time.

When behaviour will not hold, fine tune

You need a specific format or tone every single time, and prompting gets you to ninety per cent but not consistently.

Choose this when: the requirement is about how it responds, not what it knows.

The most common mistake in the field: fine tuning to teach the model facts. It does not work well, it is expensive, and every fact update means retraining. If the goal is knowledge, the answer is retrieval. Fine tuning changes behaviour.

When cost or latency is the problem, route

Not every request needs your largest model. Classify the incoming request and send simple ones to a small fast model, hard ones to the capable one.

Choose this when: you have a real cost or latency problem with actual numbers attached. Not before, because it adds a component that can itself be wrong.

When requests repeat, cache

Identical or near identical requests do not need recomputing. Cache the answers.

Choose this when: you can see repetition in your traffic. This is ordinary engineering, and it applies here exactly as it applies everywhere else.

Before any of the above, build evals

I have put this last so it stays in your head. It should be second.

An eval is a set of test inputs, criteria for a good answer, and a score. Twenty examples in a spreadsheet counts.

Without it, every change on this page is a guess. You will swap a model, feel that it improved, and have no way to know. With it, you can answer whether reranking helped, whether the prompt edit was worth it, and whether the new model version broke something that used to work.

It is the least interesting thing in this article and the one that decides whether your system gets better over time or just keeps changing.

The summary

Prompt first. Retrieve when the model lacks information. Embed when wording varies. Rerank when the right answer is retrieved but buried. Constrain output when code reads it. Add tools when actions are needed. Use an agent only when the steps genuinely cannot be written in advance. Fine tune for behaviour, never for facts. Route and cache when you have measurements showing you need to.

And measure everything, or none of the above is a decision, it is a preference.