Start from the task, not the leaderboard. Write down what good output looks like, build a small test set, and try two or three models across price tiers on it. Choose the cheapest model that passes, then check latency, context length, data policy, and whether you need an open-weight model you can host yourself.
What people tell me
I am building a project that uses an LLM (a chatbot over documents, or a tool that summarises and classifies text). There are so many models now: big ones, small ones, open-source ones, and new releases every month. Everyone online has a favourite. I do not have much budget. How do I actually choose which LLM to use, and how do I avoid picking wrong?
A composite of the messages behind this question, with personal details left out.
Key takeaways
- Define the task and what good output means before comparing models.
- A small test set of 30 to 100 real examples beats any public leaderboard for your use case.
- Try models from different price tiers; smaller models are often good enough.
- Check cost per request, latency, context length, structured output and data policies.
- Keep the model behind one interface so you can switch later without rewriting the app.
Why "which is the best LLM?" is the wrong question
Models are good at different things, prices differ widely, and new versions appear every few months. The best model for a coding agent may be wasteful for classifying support tickets. So the useful question is: which model does my task well enough, at a cost and speed I can live with?
Step 1: Describe the task
Write two or three sentences: the input, the output, and what "good" means. For example: "Given a customer message in English or Urdu, return a category from a fixed list and a one-line summary, as JSON."
Step 2: Build a small test set
Collect 30 to 100 realistic inputs, including hard and messy ones, with the output you would accept. This is your own private benchmark. The AI evals cheat sheet explains how to score it.
Step 3: Try models across tiers
| Tier | When it fits |
|---|---|
| Small, cheap hosted models | Classification, extraction, short summaries, high volume |
| Large hosted models | Hard reasoning, long documents, coding, agents |
| Open-weight models you run | Privacy needs, offline use, fixed costs at scale, or learning |
Run the same test set on two or three candidates. Many projects find a smaller model passes, which cuts cost and latency.
Step 4: Check the practical constraints
- Cost: price per input and output token, multiplied by your expected usage.
- Latency: how long a response takes, especially for chat.
- Context length: whether your inputs fit, and whether quality holds on long inputs.
- Structured output and tool calling: whether the model reliably returns valid JSON or calls tools.
- Languages: test your actual languages, such as Urdu, Bangla or Arabic, not just English.
- Data policy: what the provider does with your data, and where it is processed.
- Hosting: for open-weight models, the hardware you need. Tools like Ollama make local testing easy.
Step 5: Keep it switchable
Put model calls behind one function or a gateway such as LiteLLM, so you can change models when prices or quality change. Rerun your test set whenever you switch.
Learn the trade-offs properly
The choosing LLMs roadmap and its cheat sheet go through these decisions step by step. RAG, fine-tuning or a better prompt: how to choose helps with the next decision after the model, and the GenAI app architecture cheat sheet shows where the model sits in a real app.
A simple comparison sheet
| Model | Pass rate on my test set | Average cost per request | Typical response time | Notes |
|---|---|---|---|---|
| Candidate A | ||||
| Candidate B | ||||
| Candidate C |
Fill it in with your own numbers. It turns a debate into a decision, and it makes a strong section for your README or a project interview.
Mistakes to avoid
- Choosing by hype. The newest, biggest model is often unnecessary.
- Testing with three examples. Small samples hide failures.
- Ignoring cost at scale. A cheap demo can become expensive with real users.
- Hard-coding one provider everywhere. It makes switching painful later.
If you are still stuck
If you are building with retrieval, read Do I need a vector database for my AI project?. For project ideas that practise this, see the learned LLM router and the Roman Urdu and Bangla LLM benchmark. If you want help with your project's design, join Sefism and, as a member, book a 1:1 session.
Was this answer helpful?
Read next
- RAG, Fine Tuning, or a Better Prompt: How to ChooseRAG vs fine tuning vs a better prompt: a practical order from prompting through retrieval, embeddings, reranking, tools and agents to fine tuning, with the signal that tells you to move to the next step.
- An AI Engineering Roadmap That Puts Things in the Right OrderAn AI engineering roadmap in seven stages, from tokens and prompting through RAG, evals, tool calling, MCP and agents, with a project to build at each step, what to skip, and a realistic timeline.
- JEPA vs LLM: Yann LeCun's World Models Explained for DevelopersWhat JEPA is, how it differs from next-token LLMs, what V-JEPA 2 and AMI Labs have actually shown as of 2026, the strongest arguments against it, and what a student or developer should learn now.
Your situation is not quite this one?
Members get written answers to their own questions, a roadmap built for them and feedback on their projects. Early access is open: sign up and your account is active shortly after. Prefer to talk? A free call works too.