The AI Engineering Words You Keep Hearing, Explained Properly

The short answer: AI engineering terms are easier than they sound

Most AI engineering terms describe ideas that are not difficult. What makes the field feel impenetrable is that every explanation assumes you already know eight other words, and the definitions you find are either marketing copy or a research paper.

So this is the glossary I wish someone had handed me: tokens, context windows, embeddings, RAG, fine tuning, tool calling, MCP, agents, evals and the rest, in plain language, with what each one is genuinely good for and where it will let you down. I have grouped them in the order they build on each other, so if you read top to bottom, each term only relies on the ones above it.

I use these words every working day building LLM features into products, and I also hear students use them in ways that are slightly wrong in important ways. "We fine tuned it on our docs" is the classic one. By the end of this post you will know why that sentence should worry you.

AI engineering terms at a glance

The foundations: tokens, models and context

Token

A token is a chunk of text, usually a word or part of a word. Models do not read characters, they read tokens. OpenAI's rule of thumb for English is roughly four characters per token, or about 75 words per 100 tokens. Other languages can have a quite different ratio, so if your users write in Urdu or Arabic, count real samples rather than trusting the English rule.

You care for three practical reasons. It is the unit you are billed in. It is the unit context limits are measured in. And it explains a class of strange behaviour, like a model miscounting the letters in a word, because it never saw the letters.

Large language model (LLM)

A model trained on an enormous amount of text to predict the next token. Everything else it appears to do, from answering questions to writing code, is built on that one capability.

What it is good for: drafting text, writing and explaining code, answering questions, transforming data from one format to another.

Where it lets you down: it will state something false with exactly the same confidence as something true. There is no tone shift to warn you.

Hallucination

The name for exactly that failure: fluent, confident output that is not true. An invented function, a citation to a paper that does not exist, a wrong number. It is a property of how these models generate text, not a bug about to be fixed, so your systems and habits should assume it permanently.

Context window

Everything the model can see when it answers: the instructions, the conversation so far, any documents you added, the tool results. It is measured in tokens and it is finite.

Bigger windows do not mean you should fill them. The Lost in the Middle paper found models use information at the start and end of a long context better than information buried in the middle. Everything in the window competes for attention.

Temperature

A setting that controls how random the model's choices are. Low temperature gives more predictable, repeatable output, which suits extraction and code. Higher temperature gives more varied output, which can suit brainstorming. It does not make the model more or less "creative" in any deep sense; it changes how adventurously it picks among likely next tokens.

Talking to the model: prompts and context

Prompt engineering

Writing the instruction well: being specific about the task, the format you want back, and the constraints that matter, often with two or three examples (called few shot prompting).

What it is good for: it is free, it takes minutes, and it fixes more problems than people expect. Try it before anything more elaborate.

Where it lets you down: carefully tuned prompts can behave differently on a new model version, so anything you depend on needs a test rather than a hope.

System prompt

The standing instruction that sits above the conversation and sets the model's role, rules and tone for every turn. Users usually do not see it. It is where most of your product's behaviour lives.

Context engineering

The bigger idea that prompt engineering sits inside. Not just what you ask, but everything the model can see when it answers. Anthropic describes it as curating the smallest possible set of high signal tokens that maximise the chance of the outcome you want.

Once your system is more than a single question and answer, this is where most of your quality comes from: which documents you retrieve, how much conversation history you keep, which tools you expose and how their results are summarised.

Structured output

Constraining the model to return data in a schema you define, so your code can parse it reliably instead of hoping the JSON is well formed. If you are extracting data rather than producing prose, use it. It removes an entire category of parsing bugs.

Giving the model knowledge: embeddings, vectors and RAG

Embeddings

A model converts a piece of text into a list of numbers positioned so that things with similar meaning end up near each other.

What it is good for: searching by meaning rather than keyword. A search for "how do I cancel" can find a document titled "ending your subscription", which keyword search would miss.

Where it lets you down: the numbers are not interpretable. When retrieval returns something irrelevant there is no line of reasoning to inspect, and debugging becomes experimental.

Vector database

Somewhere to store embeddings and search them quickly. Comparing a query against every stored item does not scale, so these systems use approximate nearest neighbour search: fast, and it occasionally misses a genuinely good match. For many projects, pgvector inside an ordinary Postgres database is enough.

Chunking

Splitting documents into pieces before embedding them. It sounds like a detail and it is one of the biggest levers on quality: split a table or an argument in the wrong place and the right piece never matches the question.

RAG (retrieval augmented generation)

Before answering, look things up in your own documents, put what you found into the context, and answer from that.

What it is good for: questions about information the model was never trained on: your documentation, your policies, your data. Update a document and the answers change immediately, with no retraining.

Where it lets you down: if retrieval fetches the wrong thing, the model answers confidently from the wrong thing. Most RAG systems that give bad answers have a retrieval problem, not a model problem, and people spend weeks tuning prompts before checking.

Reranking

A second pass over retrieved results using a slower, more accurate model, to reorder them before they reach the context window. Cheap to add, and it often improves answers more than switching to a larger model. In Anthropic's contextual retrieval experiments, adding reranking to their improved retrieval cut retrieval failures by 67% against the baseline.

Changing the model itself: fine tuning

Fine tuning

Continuing to train a model on your own examples so it behaves a particular way by default.

What it is good for: consistent format, a specific tone, a narrow task done the same way every time. It changes behaviour more reliably than instructions do.

Where it lets you down: it teaches behaviour, not facts. If your goal is for the model to know your latest pricing, fine tuning is the wrong tool and RAG is the right one. OpenAI's guide to optimising LLM accuracy draws the same line: context for missing knowledge, fine tuning for inconsistent format, tone or reasoning. I go through this decision step by step in RAG, fine tuning or a better prompt.

Letting the model act: tools, MCP and agents

Tool calling (function calling)

Giving the model a set of functions it can request, so it can look something up or take an action instead of only producing text. The model asks, your code runs the function, and the result goes back into the conversation.

Where it lets you down: models fail at tools more often than at reasoning, and it is usually the tool's fault. A vague parameter name or unclear description produces wrong calls all day. Anthropic's guide to writing effective tools for AI agents suggests describing each tool as you would to a new hire.

MCP (Model Context Protocol)

An open standard for exposing tools and data to AI applications, so you write an integration once and any compatible client can use it. It started at Anthropic and in December 2025 was donated to the Linux Foundation's new Agentic AI Foundation, alongside projects from Block and OpenAI.

Where it lets you down: the security story is still maturing. Think carefully before exposing any tool that can write, delete or spend. MCP for Beginners is a gentle free introduction.

AI agent

A model that decides its own steps: plan, call a tool, look at the result, decide what to do next, repeat until done. Anthropic's Building Effective Agents contrasts this with a workflow, where your code decides the steps and the model fills them in.

What it is good for: tasks where the steps depend on what you find along the way.

Where it lets you down: errors compound. A small mistake at step two becomes a confidently wrong answer at step nine, and agents cost more. The useful skill is knowing when a plain script would have worked. The six common shapes are in six ways to structure work for an AI agent.

Multi agent system

Several agents, usually one coordinating others. Powerful for broad research style tasks and very expensive: Anthropic reported its multi agent research system used about fifteen times the tokens of a normal chat. Reach for it last.

Keeping it working: evals, guardrails and observability

Evals

Testing for AI systems: a set of inputs, criteria for a good output, and a score you track over time.

This is the least exciting term here and the one that separates systems that improve from systems that merely change. Without evals you cannot tell whether last week's prompt edit helped. Start with twenty to fifty real examples. Hamel Husain and Shreya Shankar call reading real outputs and categorising failures, known as error analysis, the most important activity in evals.

LLM as a judge

Using one model to grade another's output against criteria. Useful for things code cannot check, like tone or helpfulness, but only once you have checked the judge agrees with your own labels on a sample.

Guardrails

Checks around the model that block unsafe, malicious or out of scope input and output. Necessary once real users are involved, and not sufficient on their own. Treat them as one layer, not the whole security model.

Prompt injection

An attack where text the model reads, from a user, a web page or a document, overrides your instructions. It is number one on the OWASP Top 10 for LLM applications. Any system that reads untrusted content and can take actions needs to assume it will happen.

Observability

Tracing what actually happened: which documents were retrieved, what the prompt looked like after assembly, which tools ran, how long each step took and what it cost. The first thing to add when something behaves strangely, and the thing most people add last.

Caching

Reusing work you have already paid for. Prompt caching lets a provider reuse a long, repeated prefix like a system prompt or document set more cheaply. Response caching skips the model entirely for repeated questions.

What order to learn AI engineering terms in

If you are starting: tokens, context windows and prompting, then RAG, because RAG is where most real applications live.

After that: evals, before anything else, because everything past this point involves changes you need to be able to measure.

Then: tool calling, structured output, MCP and observability. That set covers most production systems.

Agents, fine tuning and multi agent systems come last, not because they are hardest to understand, but because they are the easiest to reach for when something simpler would have done the job.

That is exactly the order of the Sefism AI engineering roadmap, and I explain the reasoning in an AI engineering roadmap that puts things in the right order. If you would prefer a guided course that builds these foundations step by step, Grokking Modern AI Fundamentals covers them well, and Grokking Prompt Engineering for Software Engineers goes deeper on the prompting and context side.

Not sure which of these are worth your time right now? I answered that in which AI topics are worth learning: agents, RAG, MCP, LLMs.

Learn it on Sefism

• AI engineering roadmap and the TypeScript edition
• AI engineering cheat sheet, every term above on one page
• Grokking Modern AI Fundamentals
• Deep Dive into LLMs by Andrej Karpathy, for what is under the hood
• Generative AI for Beginners, a free hands on course
• AI resource library

FAQ

What is the difference between RAG and fine tuning?

RAG gives the model information at the moment it answers by retrieving documents. Fine tuning changes the model's default behaviour by training it on examples. Use RAG for knowledge and fine tuning for consistent format, tone or task behaviour.

What are tokens in AI?

Tokens are the chunks of text a model reads and writes, usually whole words or parts of words. In English one token is roughly four characters. You are billed per token and context limits are measured in tokens.

What is MCP in AI?

The Model Context Protocol is an open standard for connecting AI applications to tools and data. You build a server once and any compatible client, from chat apps to coding assistants, can use it.

What is the difference between an AI agent and a chatbot?

A chatbot answers messages. An agent decides and takes its own steps toward a goal, calling tools, reading results and choosing what to do next, until the task is done.

What are evals in AI engineering?

Evals are tests for AI output: a set of inputs, a definition of a good answer and a score. They tell you whether a prompt, model or retrieval change actually improved the system.

Ask your own question

If your situation is different from the ones above, the career questions library has honest answers to the questions students and engineers ask me most.

Want a learning roadmap built around you, written answers to your own questions and a place to discuss your FYP? Create your free Sefism account. Members can also book a one to one session when they want to talk it through.

Stay in touch

• LinkedIn: where I share roadmaps, career advice and honest takes on hiring. Follow, or message me.
• X (Twitter): short practical thoughts on engineering and careers.
• Instagram: quick tips and a more personal side.
• Sefism Discord: meet other students and engineers working through the same job market and share what is working.
• Sefism on YouTube: walkthroughs and talks.