Applied LLMs: What We Learned From a Year of Building

What it is

What We Learned from a Year of Building with LLMs is a collaborative guide published in June 2024 by Eugene Yan, Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu and Shreya Shankar, practitioners who had built and advised on production LLM systems. It is free to read on its own website and is organised into three parts.

The three sections

Tactical: the nuts and bolts

• Prompting fundamentals: n-shot examples, chain of thought, and structured inputs and outputs.
• Retrieval-augmented generation: why document relevance matters, and why hybrid search (keywords plus embeddings) often beats embeddings alone.
• Workflow design: breaking tasks into smaller steps and making execution deterministic where possible.
• Evaluation: unit-test style assertions, LLM-as-judge and guardrails.

Operational: day-to-day concerns

• Watching for drift between development and production data.
• Regularly reading real inputs and outputs to catch new failure patterns.
• Using structured output so downstream code can rely on responses.
• Choosing smaller models when they are good enough.
• Involving designers early and building around the user's job to be done.

Strategic: long-term positioning

• Not investing in GPU infrastructure before product-market fit.
• Building systems around models rather than training from scratch.
• Going narrow rather than broad, and designing human-in-the-loop workflows.

Why it is unusually valuable

It is specific about failure. The central message is that the model is not the product; the system around it is. Durable advantages come from evaluations, guardrails, data flywheels and caching, not from access to a particular model. Very little writing in the field is this candid about what did not work.

Which parts to read first

• Evaluation, because building a reliable evaluation set is the single highest-leverage thing most teams skip.
• Look at your data, the operational habit of reading real samples, which is cheap and catches most problems early.
• The strategic advice on fine-tuning and infrastructure, which can save a team months of misdirected effort.

How to use it

Read it once end to end, then return to the section matching your current problem. It works well as a team reading list: each person takes a section and brings three changes the team should make. Students building an AI project can use the tactical section as a checklist before a demo.

Where it shows its age

It reflects lessons from 2023 to 2024. Models have become cheaper and more capable, and agent tooling has matured, so some specific techniques have evolved. The principles about evaluation, data and systems thinking have aged very well.

Common questions

Do I need production experience to benefit? No, but it lands hardest once you have shipped something and watched it fail in unexpected ways.

Is it technical? Moderately. The tactical section assumes familiarity with prompts, embeddings and retrieval.

How long is it? Long enough to read over a few sittings; the sections stand alone.

Best for: teams beyond the prototype stage, and engineers who want to build LLM features that hold up in production.