Project structure · AI and ML
GenAI RAG app (FastAPI) project structure
Opens with creditsFor developers turning a "chat with your documents" demo into an app people can rely on. A FastAPI backend where every layer has a home: ingestion workers, retrieval, prompts as versioned files, model clients with fallbacks and cost caps, guards, evals that gate CI, and tracing, so a wrong answer can be traced to the step that caused it.
Curated by Tauseef FayyazIntermediate229 files and foldersPython 3.14, FastAPI 0.142, Pydantic 2.13Checked October 2026How it grows
Each stage opens the explorer there
- 1Day 1What the scaffold and your first hour give you19 files and folders
- 2GrowingWhat a real app with features and tests adds+107 files and folders
- ProductionWhat running it for real users adds+62 files and foldersOpens with credits
- ExpertWhat a team needs to keep it healthy for years+41 files and foldersOpens with credits
How it is organised
- One folder per layer of the request: api (HTTP), services (business rules), db (Postgres rows), rag (find the right text), llm (talk to models), guards (check what goes in and out), observability (see what happened). A route never calls a model SDK directly.
- The orchestrator (`rag/answer.py`) lives outside the routes, so the API, the eval runner and the load test all call the exact same function.
- Prompts are files with a version in the name, listed in one manifest, reviewed like code and logged with every answer, so any answer can be traced to the exact prompt text.
- Ingestion is a separate path from answering: documents are parsed, chunked and embedded by a background worker, never inside a chat request.
- Evals live in the repo next to the prompts they test and gate CI whenever prompts, retrieval or models change. Unit tests never call a real model.
- Model ids, thresholds, prices and limits come from typed config read from the environment; switching a model never needs a code change. Permissions are enforced in code and in the retrieval query, never only in the prompt.
- Start as one FastAPI service plus one worker with clear folders, and enforce the layer rules with import-linter once the team grows. Split into separate services only when a part needs to scale or deploy on its own.
At the top of the project
Open one in the explorer for the full story
- rag-app/
- .git/The Git repository uv init creates for you.
- .venv/The virtual environment uv creates on the first `uv add`, `uv sync` or `uv run`.
- .claude/Claude Code project settings and rules, committed so the whole team shares them.1 inside
- .vscode/Shared editor settings, recommended extensions and debug configurations.3 inside
- alembic/Database migrations for Postgres, created by `uv run alembic init alembic`.3 inside
- app/The FastAPI application package: every module the running API and the worker import.15 inside
- data/Local working data for ingestion: source files, processed chunks and the index manifest.4 inside
- docs/Short Markdown docs for people and coding agents: how the system works and how to operate it.2 inside
- evals/The evaluation suite: golden datasets, metrics, judges, the offline runner and online scoring.6 inside
- frontend/An optional chat UI that streams answers and shows sources, kept in the same repo.
- scripts/Operational commands: seed the database and index, migrate the index, back it up, mine feedback.2 inside
- tests/pytest suites in layers: unit, API, integration with real services, guardrails, then contract and load.4 inside
- htmlcov/The HTML coverage report written by `pytest --cov --cov-report=html`.
- .editorconfigIndentation, line endings and final newlines for every editor.
- .python-versionPins the Python version uv uses for this project.
- .gitignoreFiles Git must never track: .venv, caches, .env, local data and eval results.
- README.mdHow to run, ingest, evaluate and deploy the app.
- pyproject.tomlProject metadata, dependencies and tool settings: FastAPI entrypoint, pytest, coverage, Ruff and mypy.
- uv.lockExact versions of every package, written by uv.
- .envYour local API keys and settings, read by app/config.py.
- .env.exampleEvery environment variable the app needs, with placeholder values. Committed.
- alembic.iniAlembic's config file, written by `alembic init`: script location and logging.
- docker-compose.ymlLocal Postgres, Redis and Qdrant, then the API, worker and an OpenTelemetry collector.
- .pre-commit-config.yamlRuns Ruff, a lock check and a secret scan before every commit.
- justfileShort names for the commands everyone runs: setup, dev, test, evals, migrate, seed.
- AGENTS.mdInstructions for AI coding agents: commands, layout, conventions and rules for this repo.
- CLAUDE.mdClaude Code's project instructions; here a one-line import of AGENTS.md plus anything Claude-specific.
Learn more on Sefism
Roadmaps, cheat sheets and projects that go with GenAI RAG app (FastAPI).
- Roadmap· PythonAI engineering roadmapBuild products on LLMs and generative AI
- Roadmap· PythonBackend development roadmapAPIs, databases, auth, deploys
- RoadmapChoosing an LLM and AI stack roadmapModels, hosting and tools for your use case
- Roadmap· App and agent securityAI security roadmapSecure LLM apps and agents, test models safely
- Cheat sheetGenAI app architecture cheat sheetHow a production GenAI app fits together: request flow, layers, RAG, agent loop, memory, caching, guards, evals, tracing, folder layout and launch checklist.
- Cheat sheetAI evals cheat sheetEvals for LLM apps and agents: golden datasets, metrics, LLM-as-judge and its biases, RAG and agent metrics, offline and online evals, CI, human review, tools.
- Cheat sheetAI engineering cheat sheetBuild products on LLMs and generative AI.
- Cheat sheetLLM APIs cheat sheetBuilding on OpenAI, Anthropic and Gemini APIs: prompts, streaming, structured output, tools, RAG, caching, cost, evals and safety.
- FYP idea· AdvancedSentence-Level Groundedness Checker for RAG Answers with Human-Calibrated ThresholdsRAG apps cite sources, but a citation does not prove the sentence next to it is supported. Build a drop-in checker that splits each answer into claims, checks every claim against the retrieved passages with a natural language inference model and an LLM judge, highlights unsupported sentences in the UI, and sets its thresholds from human labels rather than guesses. Test it on English and Urdu answers.
See 11 moreShow fewer
- FYP idea· AdvancedPrompt Injection Regression Test Harness for RAG Support ChatbotsBuild a CI-friendly test harness that small product teams point at their own retrieval-based support chatbot before every release. It checks the bot against direct and indirect prompt injection, sensitive data leakage and unsafe output handling, mapped to the OWASP Top 10 for LLM Applications, and produces a scored report that shows whether the latest prompt, model or document change made things better or worse.
- FYP idea· AdvancedSandboxed Spreadsheet Analyst Agent for Small Business OwnersBuild an agent that answers a shop owner's questions about their messy Excel and CSV files by writing and running Python in an isolated sandbox, then returns a chart, a plain-language answer and the exact script it ran. The project is about safe code execution, cost limits, reproducibility and trajectory evals on real-world data, not about a chat interface.
- QuestionEveryone talks about agents, RAG, MCP and LLMs. How do I know what is actually worth learning?Learn the layer underneath the buzzwords: how an LLM call works, how to give it the right context, how to connect it to tools, and how to measure whether the result is good. Agents, RAG and MCP are patterns built from those pieces. Filter every new term by whether it solves a problem you actually have.
- QuestionHow do I choose which LLM to use for my project?Start from the task, not the leaderboard. Write down what good output looks like, build a small test set, and try two or three models across price tiers on it. Choose the cheapest model that passes, then check latency, context length, data policy, and whether you need an open-weight model you can host yourself.
- QuestionWhat does an AI evals engineer do?An AI evals engineer designs the tests that tell a team whether an AI system is good enough: the datasets, the scoring (code checks, human review, or a model as a judge), and the process that runs them on every change. It is a real and growing specialism inside AI engineering, and a strong angle for juniors because few people do it well.
- QuestionShould I become an AI engineer or stay a full stack developer?For most full stack developers this is not an either/or. AI engineering is largely full stack engineering plus models, evaluation and data. Keep your full stack strength, add AI features to real work, and move titles only when you have shipped AI work you can point to.
- Resource· GuideFastAPI Tutorial: User GuideThe official FastAPI tutorial and user guide builds an API one concept at a time: path and query parameters, request bodies with Pydantic validation, response models, error handling, dependency injection, OAuth2 with hashed passwords and JWT, SQL databases, bigger applications, background tasks and testing. One of the few framework docs that doubles as a genuinely good backend course, for Python developers building modern APIs.
- Resource· Tooluv DocumentationThe official docs for uv, the fast Python package and project manager from Astral. It replaces pip, virtualenv, pip-tools and pyenv for most projects, and its lockfile is what makes an ML project install the same way on your laptop, in CI and in a Docker image. Free, open source and quick to learn.
- Resource· GuideQdrant DocumentationThe official docs for Qdrant, an open source vector database written in Rust. Learn collections, points and payload filtering, then build semantic search and RAG retrieval on your own laptop or a free cloud cluster.
- Resource· GuideOpenTelemetry DocumentationThe official documentation for OpenTelemetry, the vendor-neutral open source standard for generating and collecting traces, metrics and logs. It covers the core concepts, per-language APIs and SDKs, zero-code instrumentation and the Collector. Instrument your services once with it and you can send the data to any of the more than ninety observability tools that support it, instead of locking into one vendor's agent.
- Resource· GuideOWASP Top 10 for LLM ApplicationsThe OWASP GenAI Security Project's list of the ten most serious security risks in LLM applications, currently the 2025 edition: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. Each entry explains the risk, examples and mitigations. It is a practical checklist before an LLM feature with tools or retrieval goes live.
Checked against 15 official sources, October 2026. If something has changed, the docs win.Show sourcesHide sources
- FastAPI: Bigger Applications, multiple files
- FastAPI: Settings and environment variables
- FastAPI: Testing dependencies with overrides
- uv: Creating projects (uv init)
- uv: Using uv in Docker
- uv: Using uv in GitHub Actions
- Alembic tutorial
- Celery: First steps
- Qdrant: Local quickstart
- testcontainers-python
- OpenTelemetry semantic conventions for generative AI
- OWASP Top 10 for LLM Applications
- import-linter
- Claude Code: CLAUDE.md, AGENTS.md and .claude/rules
- AGENTS.md
Best practices and checklists for this stack
Picked from the other libraries by shared topics.
Best practices
See all- Best Practices for Building LLM AppsHow to build LLM apps and agents that hold up in production: prompts, evals, prompt injection, tool permissions, cost, logging and privacy.
- Prompt Engineering Best PracticesHow to write prompts that work the same way on the hundredth input as on the first: clear instructions, structure, examples, reasoning, output control and testing.
- Best Practices for AI Coding AssistantsHow to get reliable code from AI coding assistants and agents: instruction files, task prompts, verification, review, security, dependencies and session hygiene.
Checklists
See all- RAG Production ChecklistA RAG production checklist: chunking, embeddings, hybrid search, reranking, per-user permissions, citations, evals, re-indexing and cost, with how to verify each check.
- LLM Evaluation ChecklistAn LLM evaluation checklist: success criteria, eval datasets, code and model graders, judge calibration, agent evals, safety tests and production monitoring, with how to verify each check.
- AI Feature Launch ChecklistAn AI feature launch checklist: use case fit, evals, UX for errors, disclosure, human override, abuse filters, rate limits, cost caps, prompt privacy, monitoring and rollback.
Work with me
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com