Sefism is free for learners. We are here to help you grow, one step at a time.Join for free
Sefism.
Other

Instruction File Linter for AI Coding Agents: AGENTS.md, CLAUDE.md, Cursor Rules and Copilot Instructions

Build a command-line tool and CI check that reads the instruction files AI coding agents load, AGENTS.md, CLAUDE.md, .cursor/rules and .github/copilot-instructions.md, and checks them against the real repository: commands that no longer exist, paths that moved, glob patterns that match nothing, rules that contradict each other across files, and files too long to be followed. Teams using several AI tools get one report that keeps their agents working from the truth.

Topics
Difficulty

Intermediate

A small team, or one strong student willing to learn something new.
Effort

1 semester · 2 students

Deliverables

7 to ship

4 optional extras

Suggested stack

TypeScript or Pythonremark or markdown-it-pyYAML frontmatter parsingGitHub ActionsAn LLM API (optional)
A suggestion, not a requirement. Swap anything for what you already know.

What you should ship

  • Parsers for AGENTS.md, CLAUDE.md with @imports, .claude/rules, .cursor/rules .mdc files and .github/copilot-instructions.md with .instructions.md files
  • Checks that every command mentioned exists in package.json scripts, a Makefile or on the PATH, and that every referenced path exists
  • Glob and applyTo pattern checks that report rules matching no files in the repository
  • Size budgets per file and per tool, with the lines that push a file over
  • Cross-file duplicate and conflict detection for rules that say opposite things
  • Text, JSON and SARIF output, and a GitHub Action that comments on pull requests
  • An evaluation on at least 30 public repositories that use these files, with precision of each check

If you have time left

  • An optional LLM pass that flags vague lines such as "write clean code" with a suggested rewrite
  • A --fix mode that updates renamed paths and scripts
  • A VS Code extension that shows findings inline
  • Rules to detect secrets or internal URLs accidentally written into instruction files

The problem: agents following stale instructions

Most teams that use AI coding tools now keep instruction files in their repositories: AGENTS.md, CLAUDE.md, rules in .cursor/rules or .github/copilot-instructions.md. The tools load these into every session, so they shape every change an agent makes. But nobody tests them. Scripts get renamed, folders move, a rule in one file contradicts a rule in another, and files grow until the tool's own docs say they will be followed less well. The agent then works confidently from instructions that are no longer true.

It is the same problem documentation always had, with a twist: this documentation is executed, by a model, on every task.

What already exists and the gap

  • markdownlint and similar tools check Markdown style, not whether the content matches the repository.
  • Claude Code has a /doctor prompt-audit command that reviews its own instruction files for outdated or conflicting guidance, inside one tool.
  • Cursor and VS Code show which rules are active in their own editors.
  • The AGENTS.md format gives teams one shared file, but no validator for what is inside it.

The gap is a tool-agnostic checker that runs in CI, understands every major tool's file layout, and checks the instructions against the actual repository.

Inspiration

The official docs for these tools all give the same advice: keep instruction files short, specific and current. Anthropic suggests keeping each CLAUDE.md under 200 lines, Cursor suggests keeping rules under 500 lines, and GitHub warns that some settings do not apply in every mode. Linters such as ESLint turned style advice into automatic checks; this project does the same for agent instructions.

What you build

  • Developers run lint-agents locally and get a list of broken commands, missing paths, dead globs and conflicts.
  • Teams add the GitHub Action, which comments on any pull request that breaks an instruction file or renames something an instruction refers to.
  • Maintainers get a JSON or SARIF report for dashboards and code scanning.

The technically interesting part

  • Parsing five formats. Markdown with frontmatter, @path imports in CLAUDE.md, .mdc rules with globs and alwaysApply, and .instructions.md files with applyTo.
  • Extracting claims. Find commands inside code spans and code blocks, paths and file names, and resolve them against package.json, Makefiles and the file tree.
  • Glob evaluation. Match every pattern against the repository and report patterns that match nothing or everything.
  • Conflict detection. Start with rules you can check exactly, such as two files naming different test commands, then add an optional model-based pass for semantic contradictions, and measure its false positives honestly.

How to evaluate it

  • Precision per check: on 30 or more public repositories, how many findings were real problems, judged by hand.
  • Recall on seeded errors: rename scripts and folders in test repositories and measure how many broken references the tool catches.
  • Speed: time to lint a large monorepo.
  • Usefulness: open issues or pull requests on a few public repositories with real findings and record the responses.

Suggested plan

  • Weeks 1 to 3: collect sample files from public repositories, write parsers for each format.
  • Weeks 4 to 7: command, path and glob checks, with tests.
  • Weeks 8 to 10: conflicts, size budgets, output formats, GitHub Action.
  • Weeks 11 to 13: evaluation on public repositories.
  • Weeks 14 to 15: optional LLM pass, report and demo.

Path to a product

Every team adopting AI coding tools has this problem and no tool for it. An open source CLI with a free GitHub Action can build a following; a hosted dashboard across many repositories, with organisation-wide rules, is a natural paid layer for larger companies.

Scope warning

Do not build your own AI coding agent, and do not try to judge whether every instruction is good advice. Check facts first: commands, paths, globs and exact conflicts. The optional model pass is a stretch goal.

Learn what you need on Sefism

Get help with this idea

The hard part is deciding which checks are exact and which are guesses, and proving precision on real repositories. Tauseef can help you choose the checks, design the evaluation and write it up. If you are new here, create a free account first.

Ideas and guidance, not finished projects

These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.

Not sure this one fits you?

I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.

Discuss this idea
Work with me

Stuck on something specific?

Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.

Book a free sessionSee all services

Follow along

New writing, resources and project ideas land here first.