Offline Coding Assistant on Consumer Hardware

The problem: coding assistants that your code cannot go to

Cloud coding assistants send your source code to somebody else's server. For a software house building for a foreign client under an NDA, a bank's in-house team, or anyone working on regulated code, that rules them out. Others simply cannot rely on them: flaky internet, load-shedding, or usage quotas that run out mid-sprint.

A local model keeps code on the machine, costs nothing per token and works offline. It is also smaller and less capable. Everybody states that trade-off. Almost nobody measures it on the hardware a student or a small company actually owns.

What already exists and the gap

• Continue is an open-source VS Code and JetBrains extension that connects the editor to any model, including local ones through Ollama.
• Tabby is a self-hosted completion server with an admin dashboard, aimed at teams.
• Ollama and llama.cpp make running open models on a laptop straightforward.
• Cloud tools such as GitHub Copilot and Cursor set the quality bar.

So the plumbing exists. The gap is honest evidence: how much worse is a 7B or 14B model on a normal laptop for real tasks, which tasks survive the downgrade, and what does it cost in latency and battery? A reproducible benchmark that a team can rerun on their own machine before deciding is what is missing.

What you build

• Editor extension: completion and "explain this" backed by a local model server.
• Code-aware retrieval: index the open project with Tree-sitter so the prompt gets whole functions and the definitions they call.
• Model switcher: several models and quantisations, selectable at runtime.
• Benchmark harness: runs every task through any model with identical prompts and context, then scores by running the project's tests.

Build the harness first, before the extension. Run it against a cloud model to establish your ceiling, then see how close local models get.

The technically interesting part

The hard problem is a fair, automatic comparison. Fifty representative tasks is harder than it sounds. Pull them from real repositories where a commit fixed a bug or added a small function and a test proves correctness, so scoring is a test run rather than a judgement call. Then hold everything constant except one variable: same prompt, same retrieved context, same scoring. This is where most comparisons of this kind fall apart.

Retrieval matters as much as model size. Splitting source every 500 characters cuts functions in half; syntax-aware retrieval of whole definitions can close part of the gap to a bigger model, and measuring that is a result in itself.

How to evaluate it

• Pass rate per model and quantisation on the 50-task benchmark, with the cloud model as the ceiling.
• Retrieval ablation: no context, naive chunks, Tree-sitter definitions.
• Latency to first token and total, peak memory, and battery drain per hour of use on the reference laptop.
• User study: five developers use the local assistant for a real task for an hour and rate usefulness.
• Reproducibility: someone else clones the repository, runs one command and reproduces your table on their machine. That is the bar.

Suggested plan

1. Weeks 1 to 4: choose reference hardware, collect and validate 50 tasks, build the scoring harness.
2. Weeks 5 to 8: run cloud and local baselines with no retrieval.
3. Weeks 9 to 14: Tree-sitter indexing and retrieval, rerun everything.
4. Semester 2, weeks 1 to 6: VS Code extension, model switching, network audit.
5. Weeks 7 to 10: quantisation study, battery and memory measurements, user study.
6. Weeks 11 to 14: report, recommendation and public release.

Path to a product

Two realistic paths. First, open source: publish the benchmark and harness so others add models, which makes you known in the local-model community and is a strong portfolio for AI tooling roles. Second, consulting and setup: software houses with NDA-bound clients will pay for a configured on-premise assistant on one shared GPU machine, with a support contract. Do not try to compete with cloud assistants on features.

Scope warning

• Do not train or fine-tune a model; choosing and measuring is the project.
• Do not build a chat product with accounts and history.
• Support one editor only.
• Do not quote benchmark numbers without the hardware they ran on.

Learn what you need on Sefism

• AI engineering roadmap : local models, retrieval and evaluation.
• AI engineering cheat sheet : quick reference for prompts, context windows and quantisation terms.
• VS Code Remote Development : how VS Code talks to processes outside the editor, useful for the shared-server stretch goal.
• Cursor : study the cloud experience you are benchmarking against.
• Is it bad to rely on ChatGPT or Cursor to write code : a useful lens for your user study questions.

Get help with this idea

The benchmark is the part examiners will judge, so the task selection and the fairness of the comparison deserve most of your planning. Tauseef can help you choose tasks, fix the reference hardware and scope the extension so it does not swallow the semester. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session