Vector Search Engine Built From Scratch
Every retrieval system now sits on a vector database, and almost nobody using one can explain what happens inside it. Build one: store embeddings on disk, implement approximate nearest neighbour search yourself, and benchmark your recall and latency against an established library. This is a classic build your own X project with a modern subject, and it will teach you more about retrieval than any number of tutorials.
Advanced
Needs a team, real planning, and a supervisor who knows the area.1 semester · 1 to 2 students
7 to ship
3 optional extrasSuggested stack
What you should ship
- A storage layer that persists vectors and metadata to disk and survives restart
- Exact nearest neighbour search as a correctness baseline
- An approximate index implemented from the paper, such as HNSW or IVF, not from a library
- Metadata filtering combined with vector search in a single query
- A benchmark against a standard dataset reporting recall at k, queries per second and index build time
- A comparison against an established library on the same dataset and hardware
- A written analysis of where your implementation loses to the reference and why
If you have time left
- Incremental insertion and deletion without a full index rebuild
- Product quantisation to cut memory use, with the recall cost measured
- A second index type, so you can compare the two families directly
The problem
Searching a million vectors exactly means comparing against all of them, which is too slow. Every vector database therefore trades a little accuracy for a lot of speed, using an index that only looks at a promising subset.
Understanding that trade off properly is the difference between choosing a vector database sensibly and copying a configuration from a blog post.
What makes this a real project
- Read the paper, not the wrapper. HNSW is a graph where each node keeps a short list of neighbours across several layers. It is genuinely implementable from the paper in a semester, and doing so is the point.
- Measure recall honestly. You need exact search as a baseline, or you cannot tell how much accuracy your index is losing. Build that first.
- Filtering is the hard part. Combining a metadata filter with an approximate index is where real systems get complicated, because filtering after search can leave you with nothing. Handle it and you have covered ground most tutorials skip.
- Losing to the reference is fine. A tuned production library will beat you. Explaining precisely why, with measurements, is a better result than a vague claim of comparable performance.
Suggested approach
Start with exact search over in-memory vectors. Add persistence. Add the approximate index. Add filtering. Benchmark at every stage so you can show the progression.
Pick a standard dataset with published recall numbers so your results are comparable to something.
How you will know it worked
You can draw the recall against queries per second curve for your engine and for the reference library on one chart, and explain every gap between the two lines.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Every session is free; a few slots open each week.
Follow along
New writing, resources and project ideas land here first.