Often not. For a student project or a small app, a vector extension on a database you already use (such as pgvector in PostgreSQL), or even an in-memory index, is enough. A dedicated vector database earns its place with very large collections, heavy query traffic, or features like advanced filtering and multi-tenancy at scale.
What people tell me
I am building a RAG chatbot over some documents and every tutorial uses a different vector database. Some are paid, some need a separate server, and I already use PostgreSQL or MongoDB for the rest of the app. Do I really need a separate vector database, or is it overkill for a project like mine?
A composite of the messages behind this question, with personal details left out.
Key takeaways
- A vector database stores embeddings and finds the most similar ones quickly.
- For thousands to many thousands of chunks, pgvector or your current database's vector search is usually enough.
- One database for app data and vectors means simpler code, backups and filtering.
- Consider a dedicated vector database for very large scale or specialised retrieval needs.
- Retrieval quality depends more on chunking, metadata and evaluation than on which store you pick.
What a vector database does
In retrieval-augmented generation (RAG), you split documents into chunks, turn each chunk into an embedding (a list of numbers), and store them. When a question comes in, you embed it and find the most similar chunks. A vector database is built to do that similarity search quickly, usually with approximate nearest neighbour indexes.
Your options, smallest first
| Option | Good for |
|---|---|
| In-memory index in your app | Demos, notebooks, a few thousand chunks |
| Vector extension in your existing database, such as pgvector for PostgreSQL | Most student and small production apps |
| Vector search in your existing managed database, such as MongoDB Atlas Vector Search | Teams already on that platform |
| A dedicated vector database | Very large collections, high query load, specialised features |
Why "use what you have" is usually right
- One system to run. Backups, access control and deployment stay simple.
- Filtering is easy. You often need "only this user's documents" or "only this course". With vectors next to your data, that is a normal query.
- Fewer moving parts to explain. In interviews and READMEs, a simple design you understand is better than a complex one you copied.
When a dedicated vector database makes sense
- Your collection is very large and growing fast.
- Similarity search is the main workload, with heavy traffic.
- You need features your current database does not handle well, such as specific hybrid search or index types at scale.
- You have measured, and the simpler option is too slow or too expensive.
The key word is measured. Start simple, then move when numbers say so.
What matters more than the store
Most RAG quality problems come from elsewhere: chunk size, missing metadata, poor queries, no re-ranking, and no evaluation. Build a small test set of questions with known answers and measure retrieval before you change infrastructure. The AI evals cheat sheet and RAG techniques help here.
Learn the design properly
The choosing databases roadmap covers vector storage alongside relational and document databases, and the GenAI app architecture cheat sheet shows where retrieval fits in a real app. The AI engineering roadmap puts RAG in the right order.
A quick decision checklist
- Fewer than a few hundred thousand chunks? Start with your existing database or an extension.
- Need to filter by user, team or document type? Keep vectors next to that data.
- Query traffic modest? No need for a separate system yet.
- Measured slow queries after adding a proper index? Now compare dedicated options.
Questions an interviewer may ask
"Why did you pick this store?" "How do you handle updates to documents?" "How do you know retrieval is working?" Good answers mention your scale, your filtering needs, and your retrieval evaluation, not just the product name.
If you are still stuck
Read How do I choose which LLM to use for my project? for the model side, and Should I use PostgreSQL or MongoDB? for the main database. For project ideas that test this properly, see the Urdu and Bangla retrieval benchmark and the sentence-level groundedness checker for RAG answers. If you want help with your project's design, join Sefism and, as a member, book a 1:1 session.
Was this answer helpful?
Read next
- RAG, Fine Tuning, or a Better Prompt: How to ChooseRAG vs fine tuning vs a better prompt: a practical order from prompting through retrieval, embeddings, reranking, tools and agents to fine tuning, with the signal that tells you to move to the next step.
- The AI Engineering Words You Keep Hearing, Explained ProperlyAI engineering terms in plain language: tokens, context windows, embeddings, RAG, fine tuning, tool calling, MCP, agents, evals and more, with what each is good for, where it lets you down, and the order to learn them.
Your situation is not quite this one?
Members get written answers to their own questions, a roadmap built for them and feedback on their projects. Early access is open: sign up and your account is active shortly after. Prefer to talk? A free call works too.