RAG Vector Databases: What They Store and What They Miss
A vector database retrieves relevant passages from your documents at query time. Here is how that works, what it cannot provide on its own, and how explicit project context fills the gap.
Updated 6 min read
In a retrieval-augmented generation (RAG) system, a vector database stores embeddings — numerical representations of document passages — so the system can find the most relevant passages at query time and add them to the model's context. The model then answers based on retrieved text rather than relying solely on what it learned during training.
Vector retrieval is excellent at finding relevant passages in a large document set. It does less well with the kind of explicit, current context that a working project needs: agreed decisions, active tasks, and where things stand right now. For that kind of context, a tool like fenbs holds the explicit project record alongside whatever retrieval system your application uses.
How a vector database stores documents
Before documents can be retrieved, they are processed into vectors. A document — a PDF, a support article, a code file — is split into chunks, usually a few hundred tokens each. An embedding model converts each chunk into a fixed-length numerical vector that encodes its meaning. Those vectors are stored in the database alongside the original text.
Different databases handle this differently. Some store only vectors and metadata; you supply the embedding model separately. Others include a built-in embedding pipeline. The choice of embedding model affects retrieval quality: the model used at index time must match the one used at query time, because the vectors only make sense in the same embedding space.
How retrieval works at query time
When a user asks a question, the query is embedded with the same model used to index the documents, producing a query vector. The database finds the stored vectors most similar to the query vector — typically measured with cosine similarity or dot product — and returns the corresponding text chunks.
Those retrieved chunks are then injected into the model's prompt alongside the user's question. The model uses them as grounding material. This is why RAG can answer questions about documents the model never saw during training: the answer comes from retrieved text, not from memorized parameters.
Retrieval quality depends on several factors: chunk size, the embedding model, the number of results fetched (top-k), and whether the chunks contain enough context to be useful in isolation. The what is RAG article covers the full pipeline in more depth.
What a vector database reliably provides
- Semantic search across large document collections that would not fit in a single prompt.
- Relevance ranking: the chunks most related to the query surface first.
- Scale: millions of vectors can be stored and queried with sub-second latency on hosted services.
- Source attribution: retrieved chunks carry metadata (filename, page, timestamp) so you can cite the source.
What a vector database misses
Retrieval finds relevant passages — it does not answer "what did we decide?" or "what is the current status?" A vector database stores snapshots of documents. It does not know which decision supersedes an older one, which task is active right now, or which rule was agreed upon after the document was written.
- Explicit decisions: "We chose provider A over provider B on September 10." A decision does not live in a passage; it lives in the record of who decided what and why.
- Current task state: what is in progress, what is blocked, and what the next step is.
- Recency: a document indexed three months ago may describe a process that changed last week. The database does not know the difference.
- Rules and priorities: standing instructions that should always apply do not emerge reliably from passage similarity.
- Structured relationships: who owns a task, what a decision applies to, and how two tasks relate are structural facts that are hard to retrieve from unstructured text.
When a vector database is the right choice
Vector retrieval is the right tool when you have a large body of relatively stable reference documents and need to answer questions that require finding relevant passages in them: a knowledge base, a policy library, a product documentation set, a collection of research papers.
It is less suited to providing the current, structured, actively-maintained context of a project in motion. For that, you need something that is updated deliberately as decisions are made and tasks progress, not something that relies on finding relevant passages in a document collection.
Popular vector database options
| Option | Good fit for | Notes |
|---|---|---|
| Chroma | Local development and prototyping. | Open source, runs in memory or on disk, simple Python API. |
| Pinecone | Production applications needing a managed service. | Fully managed, serverless option available. Pricing on the Pinecone website. |
| pgvector | Teams already using PostgreSQL. | Extension that adds vector storage and similarity search to Postgres. |
| Weaviate | Applications wanting built-in hybrid (vector + keyword) search. | Open source with a managed cloud option. |
| Qdrant | High-performance filtering alongside vector search. | Open source, strong payload filtering. Managed cloud and self-hosted both available. |
| fenbs | Shared project context: decisions, rules, active tasks, and Where we left off. | Not a document retrieval system. Best used alongside a vector database, not instead of one. |
Supply what retrieval cannot
A vector database finds relevant passages. fenbs keeps the explicit, current project record: agreed decisions, active tasks, rules that always apply, and where things stand. Connected AI apps read the fenbs context at the start of a session, giving them the structured project state that similarity search cannot reliably surface. Create a free account and connect your AI from Integrations.
Create a free accountFree for one person · No credit card · Works with Claude, ChatGPT, Cursor and more
How to do this with fenbs
- Create a free account and add your project. Write the decisions, rules, and active tasks that your AI should know before it starts.
- Keep your vector database for document retrieval: knowledge bases, support docs, research libraries.
- Connect your AI app to fenbs over MCP from Integrations for project-level context.
- At the start of each session, ask the AI to read the project context from fenbs, then proceed with any retrieval-grounded work.
- After the session, ask the AI to record any new decisions or lessons in fenbs so the next session starts from the updated state.
The agent memory vs RAG article covers how different kinds of memory complement each other. The database MCP server article explains how to connect a database directly to an AI agent over MCP.