Where vectors live

Once your chunks are embedded, you need somewhere to put thousands or millions of vectors and a way to find the nearest ones fast. That's a vector database. A brute-force search comparing your query against every vector works for a few hundred items but collapses at scale; vector databases use clever indexes (approximate nearest-neighbor) to return top matches in milliseconds.

The options

• Managed: Pinecone, Zilliz — you call an API and they handle scaling.
• Open-source: Chroma (great for prototyping), Qdrant, Weaviate, Milvus.
• Add-on to what you already run: pgvector turns Postgres into a vector store, so you don't stand up new infrastructure.

For learning and small apps, an embedded store like Chroma is the fastest way to start.

A minimal flow

import chromadb
client = chromadb.Client()
col = client.create_collection("docs")

col.add(ids=["1"], documents=["Reset your password from Settings > Security"],
        embeddings=[embed("reset password doc")])

hits = col.query(query_embeddings=[embed("how do I change my password")], n_results=3)

You add each chunk with its embedding once, then query with the embedded question to get the top n matches. Those matches become the context you hand to the LLM.

Beyond the basics

• Metadata filtering — restrict the search ("only this product's docs," "only 2026") before similarity ranking.
• top_k — how many chunks to retrieve. Too few misses the answer; too many floods the prompt with noise. Three to five is a common start.
• Store the original text, not just the vector, so you can actually put it in the prompt and cite it.

The vector database isn't the clever part — the embeddings are. The database's one job is to find "nearby" fast and reliably, so pick the simplest one that fits your scale.

Try this: Stand up Chroma locally, add five short sentences on different topics, and query it with a paraphrase of one of them. Watching it return the right sentence — by meaning, not keywords — makes the whole RAG idea click.