Key concepts

RAG (retrieval augmented generation)

Retrieve relevant context from your own data before the model answers.

RAG Pipeline

First question: do you need RAG at all? With 1M-token context windows, a well-scoped tool call or passing the documents directly is often simpler, cheaper to build, and easier to debug. RAG is worth its complexity when the corpus is large, changes often, or has per-user permissions.

RAG implementation guide

Step 1: Choose the stack

Component	Options	Recommendation
Vector DB	PGVector, Pinecone, Weaviate, Qdrant	PGVector if you already run Postgres. One less system to operate
Embeddings	Voyage, OpenAI, Cohere, local (nomic-embed)	Voyage for quality, local for privacy
Chunking	Docling, LangChain, LlamaIndex	Docling (PDFs, tables, images)
Framework	LangChain, LlamaIndex, raw API	Raw API until a framework clearly saves work

Step 2: Set up the vector store

docker run -d --name pgvector \
  -e POSTGRES_PASSWORD=password \
  -p 5432:5432 \
  pgvector/pgvector:pg16

psql -h localhost -U postgres -c "CREATE EXTENSION vector;"

CREATE TABLE document_chunk (
    id SERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    embedding vector(1024),   -- match your embedding model's dimension
    source_file TEXT NOT NULL,
    chunk_index INT NOT NULL,
    acl TEXT[] NOT NULL,      -- who may retrieve this chunk
    created_at TIMESTAMP DEFAULT NOW()
);

-- HNSW needs no training data and handles inserts well
CREATE INDEX ON document_chunk USING hnsw (embedding vector_cosine_ops);

Step 3: Chunk documents

from docling.document_converter import DocumentConverter
from docling.chunking import HybridChunker

def chunk_document(path: str) -> list[dict]:
    doc = DocumentConverter().convert(path).document
    return [
        {"content": chunk.text, "source": path, "index": i}
        for i, chunk in enumerate(HybridChunker().chunk(doc))
    ]

Step 4: Embed and store each chunk with its source and access list. Batch the embedding calls. Store the model name with the vectors so you know what to re-embed when you switch.

Step 5: Retrieve and generate

-- Top 5 chunks the current user is allowed to see
SELECT content, source_file
FROM document_chunk
WHERE acl && $2
ORDER BY embedding <=> $1
LIMIT 5;

Put the retrieved chunks, with their sources, in the prompt, and ask the model to cite them.

Step 6: Optimize, with measurements

Technique	When	How
Hybrid search	Recall misses exact terms	Vector plus keyword (BM25)
Reranking	Precision is low	Rerank the top 50 before taking 5
Query rewriting	Ambiguous queries	Model rewrites the query before search
Chunk overlap	Answers span chunk edges	10-20% overlap
Metadata filters	Large corpora	Filter by date, source, type first

What makes RAG fail in production:

Stale sources. Own the ingestion pipeline, not just retrieval. Wrong documents corrupt every answer
Missing access control. Retrieval must respect the same permissions as the source system
Tuning blind. Build a labeled eval set (questions with known right sources) before changing chunk sizes or models

Embeddings

Embeddings match meaning rather than keywords: "cancel my plan" finds the doc titled "Subscription termination". They also cut tokens, because you retrieve less, more relevant text.

Choosing a model:

Voyage: when retrieval quality is critical (legal, medical, support)
OpenAI text-embedding-3-small: a solid, cheap default
nomic-embed-text (local): privacy requirements or very high volume

Switching embedding models means re-embedding everything. Record which model produced each vector.

Key concepts

RAG (retrieval augmented generation)

RAG implementation guide

Embeddings

On this page