RAG that cites: the architecture we default to

Most retrieval systems fail the same way. They answer confidently from the wrong document, and nobody notices until someone acts on it.

This is the architecture we reach for by default, and the reasoning behind each choice. It is not novel. It is the boring version, which is the point — the interesting version is usually the one that cannot explain where an answer came from.

Start with the requirement, not the vector database

The first question is not which store. It is: what has to be true of an answer for someone to act on it?

For internal policy, that is usually a citation precise enough to check — a clause, not a document. For customer support, it is often that the answer never contradicts the published article. For anything regulated, it is that a superseded document is never presented as current.

Each of those is a different pipeline. Choosing the store first is how you end up with a system that retrieves well and cannot prove anything.

Chunk on structure, not on length

Fixed-size chunking is the default in every tutorial and it is wrong for documents that have shape.

Policy documents, contracts and technical manuals have headings, numbered clauses, tables and cross-references. A 512-token window cuts through all of them: it splits a table from its header, separates a clause from its number, and produces a citation that points at an arbitrary span of characters instead of at something a reader can look up.

1. Parse the document's structure first — headings, sections, list items, table boundaries.

2. Chunk along those boundaries, splitting further only when a section exceeds the window.

3. Carry the ancestry — the heading path — into the chunk's text, so a chunk retrieved out of context still says what it is part of.

4. Keep the source offsets. Without them a citation cannot point at a span, and span-level attribution is the entire premise.

Retrieve hybrid, then rerank

Dense retrieval finds the paraphrase. Lexical search finds the clause number somebody typed verbatim. Each fails at exactly what the other is good at.

Dense embeddings will happily miss GL-88213 — an identifier carries no semantics, and the nearest neighbours of its embedding are other identifiers. BM25 will just as happily miss "what happens if a delivery is refused" when the document says "consignment rejection". Running both and merging is not a hedge; it is covering two distinct failure modes.

Merge with reciprocal rank fusion rather than by comparing scores. The two systems' scores are not on the same scale and normalising them is a tuning exercise with no principled answer:

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    """Reciprocal rank fusion. Rank position only — scores are never compared."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for position, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + position + 1)
    return sorted(scores, key=lambda doc_id: scores[doc_id], reverse=True)

Then rerank. A cross-encoder reads the query and each candidate together rather than comparing two independently-computed vectors, and it is materially better at deciding which of twenty plausible chunks actually answers the question. It is too slow to run over the corpus and exactly right over the top fifty.

The last cell is the one to remember. A reranker cannot rescue a bad candidate set. If recall is poor, the reranker orders the wrong documents very precisely.

pgvector before a dedicated vector database

Our default store is Postgres with pgvector, and the reason is not benchmarks.

It is that retrieval is never the only query. You need permission filters, tenant scoping, document status, effective dates, and joins onto the systems that own the documents. In Postgres those are WHERE clauses against indexed columns, evaluated in the same query as the vector search. In a separate vector store they are metadata filters in a second system, kept in sync by you.

We move to a dedicated store when the numbers demand it — corpora in the tens of millions of vectors, or latency budgets that survive no other answer. That is a smaller set of projects than the tooling market implies.

Enforce permissions before retrieval, not after

The tempting design filters results after ranking. It is wrong.

A document the user may not read has already influenced the ranking, and if it survives into the context window its content can reach the model — and from there, the answer. The filter has to be part of the retrieval query, so the candidate set never contains anything the person asking could not open themselves.

Cite spans, and let it refuse

Generation is constrained to answer from the retrieved spans and to cite them. Two consequences follow, and both are features.

The first is that citations point at spans, so a reader can check the claim without re-reading the document. The second is that the model has an explicit path to say the documents do not support an answer — and that path has to be rewarded in evaluation, or it will never be taken.

A system that always answers is not more useful. It is the same system with the failures hidden.

Score retrieval and generation separately

This is the single most valuable thing in the whole pipeline, and it is an evaluation decision rather than an architectural one.

Build a labelled set of real questions with their correct source spans. Score retrieval on whether the right span was in the candidate set, and generation on whether the answer is faithful to the spans it cited. Kept separate, a fluent answer built from the wrong document is a retrieval failure with a name. Merged into one "answer quality" number, it is invisible — and it is the failure most likely to matter.

Freshness is a production metric

Sources change on their own schedule. Re-index on change rather than on a nightly sweep, because the window between the change and the sweep is exactly when the stale answer gets given.

And keep superseded documents. Do not delete them — exclude them from retrieval by status. A system that answers correctly only because the wrong document was removed will answer incorrectly the first time one is not.