The architecture of retrieval: when RAG needs more than vectors

Retrieval-augmented generation is often described as a short pipeline: split documents, create embeddings, find similar chunks, and place them in a prompt. That version is useful for a prototype. It becomes fragile when an application must handle permissions, changing facts, relationships, and questions that require more than one source.

The problem is not that vector search is weak. The problem is asking one similarity score to represent every meaning of relevance.

Vector similarity solves one part of relevance

An embedding model places text with similar meaning near other text in a vector space. That makes vector search effective when a user and a source describe the same idea using different words.

It does not automatically tell us whether a passage is current, authoritative, connected to the correct workspace, or safe for the current user. A highly similar chunk may describe an old policy. Another may mention the right project but belong to the wrong account. A third may contain an answer without the dependency that explains when it applies.

Retrieval quality comes from combining useful signals, not expecting one score to carry the whole system.

Graph context adds relationships

Relationships become important when facts are not independent. A workspace owns projects. A project contains decisions. A new decision supersedes an older one. A user belongs to a team with a particular access level.

Graph traversal does not need to replace semantic search. It can constrain and expand it. The Hybrid RAG Engine uses vector search to identify candidates, then graph-aware retrieval to expose nearby entities and relevant paths. Semantic search provides recall. Graph traversal provides structure.

This separation also makes the product easier to explain. The interface can show why a result appeared, which entity connected it to the question, and what evidence was finally used.

Context assembly deserves its own layer

Many RAG implementations jump directly from retrieved chunks to prompt construction. I prefer an explicit assembly layer between retrieval and generation. It can remove duplicates, enforce a token budget, group evidence by source, attach timestamps, and preserve citation identifiers.

candidates = semantic_search(query, workspace_id)
related = graph.expand(candidates, depth=1)
allowed = permissions.filter(user, candidates + related)
ranked = reranker.score(query, allowed)
context = assembler.build(ranked, token_budget=4200)

These boundaries make the pipeline observable. When an answer is poor, we can inspect the semantic candidates, graph expansion, permission filter, rank order, and final context independently. Without those stages, debugging becomes prompt guesswork.

Evaluation should match the architecture

End-to-end answer quality matters, but it does not identify where a system failed. Retrieval evaluation should ask whether the relevant source appeared, whether it ranked early enough, whether necessary relationships were included, and whether forbidden evidence was excluded.

A small representative test pack can reveal more than a large collection of vague prompts. Useful cases include ambiguous names, stale documents, access boundaries, multi-hop questions, conflicting claims, and questions where the correct answer is that evidence is insufficient.

The Hybrid RAG Engine includes deterministic local fixtures for those scenarios. That keeps the interface explorable even when external AI services are unavailable, and it separates product testing from model quota.

The model is the final consumer

A dependable RAG feature behaves like product infrastructure. It needs latency budgets, permission boundaries, traceable sources, predictable fallbacks, and a way to improve retrieval without rewriting every prompt.

Vector search remains a strong component. It becomes more trustworthy when it sits beside graph relationships, explicit ranking, and a context layer that can be inspected by both engineers and users.