A request we now receive most months: build an internal assistant that answers staff questions from company documents. The expectation is that this is a model problem, or an embedding problem, or a vector database selection problem.

It is almost always a document problem.

What retrieval actually does

Retrieval-augmented generation finds passages that look relevant to a question and gives them to a language model as context. The model then answers from those passages.

Which means retrieval faithfully reproduces whatever is in your documents — including the 2019 policy that was superseded but never deleted, the regional variant that contradicts head office, and the draft somebody left in the shared drive. The system does not know which is authoritative. Nothing in the pipeline is designed to.

If two documents disagree and neither is marked as current, a confident wrong answer is the expected output, not a malfunction.

The unglamorous prerequisites

Before retrieval is worth building, someone has to do the following. None of it involves a model.

  • Establish authority. For any question, which document is the current answer? If no human can say, the system certainly cannot.
  • Remove or mark superseded content. Archive it outside the index, or tag it clearly enough to filter.
  • Attach metadata. Effective date, owning department, applicable region, review date. Most useful filtering depends on metadata, not on embedding similarity.
  • Decide the boundary. Which questions is this system allowed to answer, and which must route to a person?

Cite everything

Every answer should carry a link to the passage it came from. This is partly trust — users can verify rather than believe — but mostly it is a debugging tool. When an answer is wrong, citations tell you immediately whether retrieval fetched the wrong document or the model misread the right one. Those are entirely different bugs with entirely different fixes.

Measure, do not assume

Build a test set of real questions with known-correct answers before launch. Fifty is enough to start. Score changes against it. Without a harness, every prompt adjustment is a guess, and confidence about quality drifts away from reality quickly.

When it is worth it

None of this argues against building one. An assistant over well-governed policy documents genuinely saves people a great deal of time, and the technology is ready.

The argument is about sequencing. The document work is not a preliminary step before the interesting part — it is the project. The retrieval layer on top is a few weeks.