RAG for Analysts: How to Specify and Test a Retrieval System

Retrieval-augmented generation is a search system with a language model at the end. It fails as a search system far more often than it fails as a language model, which is why an analyst specifies and tests retrieval separately from generation. The artifact that makes both possible is a golden set: real questions, the passages that should be found, and the answer a knowledgeable human would give.

An analyst gets handed a RAG project in one of two ways. Either someone built a proof of concept that demos beautifully and nobody can say whether it is ready, or someone is about to build one and the requirements document says “the assistant must answer questions about our documentation accurately”. Both are the same problem: nothing in that sentence is testable, and the things that will actually go wrong are not in it.

This article is the specification and testing model for a retrieval system. It sits next to acceptance criteria for AI systems, which covers the criteria themselves, and why acceptance criteria failed on an AI project, which is what happens when you skip this. If you want the full architecture of MCP, retrieval, and agents in one place, it is in AI at Work: MCP, RAG, and AI Agents.

What are the eight stages, and what fails at each?

A RAG system is a pipeline. Knowing which stage broke is most of the work, and it is impossible if you only look at the final answer.

Stage	What it does	How it fails	Who notices
1. Source selection	Decides which documents are in scope	Stale wiki space included, current one missed	Nobody, for months
2. Chunking	Splits documents into passages	A table or a procedure split across two chunks	Looks like a model error
3. Embedding	Turns each chunk into a vector	Domain vocabulary the model has never seen	Looks like bad search
4. Indexing	Stores vectors plus metadata	Permissions and dates not carried into the index	Security review, late
5. Retrieval	Finds the top k passages for a question	Top k too small, no filters, wrong similarity	Wrong answers
6. Reranking	Reorders candidates by relevance	Absent, so position 9 never reaches the prompt	Answers that are nearly right
7. Generation	Writes the answer from the passages	Model contradicts or embellishes the passage	Confident, plausible, wrong
8. Citation	Shows where the answer came from	Citation points at a chunk that does not support it	Only if somebody clicks

The column that matters is the last one. Five of these eight failures are invisible from the answer alone, which is why “we tested it by asking it forty questions” is not testing.

The rule to take into every design conversation: if the right passage never reaches the model, nothing downstream can fix it. Prompt engineering cannot recover content that was not retrieved. So retrieval quality is the first thing you specify and the first thing you measure.

The requirements an analyst has to write

Eight sets, one per stage. These are the ones that are almost always missing from a first draft, in the requirements as code format so they carry ids into the test suite.

- id: REQ-101
  title: Source scope and ownership
  statement: >
    The index contains the Payments Confluence space, the published API
    reference, and the operations runbooks. It excludes personal spaces,
    draft pages, archived spaces, and anything labelled restricted.
  acceptance:
    - A page in an excluded space is never returned by retrieval.
    - Each in-scope source has a named owner accountable for accuracy.
    - A page archived after indexing is removed within 24 hours.

- id: REQ-104
  title: Permission-aware retrieval
  statement: >
    Retrieval returns only passages from documents the asking user is
    entitled to read, enforced at query time against the caller's identity.
  acceptance:
    - A user without access to a page never receives its content, and
      never receives a citation revealing its title or existence.
    - Permission changes take effect within 15 minutes without re-indexing.
    - The permission filter is applied before the vector search, not after
      generation.

- id: REQ-108
  title: Abstention
  statement: >
    When no retrieved passage supports an answer, the system states that it
    could not find the information and names the sources it searched.
  acceptance:
    - A question about a topic absent from every source produces an
      explicit "not found" response, never a general-knowledge answer.
    - The response offers the closest related pages it did find.
    - Abstention rate is reported, because an abstention rate of zero
      means the system is inventing rather than declining.

REQ-104 is the one that ends projects. A retrieval system that ignores access control will, on a long enough timeline, quote a salary review or a supplier contract to somebody who was never allowed to open it. The detail that catches teams out is in the third acceptance criterion: filtering after generation is too late, because the restricted content has already been put in a prompt and has already shaped the answer. The wider data classification judgement is in AI guardrails for analysts.

REQ-108 is the one that is easiest to forget and cheapest to add. A system with no abstention behaviour answers everything, which means it invents the answers it cannot retrieve.

Chunking is a requirements decision

Teams file chunking under engineering. It is not. What may not be split is a business rule and the analyst is the one who knows it.

Ask these four questions about the corpus, and write the answers as requirements:

• What must never be split? A table of reason codes, a numbered procedure, a set of acceptance criteria. Split a nine-row table across two chunks and the system will answer confidently with five of the nine rows, and no test on the final answer alone reveals why.
• What context does a chunk lose when it is lifted out? A passage saying “this limit does not apply in the instant payments flow” is dangerous without its heading. The fix is a requirement that every chunk carries its document title and heading path.
• What is the natural unit of an answer here? For a runbook it is a procedure. For an API reference it is an endpoint. For a policy it is a clause. Chunk to that unit, not to a fixed character count, whenever the format allows.
• What metadata must travel with the chunk? Source, owner, last updated, access level, document type. Every one of those becomes a retrieval filter later, and adding it after indexing means reindexing everything.

That last point is worth insisting on early. Metadata you did not index cannot be filtered on, and “only search documents updated in the last year” is a request that arrives eventually on every project.

The golden set: your test suite and your specification

You cannot evaluate a RAG system without a reference, and building that reference is analyst work, not engineering work.

# eval/golden-set.yaml
- id: G-014
  question: What reason code do we return when a payment breaches the daily limit?
  asked_by: operations          # persona, which sets the permission context
  must_retrieve:
    - doc: PAY/limits-design
      passage_contains: "AM04"
  reference_answer: >
    AM04. The payment is rejected, not queued, and the customer is told
    within 5 seconds.
  must_cite: [PAY/limits-design]
  must_not_say: ["AM02", "the payment is held"]

- id: G-031
  question: What is the daily limit for customer NP-4471?
  asked_by: external-support
  expect: abstain
  reason: >
    Customer-specific values are not in the corpus. The system must decline
    rather than infer from an example in the design document.

- id: G-042
  question: What were the findings of the 2026 compensation review?
  asked_by: operations
  expect: not-found
  reason: >
    The HR space is out of scope. The response must not reveal that a
    document with this title exists.

Three kinds of entry, and you need all three. G-014 is a normal question with a known answer. G-031 tests abstention, which is the behaviour most likely to be missing. G-042 tests permissions and scope, and note that the requirement is not merely to refuse the content: revealing the title is already a leak.

Aim for 80 to 150 entries, built from real questions. The best source is your support channel, because the questions people actually ask are shaped differently from the questions a project team invents. Every persona in asked_by needs entries, because permission behaviour is only tested by asking as someone restricted.

Build this before the system is finished. It doubles as the clearest requirements document you will produce, because a stakeholder who disagrees with a reference_answer is disagreeing with a requirement, in a form specific enough to resolve.

Measuring retrieval separately from generation

Two suites, two sets of numbers, because a single accuracy figure hides which stage broke.

Retrieval metrics. Run every golden question, capture what came back, and ignore the model entirely.

• Recall at k: the share of questions where at least one required passage appears in the top k. This is the headline. Below roughly 0.9 the system cannot be good, whatever the answers look like.
• Mean reciprocal rank: how high the right passage ranks. A required passage at position 8 will often be crowded out of the prompt.
• Permission violations: any question where a passage the persona cannot read was retrieved. This is a count, and the acceptable value is zero.

Generation metrics. Given the retrieved passages, assess the answer.

• Groundedness: every claim in the answer traces to a retrieved passage. Anything ungrounded is a hallucination, regardless of whether it happens to be true.
• Correctness: the answer matches the reference on the facts that matter.
• Abstention correctness: it declined when it should have declined, and did not decline when the passage was right there.

# eval/run.py  the shape of the harness, model call elided
import yaml, json, statistics

golden = yaml.safe_load(open("eval/golden-set.yaml", encoding="utf-8"))
rows, violations = [], []

for case in golden:
    hits = retrieve(case["question"], persona=case["asked_by"], k=8)

    # permission check first: a violation invalidates the run
    for h in hits:
        if not allowed(case["asked_by"], h["doc"]):
            violations.append((case["id"], h["doc"]))

    required = {m["doc"] for m in case.get("must_retrieve", [])}
    found = [i for i, h in enumerate(hits, 1) if h["doc"] in required]
    recall = 1 if found else 0
    rr = 1 / found[0] if found else 0

    answer = generate(case["question"], hits)
    judged = judge(answer, case)   # groundedness, correctness, abstention

    rows.append({"id": case["id"], "recall": recall, "rr": rr, **judged})

summary = {
    "recall_at_8": statistics.mean(r["recall"] for r in rows),
    "mrr": statistics.mean(r["rr"] for r in rows),
    "groundedness": statistics.mean(r["grounded"] for r in rows),
    "correctness": statistics.mean(r["correct"] for r in rows),
    "permission_violations": len(violations),
}
json.dump({"summary": summary, "rows": rows, "violations": violations},
          open("build/rag-eval.json", "w"), indent=2)

assert summary["permission_violations"] == 0, violations
assert summary["recall_at_8"] >= 0.90
assert summary["groundedness"] >= 0.95

Two things make this useful rather than decorative. The permission assertion is absolute: one violation fails the run, full stop, because there is no acceptable rate of leaking documents. And the thresholds are in the file, so a change that improves correctness while quietly dropping recall gets caught.

Run the suite on every change to chunking, the embedding model, top k, the prompt, or the model version. Those five are the levers, and all five interact. This is the regression suite for an AI feature, and it belongs in CI exactly like the API suite in a release gate.

The questions to ask in the design review

Take these into the first architecture conversation. They are the ones that produce silence.

1. What happens when a source document changes? How long until an answer reflects it, and how would we know it had not?
2. Where are permissions enforced, and what is the test that proves it?
3. What does the system do when it cannot find an answer, and what is our current abstention rate?
4. What is our recall at k on the golden set, and when did we last measure it?
5. What happens when two retrieved documents contradict each other? Does the answer say so, or does it pick one silently?
6. Which stage do we look at first when an answer is wrong, and can we see the retrieved passages for a past answer?
7. How does a user report a bad answer, and what happens to that report?

Question five is the one that separates a considered design from a demo. Corporate documentation contradicts itself constantly, because the old page was never deleted. A system that silently picks one version is worse than a search box, since the search box at least showed you both.

Question six is about observability. If you cannot retrieve the passages that produced last Tuesday’s wrong answer, you cannot investigate it, and every incident becomes speculation. Log the retrieved chunk ids with every response.

What analysts get wrong about RAG

• Treating it as a model problem. The instinct is to change the model or the prompt. The fix is usually in chunking, metadata, or top k.
• Testing with questions the team invented. They are unrepresentatively well phrased. Real users ask half-questions with internal shorthand and no context.
• Measuring once, at the end. All five levers interact, so a quality number from three weeks ago describes a system that no longer exists.
• Skipping the permission persona. If every golden question is asked as an administrator, the entire permission model is untested.
• Accepting a demo as evidence. A demo is twelve questions somebody chose. The golden set is a hundred they did not.

The takeaway

A RAG system is a search pipeline with a model on the end, and the analyst’s job is to specify and measure each stage rather than the final answer alone. Write requirements for source scope, chunking, metadata, freshness, retrieval depth, permission-aware filtering, citation, and abstention. Build a golden set of real questions with required passages, reference answers, and personas, and treat it as both the specification and the regression suite. Measure retrieval and generation separately, fail the run on any permission violation, and re-run on every change to the five levers.

Do that and “the assistant must answer accurately” becomes a set of numbers with thresholds, which is the difference between a feature you can sign off and a demo you can only admire. For the full architecture of retrieval, agents, and MCP as an analyst needs to understand it, see AI at Work: MCP, RAG, and AI Agents, or browse everything at The Tech BA Toolkit.

Ahmed is a Senior Technical Business Analyst with 10+ years in banking and payments. He builds practical guides and tools for analysts at The Tech BA Toolkit.

Tags: Artificial Intelligence, RAG, Requirements, Software Testing, LLM