Test Strategy to Execution: The Pipeline That Runs Your Tests and Proves the Coverage

A test strategy is a set of decisions about risk, levels, environments, and who can ship with a known failure. An execution pipeline turns those decisions into four layers that run at different frequencies, each emitting JUnit XML tagged with the requirement id and the commit SHA. The coverage proof is then generated from the last run rather than assembled before the audit.

Most test strategy documents are twenty pages describing what testing is. They list the levels, define the terms, name the tools, and make no decisions. You can tell because you could swap the project name for another project’s and nothing would need changing.

A strategy that makes no decisions leaves every real question to be argued per feature under deadline pressure, which is how coverage becomes a function of how tired the team was that week. This article is the last stage of the requirements to UAT pipeline: the decisions worth writing down and the machinery that executes them. The seven-step method for building the plan itself is in building a whole test plan with AI. If you want the testing playbook behind this, it is in API Testing and QA Mastery for BAs.

What decisions does a test strategy actually have to make?

Nine. If your document does not answer these, it is a description rather than a strategy.

Decision	The question it settles
Risk model	How depth of coverage is chosen, so it is not argued per feature
Levels and ownership	What each level proves, and who writes and owns it
What is not tested	Named, with the reason and who accepted it
Environments	Which environment each level runs against, and who can break it
Test data	Where it comes from, whether it is production-derived, how it is masked
Entry and exit criteria	What must be true to start testing and to call it finished
Defect severity	What severity 1 means, with examples, so triage is not a negotiation
Automation boundary	What is automated, what stays manual, and why
Release authority	Who can decide to ship with a known open defect

The third and the ninth are the ones that get left out and the ones that matter most. A strategy that does not name what is out of scope implies everything is in scope, and a strategy that does not name who can accept a known defect means that decision gets made at 18:00 on release day by whoever is most senior in the room.

Write each as a decision with a rationale:

## D3. What we are not testing

- **Third party sanctions screening internals.** We test our request and
  our handling of each response type. We do not test their matching
  accuracy. Accepted by: Head of Financial Crime, 2026-09-10.
- **Browser support below Chrome 120 and Safari 17.** Analytics shows
  0.4% of portal sessions. Accepted by: Head of Digital, 2026-09-10.
- **Disaster recovery failover.** Covered by the infrastructure DR test
  in November, out of scope for this release. Accepted by: CTO.

Three bullets, each with an owner and a date. That is worth more than fifteen pages defining regression testing.

Ranking risk so depth is decided rather than argued

Score every area on three factors. Multiply. Let the score pick the coverage.

# tests/risk.yaml
- area: Daily limit evaluation
  requirements: [REQ-014]
  impact: 5          # 1 cosmetic .. 5 financial loss or regulatory breach
  likelihood: 4      # 1 stable and simple .. 5 new, complex, or churning
  detection: 4       # 1 fails loudly .. 5 silently wrong for weeks
  score: 80
  coverage: exhaustive

- area: Payment reference formatting
  requirements: [REQ-022]
  impact: 2
  likelihood: 2
  detection: 1
  score: 4
  coverage: smoke

The detection factor is the one most teams omit, and it is the one that catches the dangerous cases. A limit check that silently allows a breach produces no error, no alert, and no support call. It is wrong for a month. That is worth far more testing than a failure that throws a 500 on the first request, even though the second one feels more alarming.

Then map score to depth, once, for the whole programme:

Score	Coverage	What that means concretely
48 and above	Exhaustive	Every boundary, every negative case, every exception flow, at contract and end-to-end level, plus a UAT scenario
18 to 47	Thorough	Main flow, every exception flow, key boundaries, at integration level
6 to 17	Standard	Main flow plus the most likely failure, at integration level
Below 6	Smoke	One happy path check

Now “how much should we test this” has an answer that survives a conversation with a project manager who wants to cut the window.

The four layers of the execution pipeline

Each layer has a different speed, a different environment, and a different tolerance for flakiness. Putting a test in the wrong layer is how suites get ignored.

Layer	Runs on	Against	Target time	Blocks
Contract	Every commit	Mocks and schema validation	Under 2 minutes	The commit
Integration	Every merge to main	Deployed test environment, real dependencies	Under 15 minutes	The merge
End to end	Nightly and pre-release	Full integrated environment	Under 90 minutes	The release
UAT	On demand	UAT environment, business data	Manual	Sign-off

The rule that keeps this healthy: nothing that can fail for reasons unrelated to the change may block a commit. A contract test that depends on a shared database will fail because somebody else was working, the team will learn that red does not mean broken, and within a month the signal is gone. The detail of wiring the first two layers is in API tests in CI.

# .github/workflows/test.yml
name: tests
on:
  push:
  pull_request:
  schedule:
    - cron: "0 2 * * *"   # nightly end to end

jobs:
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx @usebruno/cli run collections/contract -r --env ci
              --reporter-junit build/junit-contract.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: junit-contract, path: build/junit-contract.xml }

  integration:
    if: github.ref == 'refs/heads/main'
    needs: contract
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx @usebruno/cli run collections/integration -r --env test
              --reporter-junit build/junit-integration.xml
      - run: bash scripts/xray-import.sh
        env:
          XRAY_CLIENT_ID: ${{ secrets.XRAY_CLIENT_ID }}
          XRAY_CLIENT_SECRET: ${{ secrets.XRAY_CLIENT_SECRET }}
          TEST_PLAN_KEY: NP-402
      - run: python scripts/coverage-proof.py --junit build/junit-integration.xml

  e2e:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx cucumber-js tests/features
              --format junit:build/junit-e2e.xml
      - run: python scripts/triage.py --junit build/junit-e2e.xml
        if: failure()

Generating the executable suite from the conditions

The test conditions from stage six of the pipeline are structured, so the suite is generated rather than typed.

# tests/conditions.yaml
- id: TC-041
  covers: [FR-031-R1, FR-031-R2]
  requirement: REQ-014
  risk_score: 80
  condition: Daily limit evaluation at and around the threshold
  data_variants:
    - { used: 45000, amount: 4999.99, expect: accepted, note: below }
    - { used: 45000, amount: 5000.00, expect: accepted, note: exactly at limit }
    - { used: 45000, amount: 5000.01, expect: rejected, reason: AM04 }
    - { used: 50000, amount: 0.01,    expect: rejected, reason: AM04 }
    - { used: 0,     amount: 50000.00, expect: accepted, note: single payment at limit }

Then the generation prompt, constrained so it cannot invent behaviour:

Generate Bruno requests and assertions for test condition TC-041 using
the attached openapi.yaml and functional.yaml.

Rules:
- One request per data variant. Name it TC-041-<note or reason>.
- Assert the status code, the reason code, and the resulting available
  balance. Assert the absence of a payment record where expect is rejected.
- Take every field name and every enum value from openapi.yaml. If a
  field you need is not in the contract, output CONTRACT GAP: <field>
  and stop rather than inventing it.
- Add the tag @REQ-014 @FR-031 @TC-041 to every request.
- Do not add tests for behaviour not stated in functional.yaml.

The CONTRACT GAP instruction has found more real problems for me than the tests it generates. A model that cannot write an assertion because the field it needs is not in the contract has located a genuine hole between the specification and the API, which is the same finding API contract analysis produces by hand.

Results as evidence: the requirement id and the commit SHA

A green test run proves almost nothing on its own. It becomes evidence when it carries two extra pieces of information.

<testsuite name="integration" tests="41" failures="0" time="412.8">
  <properties>
    <property name="commit" value="9a4c1f2e8b7d3a5c6e9f0b1d2a3c4e5f6a7b8c9d"/>
    <property name="requirements_sha" value="44d2c25f1a9b3e7c2d8f4a6b0c1e3d5f7a9b2c4e"/>
    <property name="environment" value="test-eu-1"/>
    <property name="run_started" value="2026-09-19T02:04:11Z"/>
  </properties>
  <testcase name="TC-041-exactly-at-limit [REQ-014][FR-031]" time="0.412"/>
</testsuite>

requirements_sha is the one people miss. It records which version of the specification the run was proving. Six months later, when someone asks whether the signed-off behaviour was actually tested, that field answers it in one command instead of a week of archaeology.

Then the proof generator joins everything:

# scripts/coverage-proof.py
import sys, yaml, re, argparse
import xml.etree.ElementTree as ET

ap = argparse.ArgumentParser(); ap.add_argument("--junit", required=True)
args = ap.parse_args()

reqs = {r["id"]: r for r in yaml.safe_load(open("requirements/business.yaml", encoding="utf-8"))}
tree = ET.parse(args.junit); root = tree.getroot()
props = {p.get("name"): p.get("value") for p in root.iter("property")}

results = {}
for case in root.iter("testcase"):
    ids = re.findall(r"\[(REQ-\d+)\]", case.get("name", ""))
    failed = case.find("failure") is not None or case.find("error") is not None
    for rid in ids:
        r = results.setdefault(rid, {"pass": 0, "fail": 0, "tests": []})
        r["fail" if failed else "pass"] += 1
        r["tests"].append(case.get("name"))

rows = ["| Requirement | Priority | Tests | Passed | Failed | Verdict |",
        "|---|---|---|---|---|---|"]
blocking = []
for rid, req in sorted(reqs.items()):
    r = results.get(rid)
    if not r:
        verdict = "NOT COVERED"
    elif r["fail"]:
        verdict = "FAILED"
    else:
        verdict = "PROVEN"
    if req["priority"] == "must" and verdict != "PROVEN":
        blocking.append(f'{rid} ({verdict})')
    rows.append(f'| {rid} | {req["priority"]} | {r["pass"] + r["fail"] if r else 0} '
                f'| {r["pass"] if r else 0} | {r["fail"] if r else 0} | {verdict} |')

header = (f'# Coverage proof\n\nCode `{props.get("commit", "?")[:10]}` against '
          f'requirements `{props.get("requirements_sha", "?")[:10]}` '
          f'on `{props.get("environment", "?")}` at {props.get("run_started", "?")}.\n')
open("build/coverage-proof.md", "w", encoding="utf-8").write(header + "\n" + "\n".join(rows))

if blocking:
    print("FAIL: must-have requirements not proven: " + ", ".join(blocking))
    sys.exit(1)
print(f"{len(reqs)} requirements, all must-haves proven.")

That script is the fourth gate from the pipeline article. A release cannot be cut while a must requirement is NOT COVERED or FAILED, and the argument about whether it matters happens against a generated table rather than against a spreadsheet’s last edit date.

Automating the first pass of triage

A nightly suite that produces eleven failures and no explanation gets muted within a fortnight. Automating triage is what keeps people acting on it.

A nightly end-to-end run failed. Attached:
- the JUnit failure output with messages and stack traces
- git log since the last green run
- the deployment log for the test environment, last 24 hours
- the Datadog error rate and p95 for the affected services in the window

For each failure, classify it as exactly one of:
  PRODUCT   the system behaves differently from the specification
  TEST      the test is wrong, stale, or asserts the wrong thing
  DATA      the test data is missing, consumed, or in the wrong state
  ENV       the environment, a dependency, or a deployment caused it
  FLAKY     timing or ordering, likely to pass on rerun

For each, give: the evidence for the classification, a confidence from
1 to 5, the requirement id affected, and the one command or query a
human should run first to confirm.

Group failures with the same root cause. Say so explicitly when eleven
failures are one cause.

The grouping instruction is what saves the morning. Eleven failures from one expired certificate look like a catastrophe in a report and take four minutes to fix once somebody says so.

The classification is reliable for ENV and DATA, useful for TEST, and must never be trusted alone for PRODUCT. A human confirms before any defect ticket is raised, because a wrongly closed product defect is far more expensive than a wrongly investigated environment failure. The wider triage discipline is in defect triage for analysts.

Pulling the Datadog window into the triage context is worth the setup. A failure at 02:14 alongside an error rate spike at 02:12 in a dependency is almost always the same story, and correlating the two by hand takes ten minutes that the MCP connection removes entirely.

What to measure about the pipeline itself

Four numbers, reviewed monthly. They tell you whether the machinery is earning its keep.

• Requirements proven per run. The headline. Should rise and never fall silently.
• Escaped defects by origin stage. Of the defects found in UAT and production, which stage should have caught each. A high count originating at system test means UAT is doing work that belongs earlier.
• Mean time from failure to classification. Was hours, should be minutes once triage is automated. This is the number that tells you whether people are still reading the nightly report.
• Flaky rate. Percentage of failures that pass on a rerun with no change. Above five percent and the suite is losing credibility; fix the tests before adding any.

The second one is the metric that improves the whole pipeline rather than just the testing. It tells you where the earlier stages are weak, which is exactly what the blind spot review is supposed to fix.

The takeaway

A test strategy is nine decisions, and the two that matter most are what you are not testing and who can ship with a known defect. Rank risk on impact, likelihood, and detection difficulty so the depth of coverage is chosen rather than argued. Split execution into four layers by speed and environment, and never let a shared-environment test block a commit.

Then make the results evidence: every run tagged with the requirement id, the code commit, and the SHA of the requirements it was proving, imported into Xray so coverage in Jira is live, and joined into a coverage proof that fails the release when a must-have requirement is not proven. Automate the first pass of triage so the nightly report stays something people read.

That closes the loop from the workshop transcript to the signature. Start at the pipeline overview if you want the whole chain, or build the plan first with the seven-step test plan method. For the full testing playbook, see API Testing and QA Mastery for BAs, or browse everything at The Tech BA Toolkit.

Ahmed is a Senior Technical Business Analyst with 10+ years in banking and payments. He builds practical guides and tools for analysts at The Tech BA Toolkit.

Tags: QA, Test Strategy, CI/CD, Traceability, Automation