Developer Testing in the AI Era: Write the Oracle, Not Just the Test

Developer testing in the AI era comes down to one skill the model cannot supply: the oracle, the independent definition of what correct means. An assistant asked to test a function reads the function and asserts what it currently does, bugs included. Derive tests from the requirement instead, add property-based and contract tests where rules are wide, and prove the suite with mutation testing, because coverage says the code ran, not that a bug would be caught.

The fee incident I describe in the seven skills guide had fourteen green tests. I went back and read them afterwards. Every one had the same shape: call the function with some inputs, copy whatever it returned into the assertion. The assistant had not tested the shared charges rule. It had photographed the implementation and framed the picture.

That is not a model problem you can prompt your way out of with “write better tests”. It is an information problem. If the only thing the model sees is the code, the only thing it can assert is the code. The requirement has to come from somewhere else, and that somewhere is you. The test design method I teach analysts, deriving cases from rules, boundaries, and failure modes rather than from what the system happens to do, is the core of API Testing and QA Mastery for BAs. It transfers directly to unit and integration tests.

What is a mirror test, and how do you spot one?

A mirror test asserts the current output of the code under test without an independent reason for that output. Here is one, generated for a function that splits cross-border fees by charge bearer (ChrgBr in ISO 20022):

# Mirror test: expected values copied from the implementation's output
def test_split_shar():
    debtor, creditor = split_fees(sending_fee=Decimal("12.00"),
                                  receiving_fee=Decimal("8.00"),
                                  bearer="SHAR")
    assert debtor == Decimal("20.00")
    assert creditor == Decimal("0.00")

Under SHAR, the debtor pays the sending agent’s fee and the creditor bears the receiving side. The correct split is 12.00 and 8.00. The test encodes the bug and calls it a specification.

Four signs you are looking at mirror tests: expected values with no comment or reference explaining where they came from; test names that describe the function (test_split_shar) rather than the rule (test_shar_debtor_pays_sending_fee_only); no negative or boundary cases; and tests that were updated in the same commit that changed the behaviour, with no requirement change to justify it.

How do you derive tests from the requirement instead?

Name the rule, then derive the cases from the rule. The requirement-derived version:

import pytest
from decimal import Decimal as D

# FR-FEE-03: charge bearer determines who pays which agent's fee.
# Source: scheme rulebook section 4.2, worked examples from Payments Ops.
@pytest.mark.parametrize("bearer, expected_debtor, expected_creditor", [
    ("DEBT", D("20.00"), D("0.00")),   # debtor pays all fees
    ("CRED", D("0.00"),  D("20.00")),  # creditor pays all, deducted from amount
    ("SHAR", D("12.00"), D("8.00")),   # each side pays its own agent
])
def test_fr_fee_03_bearer_allocates_fees(bearer, expected_debtor, expected_creditor):
    debtor, creditor = split_fees(D("12.00"), D("8.00"), bearer)
    assert (debtor, creditor) == (expected_debtor, expected_creditor)

def test_fr_fee_03_unknown_bearer_is_rejected():
    with pytest.raises(InvalidChargeBearer):
        split_fees(D("12.00"), D("8.00"), "SLEV_TYPO")

def test_fr_fee_04_cred_fee_exceeding_amount_is_rejected():
    # FR-FEE-04: a CRED transfer whose fees exceed the amount is rejected, not credited as negative.
    with pytest.raises(FeesExceedAmount):
        net_credit(amount=D("15.00"), fees=D("20.00"), bearer="CRED")

The derivation is mechanical once you have the rule: one case per equivalence class (each bearer value), the invalid class (unknown bearer), and the boundaries (fees equal to, one cent under, and one cent over the amount). The expected values come from the rulebook and from a worked example someone in operations signed off, not from running the code. For the full catalogue of unhappy paths worth deriving, see Negative Test Design.

How should you prompt AI to write tests?

Give it the oracle and withhold the implementation. This is the prompt shape I use:

You are writing pytest tests for a rule. You do NOT have the implementation.

Rule FR-FEE-03: <paste the requirement text>
Inputs and valid ranges: <fields, types, min, max, allowed values>
Worked examples (authoritative): <input -> expected output, from the business>
Failure behaviour: <what must be rejected, with which error>

Write one parametrized test per rule, with the rule id in the test name.
Include every equivalence class, the invalid class, and each boundary.
After the tests, list every case where you could not determine the expected
result from the rule and examples. Do not guess those values.

The last instruction is the most valuable. The list of cases the model could not derive is a list of gaps in the requirement, and each one is a question for the business before it becomes a defect. I keep a library of these prompts, for test derivation, gap interrogation, and adversarial review, in The Tech BA Prompt Toolkit.

How does mutation testing prove your tests catch bugs?

Coverage tells you a line executed. Mutation testing tells you whether a test would notice if that line were wrong. The tool makes small changes to the code (a > becomes >=, a constant changes, a branch is removed), reruns the suite for each one, and reports the mutants that survived.

# Python
pip install mutmut
mutmut run
mutmut results

# JavaScript / TypeScript
npx stryker run

# Java (Maven)
mvn org.pitest:pitest-maven:mutationCoverage

Run it on the fee module with the mirror tests and the SHAR mutant, where the creditor share is replaced with zero, survives: the tests already expected zero. Run it with the requirement-derived tests and it dies. That difference is the whole argument for this article in one report.

Practical rules for using it on a real codebase:

• Scope it to the modules that carry business rules (fees, limits, status transitions, validation). Running it across the whole repository is slow and mostly noise.
• Track the mutation score per module over time rather than setting one global threshold on day one.
• Treat each surviving mutant as a question: is this a missing test, or is the mutated code unreachable and deletable? Both answers improve the codebase.
• Run it on pull requests that touch rule modules, or nightly, not on every commit.

When should you use property-based testing?

When the rule has to hold for every input and the input space is too big to enumerate. Money handling is the canonical case. Instead of choosing examples, you state the invariant and let the tool search for a counterexample:

from hypothesis import given, strategies as st
from decimal import Decimal

fees = st.decimals(min_value=Decimal("0"), max_value=Decimal("10000"), places=2)
bearers = st.sampled_from(["DEBT", "CRED", "SHAR"])

@given(sending=fees, receiving=fees, bearer=bearers)
def test_split_always_sums_to_total_fee(sending, receiving, bearer):
    debtor, creditor = split_fees(sending, receiving, bearer)
    assert debtor + creditor == sending + receiving
    assert debtor >= 0 and creditor >= 0

Hypothesis generates hundreds of combinations and, when one fails, shrinks it to the smallest input that still fails, which is usually the one that explains the bug. The same idea exists as fast-check for TypeScript and jqwik for Java.

Good invariants to look for in payments and business code: totals are conserved across a split or allocation; a serialise then parse round-trip returns the original; a state machine never reaches a state that is not in the allowed transition table; applying the same request twice gives the same result as applying it once, which is the property behind idempotency testing; and currency amounts never carry more decimals than the currency allows.

What are characterization tests, and why do you need them before an AI refactor?

A characterization test pins the current behaviour of code you do not fully understand, so that any change in behaviour becomes visible. It is the one legitimate use of a mirror test, and the difference is intent: you label it as a record of current behaviour, not as a statement of correct behaviour.

The workflow before asking an assistant to refactor legacy code:

1. Capture real inputs from logs or a test environment (masked, following your data guardrails).
2. Record the current outputs as approved snapshots, using approval testing libraries such as ApprovalTests or snapshot assertions.
3. Mark the file clearly: # CHARACTERIZATION: current behaviour, not verified against requirements.
4. Let the refactor run. Any snapshot diff is either an intended change you can justify or a regression.
5. Promote the snapshots you have verified against a requirement into real tests, and delete the rest over time.

Without step 2, “the refactor passed the tests” means very little, because an assistant refactoring code is also perfectly capable of refactoring the tests to match.

Which tests belong at the service boundary?

Unit tests prove a function. The expensive failures in distributed systems happen between services: a field renamed, an enum value added, a status code changed from 200 to 202. Two kinds of test catch these.

Contract tests check that a provider still satisfies what each consumer relies on. Consumer-driven tools such as Pact record the consumer’s expectations and verify them against the provider in the provider’s pipeline. Schema-based checks with oasdiff flag breaking changes to an OpenAPI contract before merge. The method and the failure cases are in Contract Testing.

Integration tests against real dependencies replace mocks that an assistant wrote to agree with the implementation. Testcontainers starts a real PostgreSQL, Kafka, or Redis in the test run, so the test exercises the real driver, real transactions, and real serialisation. A mock generated alongside the code tends to share the code’s assumptions, which is exactly the thing you are trying to test.

For event-driven flows, validate the event itself: the schema, the key, the headers, and the consumer side effect, with a correlation identifier you generated. I walk through doing that repeatably in a collection, with a CI gate, in Automate Kafka Validation with Postman.

What does a test review checklist look like now?

When you review a pull request, review the tests before the code. Seven checks:

1. Does every new rule have a test named after the rule or requirement id?
2. Where did each expected value come from? If the answer is “the output”, it is a mirror test.
3. Are the invalid class and the boundaries covered, not only the happy path?
4. Did any existing test change? If so, which requirement changed to justify it?
5. Are mocks standing in for something that should be exercised for real (database, broker, serialisation)?
6. For rule-heavy modules, what is the mutation score before and after this change?
7. Would a support engineer understand from the test names what the system is supposed to do?

Item 4 catches the most dangerous AI failure I see: a test that failed, and was edited until it passed. That is not a fix. It is a deletion of evidence, and it deserves a blocking comment every time. For how these checks fit into the wider review, see Code Review When AI Wrote the Diff.

The takeaway

AI made test code cheap. It did not make correctness cheap, because correctness needs an oracle, and the oracle lives in the requirement, the rulebook, and the worked example, not in the implementation. Derive tests from rules, withhold the code when you prompt, state invariants as properties, pin legacy behaviour before a refactor, test the boundaries between services for real, and run mutation testing to prove your suite would catch the bug you are worried about.

Do that and your green build means something again, which is the only reason to have one. Wire the suites into your pipeline with the patterns in Running API Tests in CI.

For the complete test design method with banking examples, start with API Testing and QA Mastery for BAs, or browse everything at The Tech BA Toolkit. More hands-on testing lives in the QA Analyst hub and the Developer Analyst hub.

Ahmed is a Senior Technical Business Analyst with 10+ years in banking and payments. He builds practical guides and tools for analysts at The Tech BA Toolkit.

Tags: Software Testing, Banking, Career Growth, Python, Artificial Intelligence