Framework · v1.0
The Analyst Engineering Review Scorecard is a fixed rubric of 12 checks for reviewing an API or event contract before a team integrates with it. Each check scores 0, 1, or 2, for a maximum of 24. Every point lost, and most points kept, becomes a finding with a severity and a question. The same scorecard is applied in public to real APIs in the teardown series below, and practised on the Labs.
Written and maintained by Ahmed at Analyst Engineering, a Senior Technical Business Analyst with 10+ years in banking and payments. Version 1.0, published 2026-09-21. Check ids are permanent: a check is never renumbered, only revised with a version note.
Read the contract and everything the provider publishes around it: the reference, the guides, the changelog, the error list. Then give each check one of three scores. Do not average, do not weight. The value of the scorecard is that two analysts reviewing the same contract land within a point or two of each other.
| Score | Meaning |
|---|---|
| 0 | Absent. The topic is not addressed anywhere the consumer can find. |
| 1 | Partial. Implied, scattered across pages, or stated without numbers ("may", "in some cases", "too many"). |
| 2 | Explicit. Stated once, with numbers or examples, and consistent across the contract. |
| Total | Verdict |
|---|---|
| 20 to 24 | Integrate. Send the question list this week; nothing blocks the start. |
| 14 to 19 | Integrate after answers. Start on the explicit parts; hold the money-moving paths until the high findings are answered in writing. |
| 0 to 13 | Do not start. The contract needs a design review, not an integration. Send the register to the provider. |
Four lenses, three checks each. The question is what you ask of the contract. The second line is what a 2 looks like, so the score is a comparison, not an opinion.
Lens 1
Does the contract say what the business means, unambiguously?
Does every resource, field, and enum map to exactly one business term?
A 2 looks likeOne name per concept across the whole contract, no synonyms, no database or legacy names left beside the current ones, and every enum value explained.
Are every state and every legal transition written down, with the trigger and the terminal states?
A 2 looks likeA complete state list, each transition with who or what causes it, which states are terminal, and every automatic transition quantified (after how many attempts, after how long).
Do amounts carry an explicit unit and currency, and timestamps an explicit zone and precision, the same way everywhere?
A 2 looks likeAmounts as integers in minor units or decimal strings with a currency, never floats; timestamps with a declared epoch or zone and a declared precision; the same rule on every endpoint and event.
Lens 2
Does the contract cover the whole process, including the paths that go wrong?
Is there an operation for every process step, including cancel, correct, and retry?
A 2 looks likeEvery step of the business process has an operation, reversal and correction paths exist, and their constraints (once only, partial allowed, window) are stated.
Does every failure the consumer must handle have a stable code, a status, and a recovery action?
A 2 looks likeA single error shape, a published code list, the HTTP status per code, which messages are safe to show a user, and what the consumer should do next for each class.
Are lengths, sizes, minimums, maximums, rate limits, and retention windows stated as numbers?
A 2 looks likeEvery limit is a number with a unit, including field lengths, amount bounds, request rates, per-object rates, and how long keys, holds, and events are kept.
Lens 3
Can a consumer integrate without losing money or data when something is retried, raced, or leaked?
Do creates and money-moving actions define a key, its scope, its retention, and what happens on mismatch?
A 2 looks likeThe header or field, its maximum length, how long the server remembers it, whether failures are replayed, what a reused key with different parameters returns, and which operations honour it.
Is the behaviour under concurrent writers stated, and are event ordering and duplication stated?
A 2 looks likeLocking or conflict behaviour on the same object, whether events can arrive out of order or more than once, what to deduplicate on, and exactly when duplicates are produced.
Is it explicit which credential can do what, and which values must never be logged or exposed?
A 2 looks likeCredential types and their scopes per operation, signature or token verification rules with tolerances, secret rotation, and a named list of values to keep out of logs and front ends.
Lens 4
Will the integration still be correct in a year, and can you prove what happened?
Are the version mechanism, the definition of a breaking change, and the deprecation policy written down?
A 2 looks likeHow a version is selected per request and per event, what the provider promises not to break, how often versions ship, and how long old ones live.
Does every object and event carry a stable identifier and a handle the consumer can correlate and reconcile with?
A 2 looks likeStable ids on every object and event, a request identifier per call, a place for the consumer's own reference, and a stated way to find an object from that reference.
Do examples, a sandbox, and test fixtures exist for the paths that matter, including failures?
A 2 looks likeRealistic examples per request and response, a sandbox whose behaviour differences from production are documented, and a way to trigger every failure and event type on demand.
A score without findings is a grade, and grades do not change a delivery. Every 0 or 1 produces at least one finding. A 2 produces one whenever the explicit rule still forces a decision on your side (a retention window you must respect, a duplicate you must handle). Rate severity from the consumer's seat, not the provider's.
| Severity | If ignored |
|---|---|
| High | Money or data goes wrong: a duplicate charge, a lost payment, a leaked secret, a state you cannot recover. |
| Medium | An incident or a rework: a parser breaks on an upgrade, an operations team works blind, a hold expires unnoticed. |
| Low | A question to close before build. Cheap now, a change request later. |
| Note | Context the team should know. No action beyond writing it down. |
The register has eight columns, and the last one is the one that gets read:
The Labs solutions use the same severities and the same evidence-first shape, so a mission register and a real review register look alike on purpose.
Free download
The 12 checks as a Markdown checklist you can paste into Confluence or a pull request, and the findings register as a CSV that opens in Excel or Google Sheets. No sign-up.
The series
Each teardown applies the 12 checks to a public contract, with the score, the evidence, and the findings register a consuming team would inherit. Public documentation only, reviewed on a stated date and API version. New teardowns ship roughly every two weeks.
Next in the queue: Adyen webhooks, Wise transfers, and a SEPA Instant usage guideline. A free account with the newsletter box ticked gets each one the day it ships.
Mission 01 of the Labs hands you the Northline Pay OpenAPI contract, the requirements, the sample responses, and the business rules. Score it with the 12 checks, write the register, then compare with the practitioner solution, which uses the same severities. Sixty minutes, free.
The scorecard tells you what to look for. These two guides teach how to write a contract that scores 24, and how to prove an API behaves as its contract says.
Not ready to buy? The free downloads are a no-cost place to start, and every article here stays free.
Free account
A free account, no password. It saves your Labs progress, unlocks Mission 01's solution so you can compare registers, and, if you tick the box, emails you each new scorecard teardown the day it ships.
Your email is used to sign you in. Nothing else, unless you ask. Privacy.