01Read the ticket and write the expected flow
Turn the merchant's complaint and the platform architecture into a chain of events you can verify one link at a time.
The ticket says what the merchant sees, not what happened
Merchant tickets describe symptoms in the merchant’s vocabulary: a missing webhook, a payment not in a report. Your first job is to translate the symptom into the platform’s vocabulary: which event, on which topic, was supposed to trigger the thing that did not happen.
Read the ticket for three things: the payment identifier, the timeline the merchant gives, and what they did receive. What they did receive is as useful as what they did not. If the captured webhook arrived, then everything up to and including the captured event worked. That already rules out half the platform.
Write the chain before you look at any data
From the settlement flow and the event catalogue, write the expected chain for this payment as a numbered list. For each link: the event type, the topic, the producer, the consumer, and roughly when it should have happened relative to the capture. The chain is your checklist for the next step. Without it you will read the events looking for something wrong. With it you will read them looking for the first link that is missing, which is a different and faster kind of reading.
Decide the first check
There is one query that splits the problem in half. Decide what it is before you get the data. Usually it is: “does the event that should have triggered the missing thing exist on its topic?” If it exists, the problem is downstream of it. If it does not, the problem is upstream. Write both branches down.
Your task
Write the expected chain for one captured payment: each event, the topic it appears on, the service that produces it, and the service that consumes it. Then write the single first check you would run, and what each possible result would tell you.
Evidence for this step
Read it in the page or download it and open it in your own tools.
Mission artifact ticket.md 18 lines download show
# NLP-4821: payment.settled webhook never received
Reporter: support@maisonverte.example (Maison Verte marketplace, merchant `mer_A7K3P2Q9`)
Opened: 2026-09-12 16:41 UTC
Priority: P2 (finance blocked)
Assignee: platform support rotation
## Merchant's message
> Hi, our finance team is closing the week and one payment is missing from settlement. Payment `pay_5Rt8kM2xQ7wL9nB4`, captured on Wednesday 10 September around 09:15 UTC, 340.00 EUR. We received the `payment.authorized` and `payment.captured` webhooks a few seconds apart, both verified fine. According to your docs we should get `payment.settled` at T+2, so Friday. Nothing arrived, and the payment is not in the settlement report in the dashboard either. Our endpoint has been up all week, no errors on our side. Can you check what happened on yours?
## Support first-line notes
- Merchant dashboard: payment shows status `captured`, captured 2026-09-10 09:15:31 UTC, EUR 34000. Settlement: none.
- Webhook delivery log in the dashboard: 2 deliveries, both 200. No failed deliveries for this merchant this week.
- Checked `settlement.payment.settled` topic for the payment id: no event.
- Escalating to platform support for the event chain.Mission artifact settlement-flow.md 27 lines download show
# Capture to settlement: the expected flow
Source: platform team runbook, section "Money flow". Applies to card payments with automatic or manual capture.
## Sequence for one payment
| Step | Event / action | Topic | Producer | Consumer | Expected timing |
|---|---|---|---|---|---|
| 1 | Payment authorized | `payments.payment.authorized` | orchestrator | webhook-dispatcher, risk | at authorization |
| 2 | Payment captured | `payments.payment.captured` | orchestrator | ledger, webhook-dispatcher | at capture (seconds after 1 for automatic capture) |
| 3 | Ledger entry posted (kind `capture`) | `ledger.entry.posted` | ledger | settlement | within 5 seconds of 2 |
| 4 | Payment included in an open settlement | (database only: `settlement_items`) | settlement | none | within 1 minute of 3 |
| 5 | Settlement paid, payment settled | `settlement.payment.settled` | settlement | webhook-dispatcher | T+2 business days, 06:00 UTC batch |
| 6 | `payment.settled` webhook delivered | (HTTP to merchant) | webhook-dispatcher | merchant | within minutes of 5 |
## Consumer behavior that matters
- The ledger validates every message against the schema in the event catalogue before processing. A message that fails validation is sent to `ledger.dlq` with the error in the headers. There is no automatic retry and no alert below 5,000 DLQ messages per hour.
- Settlement only knows about a payment through `ledger.entry.posted`. It never reads `payments.*` topics.
- The webhook dispatcher delivers what it consumes. It cannot deliver an event that was never produced.
## Where to look
- Kafka UI: search by key (the payment id) across `payments.*`, `ledger.*`, `settlement.*`. Dead letter topics are keyed by source offset, search them by payload.
- Logs: `ledger-service`, `settlement-service`, `webhook-dispatcher`, filter on `payment_id`.
- Database: `ledger_entries` and `settlement_items` by `payment_id`.Platform artifact topics.yaml 146 lines download show
# Northline Pay event catalogue
# Owner: platform team. Every topic is keyed by payment_id so all events of
# one payment land on the same partition, in order.
#
# Conventions
# - event_id is unique per event, prefixed evt_.
# - occurred_at is RFC 3339 in UTC with the Z suffix. Consumers validate it.
# - Schemas are JSON Schema. A consumer that fails validation sends the
# message to its dead-letter topic and does not retry.
topics:
- name: payments.payment.authorized
version: 1
key: payment_id
producer: orchestrator
consumers: [webhook-dispatcher, risk]
public: true # delivered to merchants as payment.authorized
schema:
type: object
required: [event_id, event_type, occurred_at, payment_id, merchant_id, amount, currency]
properties:
event_id: { type: string, pattern: "^evt_[A-Za-z0-9]{16}$" }
event_type: { type: string, const: payment.authorized }
occurred_at: { type: string, format: date-time }
payment_id: { type: string, pattern: "^pay_[A-Za-z0-9]{16}$" }
merchant_id: { type: string, pattern: "^mer_[A-Za-z0-9]{8}$" }
amount: { type: integer, minimum: 1 }
currency: { type: string, enum: [CAD, EUR] }
- name: payments.payment.captured
version: 1
key: payment_id
producer: orchestrator
consumers: [ledger, webhook-dispatcher]
public: true # delivered as payment.captured
schema:
type: object
required: [event_id, event_type, occurred_at, payment_id, merchant_id, captured_amount, currency, captured_at]
properties:
event_id: { type: string }
event_type: { type: string, const: payment.captured }
occurred_at: { type: string, format: date-time }
payment_id: { type: string }
merchant_id: { type: string }
captured_amount: { type: integer, minimum: 1 }
currency: { type: string, enum: [CAD, EUR] }
captured_at: { type: string, format: date-time }
- name: payments.payment.failed
version: 1
key: payment_id
producer: orchestrator
consumers: [webhook-dispatcher]
public: true
schema:
type: object
required: [event_id, event_type, occurred_at, payment_id, merchant_id, failure_code]
properties:
event_id: { type: string }
event_type: { type: string, const: payment.failed }
occurred_at: { type: string, format: date-time }
payment_id: { type: string }
merchant_id: { type: string }
failure_code: { type: string }
failure_message: { type: string }
- name: refunds.refund.succeeded
version: 1
key: payment_id
producer: orchestrator
consumers: [ledger, webhook-dispatcher]
public: true
schema:
type: object
required: [event_id, event_type, occurred_at, payment_id, refund_id, merchant_id, amount, currency]
properties:
event_id: { type: string }
event_type: { type: string, const: refund.succeeded }
occurred_at: { type: string, format: date-time }
payment_id: { type: string }
refund_id: { type: string, pattern: "^re_[A-Za-z0-9]{16}$" }
merchant_id: { type: string }
amount: { type: integer, minimum: 1 }
currency: { type: string, enum: [CAD, EUR] }
- name: ledger.entry.posted
version: 1
key: payment_id
producer: ledger
consumers: [settlement]
public: false
schema:
type: object
required: [event_id, event_type, occurred_at, payment_id, entry_id, merchant_id, kind, amount, currency]
properties:
event_id: { type: string }
event_type: { type: string, const: ledger.entry.posted }
occurred_at: { type: string, format: date-time }
payment_id: { type: string }
entry_id: { type: string, pattern: "^led_[A-Za-z0-9]{16}$" }
merchant_id: { type: string }
kind: { type: string, enum: [capture, refund] }
amount: { type: integer }
currency: { type: string, enum: [CAD, EUR] }
- name: settlement.payment.settled
version: 1
key: payment_id
producer: settlement
consumers: [webhook-dispatcher]
public: true # delivered as payment.settled
schema:
type: object
required: [event_id, event_type, occurred_at, payment_id, settlement_id, merchant_id, amount, currency]
properties:
event_id: { type: string }
event_type: { type: string, const: payment.settled }
occurred_at: { type: string, format: date-time }
payment_id: { type: string }
settlement_id: { type: string, pattern: "^set_[A-Za-z0-9]{12}$" }
merchant_id: { type: string }
amount: { type: integer }
currency: { type: string, enum: [CAD, EUR] }
dead_letter_topics:
- name: ledger.dlq
owner: ledger
description: Messages the ledger consumer could not validate or process. Each message keeps the original payload and adds error, source_topic, source_offset, and failed_at headers.
- name: settlement.dlq
owner: settlement
- name: webhook-dispatcher.dlq
owner: webhook-dispatcher
consumer_groups:
ledger:
topics: [payments.payment.captured, refunds.refund.succeeded]
dead_letter: ledger.dlq
on_validation_error: dead_letter # no retry, message is never reprocessed automatically
settlement:
topics: [ledger.entry.posted]
dead_letter: settlement.dlq
webhook-dispatcher:
topics: [payments.payment.authorized, payments.payment.captured, payments.payment.failed, refunds.refund.succeeded, settlement.payment.settled]
dead_letter: webhook-dispatcher.dlq
delivery_retries: 8 # exponential backoff over 24 hours