03Read the logs and the dead letter queue
The ledger and webhook dispatcher logs around the capture, plus a sample of the ledger's dead letter queue. Name the root cause, size the blast radius, write the replay plan.
Logs answer why; the DLQ answers how many
You now know which consumer stopped the chain. Its logs, filtered on the payment id and on the minutes around the capture, will tell you why. Read them for the exact error, and then read them for what happened just before: a deployment, a config change, a restart. Root causes have a start time.
The webhook dispatcher log is there for a reason too. Part of a diagnosis is saying which services behaved correctly. A merchant who was told “the webhook system is broken” will not trust the next ticket answer. If the dispatcher delivered everything it was given, say so.
Size it before you write it
A diagnosis that covers one payment is a support answer. A diagnosis that says how many payments are affected is an incident report. The dead letter queue sample gives you what you need: the count, the time of the first message, and whether every message fails for the same reason. Compare the first failure time with what the ledger log shows just before it.
The replay plan
Messages in a dead letter queue do not process themselves. Once the cause is fixed, someone replays them into the consumer, in order, and verifies that each one produces the downstream event this time. Your plan should say what must be fixed first, what is replayed, from where to where, and how you will confirm the merchant’s payment settles. Order matters: replaying before the fix sends every message straight back to the DLQ.
Write the diagnosis
One page, in this order: what the merchant saw, where the chain stopped, the root cause and the evidence line, what changed and when, the blast radius, the fix, the replay plan, and what you will tell the merchant. Then open the solution and score yourself on the findings you actually wrote down.
Your task
Write the diagnosis: the root cause in one sentence with the log line that proves it, what changed and when, how many payments are affected and how you know, and the replay plan in order. Then compare with the solution.
Evidence for this step
Read it in the page or download it and open it in your own tools.
Mission artifact ledger-service.log 13 lines download show
2026-09-10T09:11:40.118Z INFO [ledger-consumer] topic=payments.payment.captured partition=3 offset=5517902 payment_id=pay_2Nc8vB4mK7qX1tL6 producer=orchestrator/2.13.2 msg="entry posted" entry_id=led_8Ks2mP4wR7tQ9nX1 kind=capture amount=1299 currency=CAD duration_ms=41
2026-09-10T09:11:40.166Z INFO [ledger-producer] topic=ledger.entry.posted partition=3 offset=2210417 payment_id=pay_2Nc8vB4mK7qX1tL6 event_id=evt_Lp4wQ9sT2mK7xR8n msg="published"
2026-09-10T09:11:55.002Z INFO [ledger-consumer] msg="consumer group healthy" group=ledger lag=0 partitions=12
2026-09-10T09:12:47.531Z WARN [ledger-consumer] topic=payments.payment.captured partition=1 offset=5518004 payment_id=pay_6Tg3hW8kL2mQ4rY9 producer=orchestrator/2.14.0 msg="schema validation failed: $.captured_at: '2026-09-10 09:12:47' is not a valid date-time (RFC 3339)" action=dead_letter dlq=ledger.dlq dlq_offset=1188
2026-09-10T09:12:49.870Z WARN [ledger-consumer] topic=payments.payment.captured partition=9 offset=5518007 payment_id=pay_9Xb2nM5tP7kL3wQ8 producer=orchestrator/2.14.0 msg="schema validation failed: $.captured_at: '2026-09-10 09:12:49' is not a valid date-time (RFC 3339)" action=dead_letter dlq=ledger.dlq dlq_offset=1189
2026-09-10T09:13:02.114Z INFO [ledger-consumer] topic=refunds.refund.succeeded partition=4 offset=1993320 payment_id=pay_4Wm7rK2sQ9tL5nB3 producer=orchestrator/2.14.0 msg="entry posted" entry_id=led_3Rt9kM2pW7sQ4xL8 kind=refund amount=-4500 currency=EUR duration_ms=38
2026-09-10T09:13:02.160Z INFO [ledger-producer] topic=ledger.entry.posted partition=4 offset=2210418 payment_id=pay_4Wm7rK2sQ9tL5nB3 event_id=evt_Wq2sM8kT4pR9xL1n msg="published"
2026-09-10T09:15:31.412Z WARN [ledger-consumer] topic=payments.payment.captured partition=7 offset=5518240 payment_id=pay_5Rt8kM2xQ7wL9nB4 producer=orchestrator/2.14.0 msg="schema validation failed: $.captured_at: '2026-09-10 09:15:31' is not a valid date-time (RFC 3339)" action=dead_letter dlq=ledger.dlq dlq_offset=1207
2026-09-10T09:15:33.905Z WARN [ledger-consumer] topic=payments.payment.captured partition=2 offset=5518243 payment_id=pay_1Kq8wT3mR6sL9pN2 producer=orchestrator/2.14.0 msg="schema validation failed: $.captured_at: '2026-09-10 09:15:33' is not a valid date-time (RFC 3339)" action=dead_letter dlq=ledger.dlq dlq_offset=1208
2026-09-10T09:16:55.003Z INFO [ledger-consumer] msg="consumer group healthy" group=ledger lag=0 partitions=12
2026-09-10T09:21:12.640Z INFO [ledger-consumer] topic=refunds.refund.succeeded partition=11 offset=1993331 payment_id=pay_8Lm4pT7wK2rQ6sN9 producer=orchestrator/2.14.0 msg="entry posted" entry_id=led_6Pw3kR8mT2sQ9xL4 kind=refund amount=-12000 currency=CAD duration_ms=44
2026-09-10T09:21:12.688Z INFO [ledger-producer] topic=ledger.entry.posted partition=11 offset=2210419 payment_id=pay_8Lm4pT7wK2rQ6sN9 event_id=evt_Rk7sT2mW9pQ4xL8n msg="published"Mission artifact webhook-dispatcher.log 8 lines download show
2026-09-10T09:15:29.220Z INFO [dispatcher] event_id=evt_Qm3xR8tK2pL9wN4s type=payment.authorized payment_id=pay_5Rt8kM2xQ7wL9nB4 merchant_id=mer_A7K3P2Q9 msg="delivery scheduled" attempt=1
2026-09-10T09:15:29.611Z INFO [dispatcher] event_id=evt_Qm3xR8tK2pL9wN4s type=payment.authorized payment_id=pay_5Rt8kM2xQ7wL9nB4 merchant_id=mer_A7K3P2Q9 url=https://api.maisonverte.example/webhooks/northline status=200 duration_ms=371 msg="delivered"
2026-09-10T09:15:31.702Z INFO [dispatcher] event_id=evt_Hs7wT2yM9kR4pQ1x type=payment.captured payment_id=pay_5Rt8kM2xQ7wL9nB4 merchant_id=mer_A7K3P2Q9 msg="delivery scheduled" attempt=1
2026-09-10T09:15:32.088Z INFO [dispatcher] event_id=evt_Hs7wT2yM9kR4pQ1x type=payment.captured payment_id=pay_5Rt8kM2xQ7wL9nB4 merchant_id=mer_A7K3P2Q9 url=https://api.maisonverte.example/webhooks/northline status=200 duration_ms=366 msg="delivered"
2026-09-12T06:00:14.905Z INFO [dispatcher] msg="settlement batch received" topic=settlement.payment.settled events=4213 window=2026-09-10
2026-09-12T06:03:41.377Z INFO [dispatcher] msg="settlement batch delivered" delivered=4198 retrying=15 failed=0 window=2026-09-10
2026-09-14T08:05:10.120Z INFO [dispatcher] msg="query" filter="payment_id=pay_5Rt8kM2xQ7wL9nB4" results=2 types="payment.authorized,payment.captured"Mission artifact ledger-dlq-sample.jsonl 7 lines download show
{"_export": {"tool": "kafka-ui", "topic": "ledger.dlq", "exported_at": "2026-09-14T08:10:41Z", "total_messages": 1842, "earliest": "2026-09-10T09:12:47.531Z", "latest": "2026-09-14T07:58:03.216Z", "distinct_errors": 1, "sample": "first 4 messages and the message for pay_5Rt8kM2xQ7wL9nB4"}}
{"offset": 1188, "timestamp": "2026-09-10T09:12:47.531Z", "headers": {"x-source-topic": "payments.payment.captured", "x-source-partition": "1", "x-source-offset": "5518004", "x-error": "schema validation failed: $.captured_at: '2026-09-10 09:12:47' is not a valid date-time (RFC 3339)", "x-failed-at": "2026-09-10T09:12:47.531Z", "x-consumer": "ledger", "x-producer-version": "2.14.0"}, "value": {"event_id": "evt_Bn4kW7sT2mQ9pL3x", "event_type": "payment.captured", "occurred_at": "2026-09-10T09:12:47Z", "payment_id": "pay_6Tg3hW8kL2mQ4rY9", "merchant_id": "mer_C3D8F1G5", "captured_amount": 5900, "currency": "CAD", "captured_at": "2026-09-10 09:12:47"}}
{"offset": 1189, "timestamp": "2026-09-10T09:12:49.870Z", "headers": {"x-source-topic": "payments.payment.captured", "x-source-partition": "9", "x-source-offset": "5518007", "x-error": "schema validation failed: $.captured_at: '2026-09-10 09:12:49' is not a valid date-time (RFC 3339)", "x-failed-at": "2026-09-10T09:12:49.870Z", "x-consumer": "ledger", "x-producer-version": "2.14.0"}, "value": {"event_id": "evt_Ts9pL2wK7mR4xQ8n", "event_type": "payment.captured", "occurred_at": "2026-09-10T09:12:49Z", "payment_id": "pay_9Xb2nM5tP7kL3wQ8", "merchant_id": "mer_A7K3P2Q9", "captured_amount": 12900, "currency": "EUR", "captured_at": "2026-09-10 09:12:49"}}
{"offset": 1190, "timestamp": "2026-09-10T09:12:51.204Z", "headers": {"x-source-topic": "payments.payment.captured", "x-source-partition": "5", "x-source-offset": "5518009", "x-error": "schema validation failed: $.captured_at: '2026-09-10 09:12:51' is not a valid date-time (RFC 3339)", "x-failed-at": "2026-09-10T09:12:51.204Z", "x-consumer": "ledger", "x-producer-version": "2.14.0"}, "value": {"event_id": "evt_Mw2rQ8kT5pL9sX3n", "event_type": "payment.captured", "occurred_at": "2026-09-10T09:12:51Z", "payment_id": "pay_3Pq7tL9wM2kR8sN5", "merchant_id": "mer_H9J2K6L4", "captured_amount": 2450, "currency": "CAD", "captured_at": "2026-09-10 09:12:51"}}
{"offset": 1191, "timestamp": "2026-09-10T09:12:53.877Z", "headers": {"x-source-topic": "payments.payment.captured", "x-source-partition": "0", "x-source-offset": "5518011", "x-error": "schema validation failed: $.captured_at: '2026-09-10 09:12:53' is not a valid date-time (RFC 3339)", "x-failed-at": "2026-09-10T09:12:53.877Z", "x-consumer": "ledger", "x-producer-version": "2.14.0"}, "value": {"event_id": "evt_Kr8sT4mW2pQ7xL9n", "event_type": "payment.captured", "occurred_at": "2026-09-10T09:12:53Z", "payment_id": "pay_7Lw2kM9tR4sQ8pN3", "merchant_id": "mer_C3D8F1G5", "captured_amount": 8800, "currency": "CAD", "captured_at": "2026-09-10 09:12:53"}}
{"offset": 1207, "timestamp": "2026-09-10T09:15:31.412Z", "headers": {"x-source-topic": "payments.payment.captured", "x-source-partition": "7", "x-source-offset": "5518240", "x-error": "schema validation failed: $.captured_at: '2026-09-10 09:15:31' is not a valid date-time (RFC 3339)", "x-failed-at": "2026-09-10T09:15:31.412Z", "x-consumer": "ledger", "x-producer-version": "2.14.0"}, "value": {"event_id": "evt_Hs7wT2yM9kR4pQ1x", "event_type": "payment.captured", "occurred_at": "2026-09-10T09:15:31Z", "payment_id": "pay_5Rt8kM2xQ7wL9nB4", "merchant_id": "mer_A7K3P2Q9", "captured_amount": 34000, "currency": "EUR", "captured_at": "2026-09-10 09:15:31"}}