How do you validate an event flow?

Write down the expected chain of events for one business transaction, with the topic each event should appear on and the consumer that should react. Then find each event in order, by the transaction identifier. The first link you cannot find is where the flow stopped. The logs of the consumer that should have produced it tell you why.

What is a dead letter queue?

A dead letter queue (DLQ) is a topic where a consumer sends messages it could not process, instead of retrying forever or dropping them. Messages in a DLQ carry the original payload plus headers describing the error and the source. Nothing downstream happens for those messages until someone fixes the cause and replays them.

Why did the merchant not receive the settled webhook?

Because the settled event was never produced. Settlement only reacts to ledger entries, the ledger never posted one for this payment, and the ledger never posted one because it rejected the captured event on schema validation and sent it to its dead letter queue. The webhook dispatcher behaved correctly: there was nothing to deliver.

Labs Mission 02 Independent 60 min

Mission 02: Validate an Event Flow and Find Where the Payment Stopped

A hands-on lab for technical analysts: follow one payment across Kafka topics, consumer logs, and a dead letter queue, and prove why the settlement event never arrived.

QA Analyst Systems Analyst Developer Analyst Event flow validationLog analysis

Mission card

Your role: You are the technical analyst on the Northline Pay platform team, on support rotation. A merchant ticket about a missing webhook lands on you.
Deliverable: A written diagnosis: where in the event chain the payment stopped, the root cause with the evidence line, the blast radius, and the replay plan.
System: Northline Pay: A fictional payment service provider for marketplaces. Every lab is set inside it.
Self-assessment: 8 expected findings in the solution

The situation

It is Monday morning. A merchant ticket from Friday is at the top of your queue: their finance team closed the week and one payment captured on Wednesday has no settlement. They received the captured webhook, then nothing. Their integration is fine, they say, and the dashboard agrees with them: the payment shows as captured and nothing else.

You have read access to the Kafka topics, to the service logs, and to the event catalogue. Nobody has looked yet. You are the first pair of eyes, and the answer you write will decide whether this is one payment or a platform incident.

What you will practice

Turning an architecture into an expected sequence of events you can check one by one.
Tracing a single identifier across topics, offsets, and log lines.
Reading absence as evidence: the event that is not there is the finding.
Separating the cause from the symptom, and sizing the blast radius before you write the diagnosis.

How the mission works

Three steps. The first gives you the ticket and the expected flow. The second gives you the events that exist for this payment. The third gives you the consumer logs and a sample of the dead letter queue. Each step has one task. Write your answers as you go: the diagnosis at the end should be assembled from what you wrote, not reconstructed from memory.

The steps

Start step 1

Read first

Platform artifacts

Available from the start. Mission-specific evidence arrives with each step.

Platform artifact topics.yaml 146 lines download show

# Northline Pay event catalogue
# Owner: platform team. Every topic is keyed by payment_id so all events of
# one payment land on the same partition, in order.
#
# Conventions
#   - event_id is unique per event, prefixed evt_.
#   - occurred_at is RFC 3339 in UTC with the Z suffix. Consumers validate it.
#   - Schemas are JSON Schema. A consumer that fails validation sends the
#     message to its dead-letter topic and does not retry.

topics:
  - name: payments.payment.authorized
    version: 1
    key: payment_id
    producer: orchestrator
    consumers: [webhook-dispatcher, risk]
    public: true          # delivered to merchants as payment.authorized
    schema:
      type: object
      required: [event_id, event_type, occurred_at, payment_id, merchant_id, amount, currency]
      properties:
        event_id: { type: string, pattern: "^evt_[A-Za-z0-9]{16}$" }
        event_type: { type: string, const: payment.authorized }
        occurred_at: { type: string, format: date-time }
        payment_id: { type: string, pattern: "^pay_[A-Za-z0-9]{16}$" }
        merchant_id: { type: string, pattern: "^mer_[A-Za-z0-9]{8}$" }
        amount: { type: integer, minimum: 1 }
        currency: { type: string, enum: [CAD, EUR] }

  - name: payments.payment.captured
    version: 1
    key: payment_id
    producer: orchestrator
    consumers: [ledger, webhook-dispatcher]
    public: true          # delivered as payment.captured
    schema:
      type: object
      required: [event_id, event_type, occurred_at, payment_id, merchant_id, captured_amount, currency, captured_at]
      properties:
        event_id: { type: string }
        event_type: { type: string, const: payment.captured }
        occurred_at: { type: string, format: date-time }
        payment_id: { type: string }
        merchant_id: { type: string }
        captured_amount: { type: integer, minimum: 1 }
        currency: { type: string, enum: [CAD, EUR] }
        captured_at: { type: string, format: date-time }

  - name: payments.payment.failed
    version: 1
    key: payment_id
    producer: orchestrator
    consumers: [webhook-dispatcher]
    public: true
    schema:
      type: object
      required: [event_id, event_type, occurred_at, payment_id, merchant_id, failure_code]
      properties:
        event_id: { type: string }
        event_type: { type: string, const: payment.failed }
        occurred_at: { type: string, format: date-time }
        payment_id: { type: string }
        merchant_id: { type: string }
        failure_code: { type: string }
        failure_message: { type: string }

  - name: refunds.refund.succeeded
    version: 1
    key: payment_id
    producer: orchestrator
    consumers: [ledger, webhook-dispatcher]
    public: true
    schema:
      type: object
      required: [event_id, event_type, occurred_at, payment_id, refund_id, merchant_id, amount, currency]
      properties:
        event_id: { type: string }
        event_type: { type: string, const: refund.succeeded }
        occurred_at: { type: string, format: date-time }
        payment_id: { type: string }
        refund_id: { type: string, pattern: "^re_[A-Za-z0-9]{16}$" }
        merchant_id: { type: string }
        amount: { type: integer, minimum: 1 }
        currency: { type: string, enum: [CAD, EUR] }

  - name: ledger.entry.posted
    version: 1
    key: payment_id
    producer: ledger
    consumers: [settlement]
    public: false
    schema:
      type: object
      required: [event_id, event_type, occurred_at, payment_id, entry_id, merchant_id, kind, amount, currency]
      properties:
        event_id: { type: string }
        event_type: { type: string, const: ledger.entry.posted }
        occurred_at: { type: string, format: date-time }
        payment_id: { type: string }
        entry_id: { type: string, pattern: "^led_[A-Za-z0-9]{16}$" }
        merchant_id: { type: string }
        kind: { type: string, enum: [capture, refund] }
        amount: { type: integer }
        currency: { type: string, enum: [CAD, EUR] }

  - name: settlement.payment.settled
    version: 1
    key: payment_id
    producer: settlement
    consumers: [webhook-dispatcher]
    public: true          # delivered as payment.settled
    schema:
      type: object
      required: [event_id, event_type, occurred_at, payment_id, settlement_id, merchant_id, amount, currency]
      properties:
        event_id: { type: string }
        event_type: { type: string, const: payment.settled }
        occurred_at: { type: string, format: date-time }
        payment_id: { type: string }
        settlement_id: { type: string, pattern: "^set_[A-Za-z0-9]{12}$" }
        merchant_id: { type: string }
        amount: { type: integer }
        currency: { type: string, enum: [CAD, EUR] }

dead_letter_topics:
  - name: ledger.dlq
    owner: ledger
    description: Messages the ledger consumer could not validate or process. Each message keeps the original payload and adds error, source_topic, source_offset, and failed_at headers.
  - name: settlement.dlq
    owner: settlement
  - name: webhook-dispatcher.dlq
    owner: webhook-dispatcher

consumer_groups:
  ledger:
    topics: [payments.payment.captured, refunds.refund.succeeded]
    dead_letter: ledger.dlq
    on_validation_error: dead_letter   # no retry, message is never reprocessed automatically
  settlement:
    topics: [ledger.entry.posted]
    dead_letter: settlement.dlq
  webhook-dispatcher:
    topics: [payments.payment.authorized, payments.payment.captured, payments.payment.failed, refunds.refund.succeeded, settlement.payment.settled]
    dead_letter: webhook-dispatcher.dlq
    delivery_retries: 8               # exponential backoff over 24 hours

Next mission: Investigate a production incident

Free account

Save your progress on the Labs

A free account, no password: an email link signs you in. Your steps and your self-assessment are saved, your missions show on a dashboard, and the solutions unlock.

Your email is used to sign you in. Nothing else, unless you ask. Privacy.