How to Test Kafka: Validating Events You Cannot See

Testing Kafka means validating events you cannot see from the UI: subscribe to the topic, assert the event was published with the right schema and key, then verify the consumer handled it correctly. The hard cases, duplicates, ordering, and consumer failure, only show up when you create them on purpose.

To test Kafka, you subscribe to the topic with a test consumer, trigger the action that should produce an event, and assert the event appears with the correct topic, key, schema, and field values. Then you verify the other half: that the service consuming the event produced the right result, the database row, the downstream message, the state change. Then you test the cases that actually break payment systems: a duplicate event, an out-of-order pair, a consumer that fails mid-process. Most analysts never test any of this, because events are invisible from the user interface and missing from the specs, which is exactly why the ones who can are so valuable.

Kafka is where most analysts in payments hit a wall. The happy path of an event-driven system looks fine from the screen, while the real behavior, and the expensive defects, live in the asynchronous gaps you can only see by looking at the topic directly. Learning to validate events without a developer holding your hand is one of the highest-leverage QA analyst skills there is, and it is the entire subject of Automate Kafka Validation with Postman.

Why is testing Kafka harder than testing an API?

Testing Kafka is harder because events are invisible, asynchronous, and their worst behaviors only appear under conditions you have to engineer. None of those three is true for a synchronous API, which gives you an immediate, visible response.

Events are invisible: clicking through the application shows you nothing about the event that fired behind it, and the specification usually does not describe the event schema at all. Events are asynchronous: there is no immediate response to assert, so you have to subscribe, trigger, and wait, with a timeout, for the event to arrive. And the hard cases are engineered: out-of-order delivery, duplicate delivery, and consumer failure do not happen on demand in a normal happy-path test; you have to deliberately create them to see how the system copes.

This combination is why event testing is a genuine technical skill rather than a clicking exercise. You need to be able to consume from a topic, write the assertion, and script the failure conditions, which puts it squarely in developer-analyst territory. The payoff is that you see the part of the system everyone else is guessing about, which is where most production incidents in payments originate. The full mindset of following one transaction through the events, the database, and the logs is in You Don’t Understand the System Until You Test It.

How do you verify an event was published?

Subscribe to the topic before you trigger the action, run the action, then wait for the matching event by filtering on a key you control, and assert it appears with the right type and payload within a timeout. The order matters: subscribe first, or you may miss the event.

In practice you capture a correlation id from the action, the UETR of the payment you submitted, and use it to find your event among everything else on the topic.

// Subscribe, trigger the POST, then wait for the event keyed by uetr
const event = await waitForEvent("payments.received", { uetr }, { timeoutMs: 5000 });
expect(event).to.exist;
expect(event.key).to.equal(uetr);            // key governs ordering
expect(event.value.status).to.equal("RCVD");
expect(event.value.amount).to.equal(expectedAmount);

The assertions cover the three things that can be wrong independently: the event exists at all, its key is correct, and its payload is correct. The key assertion is easy to forget and important, because the key determines partitioning and therefore ordering; an event with the right payload but the wrong key can still cause out-of-order processing downstream. And if the event simply never arrives, you have learned something valuable: the failure is upstream, in whatever was supposed to publish it, not in the consumer everyone assumed was broken. That is a finding you can only get by checking the topic directly, which is the whole point.

What do you assert on the consumer side?

You assert that the event was handled correctly, which means checking the side effects the consumer produced: the database row it wrote, the downstream message it sent, the state it changed. Publishing the event correctly is only half the contract; the other half is the consumer doing the right thing with it.

A payment.received event might be consumed by a processor that writes a payment row and sends a pacs.008 downstream. So after you confirm the event published, you follow it to the consumer’s outputs: query the database and assert the row exists with the right status, check that the downstream message was produced. This is where you catch the consumer that received a valid event and did the wrong thing, or nothing at all. An event that fires correctly but is mishandled is just as broken as an event that never fired, and only a consumer-side assertion distinguishes them.

This two-sided check, producer published correctly and consumer handled correctly, is the complete event test. It mirrors the way a real transaction moves: an action produces an event, a consumer reacts, and state changes as a result. Verifying the database state during a test is its own skill, leaning on SQL, which is part of the developer-analyst toolkit, and the broader progression through these technical skills is in The Technical Skills Guide for BAs.

How do you test the cases that actually break: duplicates, ordering, failure?

You test them by engineering them deliberately, because they are the normal operating conditions of an event-driven system and the source of its worst defects. The happy-path event test passes easily; these are the tests that find real problems.

Duplicates. Because Kafka typically delivers at least once, a consumer will eventually get the same event twice. Deliver it twice in your test and assert the result is identical to processing it once, with no second side effect. A consumer that is not idempotent produces a double debit from a redelivered event, which is exactly the failure that makes this the most important event test in payments. The full method is in idempotency testing.

Ordering. Produce two events for the same key out of order, a payment.accepted before the payment.received, and assert the consumer either rejects the stale one or reconciles correctly, rather than ending in a wrong state. This verifies that the keying and the consumer logic actually preserve order the way the event-driven requirements demanded.

Consumer failure. Make the consumer fail mid-process, a downstream timeout, an exception, and assert the system retries, dead-letters, or otherwise recovers, and that the payment does not silently vanish. A payment whose processing event failed and was never retried is the worst outcome in payments, and you only find these stuck states by deliberately creating the failure. This is systematic negative test design applied to the event layer.

The takeaway

Testing Kafka means validating the events you cannot see from the UI: subscribe to the topic, assert the event was published with the right topic, key, schema, and values, then verify the consumer handled it correctly by checking its side effects. Then engineer the cases that actually break payment systems, duplicates, out-of-order delivery, and consumer failure, because those are the normal conditions of an asynchronous platform and the source of its expensive defects.

This is a technical skill most analysts never build, which is exactly why it sets you apart in payments. Start with Automate Kafka Validation with Postman and The Technical Skills Guide for BAs, or browse everything at The Tech BA Toolkit.

Ahmed is a Senior Technical Business Analyst with 10+ years in banking and payments. He builds practical guides and tools for analysts at The Tech BA Toolkit.

Tags: Software Testing, Kafka, Event-Driven Architecture, QA, Payments