Draw Sequence Diagrams from Splunk and Datadog Logs: The Flow as It Actually Ran

You can build an accurate sequence diagram directly from Splunk or Datadog. Query every log line carrying one correlation identifier, sort by timestamp, export the service, direction, operation, and status fields, and have a model transcribe that ordered list into Mermaid. The result is not documentation, it is evidence: every arrow has a log line behind it.

The first time I did this was not a documentation exercise. We were three hours into an incident, the design document said the payment went through five services, and I could not make the timings add up. So I pulled every log line for one stuck uetr, sorted them, and drew what I saw. There were six services. The extra one was a fraud check that had been added eight months earlier and never made it into any diagram, and it was the one timing out.

That is the real argument for this technique. A diagram drawn from logs tells you what the system does, which is frequently not what anyone believes it does. This is the evidence source I trust most in the diagrams as code with AI workflow, and it is available to any analyst with read access to a log tool.

What has to be true before this works?

One thing: every service has to log a shared correlation identifier, and you have to know its name.

Call it trace_id, request_id, correlation_id, X-Request-ID, or in payments uetr. The name does not matter, the propagation does. If the identifier survives every hop, including the asynchronous ones through a queue, you can reconstruct any transaction. If it dies at the queue boundary, you can reconstruct the synchronous half and you have just found a real observability gap worth raising.

Check this before you do anything else. Pick a transaction you know completed, search for its identifier with no other filter, and count the distinct services that come back. If you get two and you expected six, fix the propagation before you try to draw anything. Reading logs as an investigative skill is the foundation here, and the fundamentals are in reading production logs.

How do you pull the sequence out of Splunk?

Splunk’s transaction command exists for exactly this: it groups scattered events into one logical unit.

index=payments uetr="7f3c9a21-4b8e-4f2a-9c6d-1e5b8a0d3f47"
| transaction uetr
| table _time service direction operation status duration_ms
| sort _time

That gives you a time-ordered table, one row per hop. The four fields that matter for a diagram are:

• service, who is acting
• direction, inbound or outbound, which tells you which way the arrow points
• operation, what was called, such as POST /payments or publish payment.received
• status, the outcome, such as 202, ACSP, timeout

If your logs do not carry direction, derive it. An inbound line usually logs the endpoint being served; an outbound line logs the target. Where that is genuinely ambiguous, the useful trick is to pair the two services by timestamp: the service that logged first is the caller.

For the wider view, before you pick a transaction to draw, find out how many distinct paths exist:

index=payments earliest=-24h
| stats list(service) as path, count by uetr
| stats count by path
| sort -count

This is one of the highest value queries an analyst can run. It tells you the real call paths and their frequencies. If it returns nine distinct paths and your design document describes one, you have nine diagrams to draw and eight conversations to have. Pick the most common path for the canonical diagram and the rarest ones for the interesting findings.

How do you pull the sequence out of Datadog?

Datadog gives you two routes, and the first one is almost too easy.

With APM instrumented, a trace already is a sequence diagram. Each span carries a service, a resource name, a parent span id, a start timestamp, and a duration. The parent and child structure is exactly the nesting a sequence diagram needs; the flame graph is the same information rendered horizontally. Open the trace, export the spans as JSON, and you have a perfect input:

[
  {"service":"ingestion-api","resource":"POST /payments","start":"10:42:01.004","duration_ms":38,"parent":null},
  {"service":"ingestion-api","resource":"kafka.produce payment.received","start":"10:42:01.030","duration_ms":6,"parent":"POST /payments"},
  {"service":"processor","resource":"kafka.consume payment.received","start":"10:42:01.210","duration_ms":812,"parent":null},
  {"service":"processor","resource":"POST sanctions/screen","start":"10:42:01.240","duration_ms":740,"parent":"kafka.consume payment.received"},
  {"service":"processor","resource":"swift.send pacs.008","start":"10:42:02.005","duration_ms":15,"parent":"kafka.consume payment.received"}
]

Without APM, use the Logs Explorer. Filter on your correlation attribute, add service, status, and your operation field as columns, sort ascending by timestamp, and export to CSV:

@uetr:7f3c9a21-4b8e-4f2a-9c6d-1e5b8a0d3f47

Either way you end up with the same shape: an ordered list of hops with a service, an operation, a status, and a duration. That shape is the entire input to the next step.

Turning the export into a diagram

Now the transcription. Give the model three things: your context pack, the export, and an instruction that forbids invention.

Transcribe the log export below into a Mermaid sequenceDiagram.

Rules:
- One participant per distinct service, in first-appearance order.
- One arrow per log line. Do not merge, reorder, or infer any call
  that is not present in the export.
- Solid ->> for outbound requests, dashed -->> for responses.
- Add `Note over X,Y` with the elapsed time for any gap over 500ms.
- After the diagram, list separately: (a) any line you could not place,
  (b) any arrow whose direction was ambiguous, (c) any gap in the
  timeline that suggests a missing log line.

Section (c) is the one that earns its keep. When the model reports “a 4.2 second gap between the sanctions response at 10:42:02 and the next logged event, with no intervening line,” you are either looking at an unlogged internal step or at the thing that is slow. Both are findings.

The output for the trace above:

sequenceDiagram
    autonumber
    participant C as Channel
    participant I as Ingestion API
    participant K as Kafka
    participant P as Processor
    participant S as Sanctions
    participant N as Network

    C->>I: POST /payments
    I-->>C: 202 RCVD (38ms)
    I->>K: publish payment.received
    Note over K,P: 180ms consumer lag
    K->>P: consume payment.received
    P->>S: POST /screen
    S-->>P: 200 CLEAR (740ms)
    Note over P,S: 740ms, 91% of total processing time
    P->>N: pacs.008

Notice what that annotation does. The sanctions call is ninety one percent of the processing time, which is invisible in the log table and obvious in the diagram. A diagram from logs carries performance information for free, because the timestamps were already there.

Verify every arrow before you publish it

Four minutes, and it is the entire quality gate. Read the diagram back arrow by arrow and find each one in the export. Models get three things wrong here with some regularity:

• Direction reversal on responses, especially when the log line does not carry an explicit direction field.
• Collapsing retries. Three identical outbound calls at 10:42:02, 10:42:04, and 10:42:08 become one arrow unless you check. The retries are usually the point.
• Inventing the implied. A database write that the flow obviously requires but that nothing in the export actually logged. This one is dangerous, because it is plausible.

A diagram that survives this check is trustworthy in a way a hand drawn one is not, because every arrow has a receipt. That is what makes it usable as evidence in an incident review rather than as decoration.

The diff that is worth more than the diagram

Here is where this stops being a documentation task and becomes analysis.

Generate the diagram from logs. Put it next to the diagram from the design document. Then look at what is different. Every time I have done this on a system older than about a year, there has been a difference, and it has always fallen into one of four buckets:

1. A participant nobody documented. The fraud check from my opening story. Usually added under time pressure, usually never diagrammed.
2. A retry loop that is not in any specification. Visible in the logs as three identical calls, invisible in the design, and frequently the cause of duplicate side effects. Which is why idempotency testing exists.
3. A call that was supposed to be decommissioned. Still firing, still costing money, still a dependency in the incident chain.
4. An ordering that is different from the design. Screening running after the ledger write rather than before, which is a compliance conversation rather than a documentation one.

None of these are diagram problems. They are findings, and the log export is the evidence that makes raising them a short conversation rather than a debate. Working from symptom to root cause with this kind of evidence is the method in how a technical BA investigates a failed payment.

Drawing the incident, not just the happy path

The highest value version of this technique runs during or immediately after an incident.

Take the incident window. Pull one failing transaction and one succeeding transaction from the same window. Generate both diagrams. Put them side by side. The divergence point is visible instantly, and it is visible to people who do not read logs.

sequenceDiagram
    autonumber
    participant P as Processor
    participant S as Sanctions
    participant N as Network

    P->>S: POST /screen
    Note over P,S: timeout after 30000ms
    S--xP: no response
    P->>P: retry 1 of 3
    P->>S: POST /screen
    Note over P,S: timeout after 30000ms
    S--xP: no response
    P->>P: payment stuck in ACCP

--x is the Mermaid arrow for a failed or lost message, and it is worth knowing for exactly this. Attach that diagram to the postmortem and the whole room understands the failure in ten seconds, including the people who were never going to read the log excerpt. That is the part that helps non technical stakeholders, and it costs five minutes once you have the query saved.

Save the queries. A saved search in Splunk or a saved view in Datadog, parameterised by the correlation identifier, means the next incident starts with a diagram rather than a scroll. The broader set of skills this sits inside is in The Technical Skills Guide for BAs.

Making it continuous

Once the query and the prompt are stable, the last step is to stop doing it by hand.

A scheduled job runs the saved search for yesterday’s most common path, regenerates the diagram, and diffs it against the committed file in the repo. A clean diff means the documented flow is still the real flow. Any diff is either a change nobody documented or behavior nobody intended, and both deserve a look. This is the maintenance loop from diagrams as code with AI, wired to the most reliable evidence source you have.

Before any of it leaves your environment, redact. Export service, direction, operation, status, and duration_ms, and drop the message body. Replace the identifier with a placeholder. A diagram needs the shape of the flow and never the payload, so redaction here costs you nothing at all. The wider data classification judgment is in AI guardrails for analysts.

The takeaway

A sequence diagram built from Splunk or Datadog is the only diagram of your system that is provably true, because every arrow is backed by a timestamped log line. The technique is three steps: query every event carrying one correlation identifier, export service, direction, operation, status, and duration in time order, and have a model transcribe it into Mermaid with a strict instruction not to infer anything.

Then do the thing that actually creates value: diff it against the design document and treat every difference as a finding. Next, generate the same kind of diagram from a controlled run rather than production traffic with diagrams from Bruno and Postman runs, or set up the maintenance loop in diagrams as code with AI. The full investigation toolkit is at The Tech BA Toolkit.

Ahmed is a Senior Technical Business Analyst with 10+ years in banking and payments. He builds practical guides and tools for analysts at The Tech BA Toolkit.

Tags: Production Support, Observability, Splunk, Datadog, Systems Analysis