Labs / Investigate a production incident Step 2 of 3

02Requests and logs

The api_requests rows, the gateway log, and the Merchant API log around the two refunds. Build the timeline to the second and find the question the timeline raises.

Three sources, one timeline

You have the same two requests seen from three places: the gateway, which is what the merchant’s client talked to; the Merchant API service, which did the work; and the api_requests table, which is what the service recorded when it finished. They do not agree, and the disagreement is the finding.

Build the timeline in one table, sorted by time, with the source in a column. Put the gateway’s view and the service’s view of the same request on adjacent lines. Look at durations and status codes side by side.

What a 504 means

A gateway returns 504 when it stops waiting for the backend. It says nothing about whether the backend stopped. Read the service log for what the backend did after the gateway gave up. Then read what the client did next, and how long after.

The question the timeline raises

Once you have the timeline you will see two requests and two refunds. That answers “one or two” but opens a harder question. The platform enforces a refund limit: the cumulative refunded amount cannot exceed the captured amount. Two full refunds on one payment should have been impossible. Read the service log for how the limit check ran on each request, and when, relative to the other request. Write down your explanation and what in the database would prove it.

Your task

Write the timeline: every line with a timestamp, the source, the request id, and what happened. Then answer two questions with evidence: did the merchant send one request or two, and why did the second refund pass the refund limit check?

Evidence for this step

Read it in the page or download it and open it in your own tools.

Mission artifact api-requests.csv 12 lines download show

request_id,merchant_id,method,path,status_code,duration_ms,idempotency_key,resource_id,created_at
req_9Lk3mT7wQ2sP8rN5,mer_B2X9L4M7,POST,/v1/refunds,201,412,,re_7Qw2kM9sT4pL8rN3,2026-09-12T13:41:07.118Z
req_2Hs8kW4mT9pL3qR7,mer_F6G1H8J3,POST,/v1/payments,201,388,ord-77120-a,pay_8Kw3mT2sL9pQ4rN6,2026-09-12T14:02:58.402Z
req_7Hn2kQ9wT4sL8mP3,mer_B2X9L4M7,POST,/v1/refunds,201,39612,,re_A4kT7wQ2mL9sP8xN,2026-09-12T14:03:12.004Z
req_4Tm9sK2wL7pQ3rN8,mer_F6G1H8J3,POST,/v1/refunds,201,38904,,re_5Lw8kT3mQ9sP2rN7,2026-09-12T14:03:20.551Z
req_3Wq8mL2tK7pR9sN4,mer_B2X9L4M7,POST,/v1/refunds,201,8720,,re_C8pW3mK6tR2sL5qY,2026-09-12T14:03:43.061Z
req_8Pk4wT9mS2qL7rN3,mer_F6G1H8J3,POST,/v1/refunds,201,7911,,re_2Qs9kW4mT7pL3rN8,2026-09-12T14:03:51.220Z
req_6Rw2kM8tQ4sL9pN7,mer_K2L7M4N9,POST,/v1/refunds,201,30105,,re_9Tk3mW7sQ2pL8rN4,2026-09-12T14:05:14.877Z
req_1Qs7kT4mW9pL2rN8,mer_K2L7M4N9,POST,/v1/refunds,201,6408,,re_4Wm8kT2sQ9pL7rN3,2026-09-12T14:05:45.310Z
req_5Mw3kQ8tL2sP9rN6,mer_B2X9L4M7,GET,/v1/payments/pay_2Fw6hT9jK3sR8vN1,200,31,,pay_2Fw6hT9jK3sR8vN1,2026-09-12T14:11:02.774Z
req_2Kt9mW3sQ7pL4rN8,mer_B2X9L4M7,POST,/v1/refunds,201,402,,re_3Sm7kT9wQ2pL8rN5,2026-09-12T14:12:40.009Z

Mission artifact gateway.log 11 lines download show

2026-09-12T14:03:12.001Z INFO  [edge-gw] client=203.0.113.44 merchant=mer_B2X9L4M7 method=POST path=/v1/refunds request_id=req_7Hn2kQ9wT4sL8mP3 msg="proxying to merchant-api" upstream_timeout_ms=30000
2026-09-12T14:03:20.548Z INFO  [edge-gw] client=198.51.100.17 merchant=mer_F6G1H8J3 method=POST path=/v1/refunds request_id=req_4Tm9sK2wL7pQ3rN8 msg="proxying to merchant-api" upstream_timeout_ms=30000
2026-09-12T14:03:42.010Z ERROR [edge-gw] client=203.0.113.44 merchant=mer_B2X9L4M7 method=POST path=/v1/refunds request_id=req_7Hn2kQ9wT4sL8mP3 status=504 duration_ms=30009 msg="upstream timeout, response sent to client" upstream_connection=kept
2026-09-12T14:03:43.058Z INFO  [edge-gw] client=203.0.113.44 merchant=mer_B2X9L4M7 method=POST path=/v1/refunds request_id=req_3Wq8mL2tK7pR9sN4 msg="proxying to merchant-api" upstream_timeout_ms=30000
2026-09-12T14:03:50.556Z ERROR [edge-gw] client=198.51.100.17 merchant=mer_F6G1H8J3 method=POST path=/v1/refunds request_id=req_4Tm9sK2wL7pQ3rN8 status=504 duration_ms=30008 msg="upstream timeout, response sent to client" upstream_connection=kept
2026-09-12T14:03:51.217Z INFO  [edge-gw] client=198.51.100.17 merchant=mer_F6G1H8J3 method=POST path=/v1/refunds request_id=req_8Pk4wT9mS2qL7rN3 msg="proxying to merchant-api" upstream_timeout_ms=30000
2026-09-12T14:03:51.784Z INFO  [edge-gw] client=203.0.113.44 merchant=mer_B2X9L4M7 method=POST path=/v1/refunds request_id=req_3Wq8mL2tK7pR9sN4 status=201 duration_ms=8726 msg="response sent to client"
2026-09-12T14:03:59.135Z INFO  [edge-gw] client=198.51.100.17 merchant=mer_F6G1H8J3 method=POST path=/v1/refunds request_id=req_8Pk4wT9mS2qL7rN3 status=201 duration_ms=7918 msg="response sent to client"
2026-09-12T14:10:00.000Z INFO  [edge-gw] msg="window summary" window=14:00-14:10 path=/v1/refunds requests=212 status_201=185 status_400=0 status_504=27 p50_ms=1840 p99_ms=30009
2026-09-12T14:20:00.000Z INFO  [edge-gw] msg="window summary" window=14:10-14:20 path=/v1/refunds requests=198 status_201=196 status_400=2 status_504=0 p50_ms=380 p99_ms=910

Mission artifact merchant-api.log 21 lines download show

2026-09-12T14:00:00.214Z INFO  [scheduler] job=ledger_daily_reconciliation msg="started" lock="ledger_entries SHARE ROW EXCLUSIVE"
2026-09-12T14:03:12.006Z INFO  [refunds] request_id=req_7Hn2kQ9wT4sL8mP3 merchant_id=mer_B2X9L4M7 payment_id=pay_2Fw6hT9jK3sR8vN1 idempotency_key=none msg="refund requested" amount=8900
2026-09-12T14:03:12.019Z INFO  [refunds] request_id=req_7Hn2kQ9wT4sL8mP3 payment_id=pay_2Fw6hT9jK3sR8vN1 msg="limit check" captured_amount=8900 refunded_read=0 requested=8900 remaining=8900 result=pass
2026-09-12T14:03:12.031Z INFO  [refunds] request_id=req_7Hn2kQ9wT4sL8mP3 refund_id=re_A4kT7wQ2mL9sP8xN msg="refund row inserted" status=pending
2026-09-12T14:03:12.044Z INFO  [refunds] request_id=req_7Hn2kQ9wT4sL8mP3 refund_id=re_A4kT7wQ2mL9sP8xN msg="acquirer refund accepted" acquirer_ref=ACQ-77Q2M9
2026-09-12T14:03:12.049Z INFO  [ledger-client] request_id=req_7Hn2kQ9wT4sL8mP3 refund_id=re_A4kT7wQ2mL9sP8xN msg="posting ledger entry" kind=refund amount=-8900
2026-09-12T14:03:43.064Z INFO  [refunds] request_id=req_3Wq8mL2tK7pR9sN4 merchant_id=mer_B2X9L4M7 payment_id=pay_2Fw6hT9jK3sR8vN1 idempotency_key=none msg="refund requested" amount=8900
2026-09-12T14:03:43.077Z INFO  [refunds] request_id=req_3Wq8mL2tK7pR9sN4 payment_id=pay_2Fw6hT9jK3sR8vN1 msg="limit check" captured_amount=8900 refunded_read=0 requested=8900 remaining=8900 result=pass
2026-09-12T14:03:43.089Z INFO  [refunds] request_id=req_3Wq8mL2tK7pR9sN4 refund_id=re_C8pW3mK6tR2sL5qY msg="refund row inserted" status=pending
2026-09-12T14:03:43.101Z INFO  [refunds] request_id=req_3Wq8mL2tK7pR9sN4 refund_id=re_C8pW3mK6tR2sL5qY msg="acquirer refund accepted" acquirer_ref=ACQ-77Q2N4
2026-09-12T14:03:43.106Z INFO  [ledger-client] request_id=req_3Wq8mL2tK7pR9sN4 refund_id=re_C8pW3mK6tR2sL5qY msg="posting ledger entry" kind=refund amount=-8900
2026-09-12T14:03:51.402Z INFO  [scheduler] job=ledger_daily_reconciliation msg="finished" duration_ms=231188 rows_reconciled=27640
2026-09-12T14:03:51.588Z INFO  [ledger-client] request_id=req_7Hn2kQ9wT4sL8mP3 refund_id=re_A4kT7wQ2mL9sP8xN msg="ledger entry posted" entry_id=led_5Kw9mT2sQ7pL4rN8 wait_ms=39530
2026-09-12T14:03:51.601Z INFO  [refunds] request_id=req_7Hn2kQ9wT4sL8mP3 refund_id=re_A4kT7wQ2mL9sP8xN payment_id=pay_2Fw6hT9jK3sR8vN1 msg="refund succeeded" refunded_amount_after=8900
2026-09-12T14:03:51.616Z INFO  [http] request_id=req_7Hn2kQ9wT4sL8mP3 status=201 duration_ms=39612 msg="response written" note="client connection closed by gateway"
2026-09-12T14:03:51.769Z INFO  [ledger-client] request_id=req_3Wq8mL2tK7pR9sN4 refund_id=re_C8pW3mK6tR2sL5qY msg="ledger entry posted" entry_id=led_8Tm2kW7sQ4pL9rN3 wait_ms=8655
2026-09-12T14:03:51.778Z INFO  [refunds] request_id=req_3Wq8mL2tK7pR9sN4 refund_id=re_C8pW3mK6tR2sL5qY payment_id=pay_2Fw6hT9jK3sR8vN1 msg="refund succeeded" refunded_amount_after=17800
2026-09-12T14:03:51.784Z INFO  [http] request_id=req_3Wq8mL2tK7pR9sN4 status=201 duration_ms=8720 msg="response written"
2026-09-12T14:03:52.140Z INFO  [events] topic=refunds.refund.succeeded payment_id=pay_2Fw6hT9jK3sR8vN1 refund_id=re_A4kT7wQ2mL9sP8xN msg="published"
2026-09-12T14:03:52.312Z INFO  [events] topic=refunds.refund.succeeded payment_id=pay_2Fw6hT9jK3sR8vN1 refund_id=re_C8pW3mK6tR2sL5qY msg="published"

Next: Database and monitoring