Queries · q094 · Head of Data · week 1, night 3
“How much of the gap between order value and 'captured' payments is really just that refunded/partially-refunded payments were excluded by my status filter, versus a genuine hole where money was never collected?”
Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told
Week 1; shared with Tobias Lindqvist, Chief Financial Officer
Before the board pack goes out, Data must certify the key metrics. Do revenue, orders, customers, refunds and payouts agree wherever they are recorded, and where they do not, why not and by how much?
The manager, at the start of week 1
Subject: Certifying the key metrics before the board pack — what I need from you this week
Hi Yuki,
Tobias and I have to certify revenue, orders, customers, refunds and payouts before they go into the board pack. "Certify" means we're telling the board these numbers are reconcilable — so this week's job is to find out where the different places we record these figures agree, where they don't, and whether any gap is big enough to change the story we tell. If something doesn't reconcile, I need to know why and by how much, not just that it's "close enough." That feeds directly into the go/no-go Tobias and I give on the board pack, so please prioritize breadth across the five metrics over depth on any one of them — we can always dig deeper next week on whichever one looks shakiest.
You only have a handful of report slots a night, so here's what matters most to me, roughly in priority order:
Please flag anything you find inconsistent together with your best read of why, and roughly how big the discrepancy is in practice — that's what Tobias and I need to decide whether we can certify as-is, certify with a caveat, or need to hold a number back.
Thank you — I know this is a lot for one week. Let's talk before you spend your first slot tonight if you want to sanity-check the plan.
Rosa
Subject: What I need for the revenue/payout certification
Hi Yuki,
A few things from my side as you scope this week's work:
If anything else jumps out while you're in there — especially around currency conversion timing or refund treatment — flag it even if it's outside these three.
Thanks, Tobias
The analyst · week 1, night 3, request 1 of 3
Night 2 gave three follow-ups worth running tonight. (1) The order-status breakdown showed "refunded" orders at a near-total 99.8% gap and "fulfilled" orders at a persistent 22% gap — I suspect both are partly a filter artifact: my captured-payment match only counted payments.status='captured', but a payment that was later refunded moves to status 'refunded'/'partially_refunded' and would be invisible to that filter even though money genuinely changed hands. I want to re-run with a status-inclusive filter to see how much of the gap is this artifact vs a real reconciliation hole, especially for 'fulfilled' orders where there's no reason to expect any gap at all. (2) The payment-linked payout/refund check got noisier, not cleaner, than the calendar-month version (swinging -16.6% to +17.7% with no consistent sign) — payout_items has a unique constraint on payment_id (one row per payment, ever), so a refund processed after a payment's payout has already settled has no slot in that payout to be reflected in; it would have to hit some later, differently-linked settlement, breaking the clean per-payout linkage I assumed. I want to directly test how often refunds are processed after their payment's payout already settled, since that would explain the noise and is itself a traceability gap worth flagging to Tobias. (3) Rosa's customer-count question (#4) hasn't been touched yet and we're entering the back half of the week, so I'm using tonight's third slot to get a first cut at how many "customers" we have under different plausible definitions, to see if that's a quick certify or another can of worms for night 4.
The engineer · 1 dry run on the empty schema, then the submission
WITH t0 AS ( SELECT o0.id AS order_id, o0.status AS order_status, o0.grand_total / er0.rate AS grand_total_eur FROM orders o0 JOIN exchange_rates er0 ON er0.base_code = 'EUR' AND er0.quote_code = o0.currency_code AND er0.as_of = CAST(o0.placed_at AS date) WHERE o0.placed_at >= TIMESTAMP '2023-01-01 00:00:00' AND o0.placed_at < TIMESTAMP '2025-07-01 00:00:00' ), t1 AS ( SELECT p1.order_id AS order_id, p1.amount / er1.rate AS gross_eur, (p1.amount - p1.amount_refunded) / er1.rate AS net_eur FROM payments p1 JOIN exchange_rates er1 ON er1.base_code = 'EUR' AND er1.quote_code = p1.currency_code AND er1.as_of = CAST(p1.captured_at AS date) WHERE p1.status IN ('captured', 'refunded', 'partially_refunded') AND p1.order_id IN (SELECT t0.order_id FROM t0) ), t2 AS ( SELECT t1.order_id AS order_id, SUM(t1.gross_eur) AS gross_eur, SUM(t1.net_eur) AS net_eur FROM t1 GROUP BY t1.order_id ), t3 AS ( SELECT t0.order_status AS order_status, COUNT(t0.order_id) AS order_count, SUM(t0.grand_total_eur) AS orders_grand_total_eur FROM t0 GROUP BY t0.order_status ), t4 AS ( SELECT t0.order_status AS order_status, COUNT(t2.order_id) AS orders_with_money_collected_count, SUM(COALESCE(t2.gross_eur, 0)) AS gross_collected_eur, SUM(COALESCE(t2.net_eur, 0)) AS net_collected_eur FROM t0 LEFT JOIN t2 ON t2.order_id = t0.order_id GROUP BY t0.order_status ) SELECT t3.order_status AS order_status, t3.order_count AS order_count, t3.orders_grand_total_eur AS orders_grand_total_eur, t4.orders_with_money_collected_count AS orders_with_money_collected_count, t4.gross_collected_eur AS gross_collected_eur, t4.net_collected_eur AS net_collected_eur, t3.orders_grand_total_eur - t4.gross_collected_eur AS gap_vs_gross_eur, ROUND( (t3.orders_grand_total_eur - t4.gross_collected_eur) / NULLIF(t3.orders_grand_total_eur, 0) * 100, 1 ) AS gap_vs_gross_pct FROM t3 JOIN t4 ON t4.order_status = t3.order_status ORDER BY t3.orders_grand_total_eur DESC, t3.order_status ASC
“The query compiles and returns the requested columns. Submitting now.”
“Submitted. The report groups orders by orders.status over the 2023-01-01 to 2025-06-30 window, converts order totals to EUR using the placed_at date and payment amounts using the captured_at date, and compares orders_grand_total_eur against both gross and net collected amounts from payments with status in captured/refunded/partially_refunded — isolating how much of the original gap was really refund-inclusive collections versus a true collection shortfall.”
As Yuki Tanaka saw it, from the small dataset the agents worked on
order_status | order_count | orders_grand_total_eur | orders_with_money_collected_count | gross_collected_eur | net_collected_eur | gap_vs_gross_eur | gap_vs_gross_pct -------------+-------------+------------------------+-----------------------------------+---------------------+-------------------+--------------------+----------------- fulfilled | 18540 | 4863934.962385263 | 16619 | 4822446.930504212 | 4433540.985117448 | 41488.03188105114 | 0.9 cancelled | 788 | 158380.30196644366 | 259 | 46949.160132157034 | 0.0 | 111431.14183428662 | 70.4 refunded | 923 | 125051.87291907084 | 923 | 118419.97198707449 | 2606.396122521615 | 6631.900931996352 | 5.3 (3 rows)
The memo and the queries that followed; links found by title, so inferred
Later queries that build on this report:
It builds on:
Changed only where the benchmark needs the same result on every system
WITH t0 AS ( SELECT o0.id AS order_id, o0.status AS order_status, o0.grand_total / er0.rate AS grand_total_eur CAST(o0.grand_total / er0.rate AS NUMERIC(18,6)) AS grand_total_eur FROM orders o0 JOIN exchange_rates er0 ON er0.base_code = 'EUR' AND er0.quote_code = o0.currency_code AND er0.as_of = CAST(o0.placed_at AS date) WHERE o0.placed_at >= TIMESTAMP '2023-01-01 00:00:00' AND o0.placed_at < TIMESTAMP '2025-07-01 00:00:00'),t1 AS ( SELECT p1.order_id AS order_id, p1.amount / er1.rate AS gross_eur, (p1.amount - p1.amount_refunded) / er1.rate AS net_eur CAST(p1.amount / er1.rate AS NUMERIC(18,6)) AS gross_eur, CAST((p1.amount - p1.amount_refunded) / er1.rate AS NUMERIC(18,6)) AS net_eur FROM payments p1 JOIN exchange_rates er1 ON er1.base_code = 'EUR' AND er1.quote_code = p1.currency_code AND er1.as_of = CAST(p1.captured_at AS date) WHERE p1.status IN ('captured', 'refunded', 'partially_refunded') AND p1.order_id IN (SELECT t0.order_id FROM t0) AND p1.order_id IN (SELECT t0.order_id AS order_id FROM t0)),t2 AS ( SELECT t1.order_id AS order_id, SUM(t1.gross_eur) AS gross_eur, SUM(t1.net_eur) AS net_eur FROM t1 GROUP BY t1.order_id),t3 AS ( SELECT t0.order_status AS order_status, COUNT(t0.order_id) AS order_count, SUM(t0.grand_total_eur) AS orders_grand_total_eur FROM t0 GROUP BY t0.order_status),t4 AS ( SELECT t0.order_status AS order_status, COUNT(t2.order_id) AS orders_with_money_collected_count, SUM(COALESCE(t2.gross_eur, 0)) AS gross_collected_eur, SUM(COALESCE(t2.net_eur, 0)) AS net_collected_eur FROM t0 LEFT JOIN t2 ON t2.order_id = t0.order_id GROUP BY t0.order_status)SELECT t3.order_status AS order_status, t3.order_count AS order_count, t3.orders_grand_total_eur AS orders_grand_total_eur, t4.orders_with_money_collected_count AS orders_with_money_collected_count, t4.gross_collected_eur AS gross_collected_eur, t4.net_collected_eur AS net_collected_eur, t3.orders_grand_total_eur - t4.gross_collected_eur AS gap_vs_gross_eur, ROUND( (t3.orders_grand_total_eur - t4.gross_collected_eur) / NULLIF(t3.orders_grand_total_eur, 0) * 100, 1 ) AS gap_vs_gross_pct (t3.orders_grand_total_eur - t4.gross_collected_eur) / NULLIF(t3.orders_grand_total_eur, 0) * 100 AS gap_vs_gross_pctFROM t3JOIN t4 ON t4.order_status = t3.order_statusORDER BY t3.orders_grand_total_eur DESC, t3.order_status ASCORDER BY t3.orders_grand_total_eur DESC NULLS FIRST, t3.order_status ASC NULLS LAST
WITH t0 AS ( SELECT o0.id AS order_id, o0.status AS order_status, CAST(o0.grand_total / er0.rate AS NUMERIC(18,6)) AS grand_total_eur FROM orders o0 JOIN exchange_rates er0 ON er0.base_code = 'EUR' AND er0.quote_code = o0.currency_code AND er0.as_of = CAST(o0.placed_at AS date) WHERE o0.placed_at >= TIMESTAMP '2023-01-01 00:00:00' AND o0.placed_at < TIMESTAMP '2025-07-01 00:00:00' ), t1 AS ( SELECT p1.order_id AS order_id, CAST(p1.amount / er1.rate AS NUMERIC(18,6)) AS gross_eur, CAST((p1.amount - p1.amount_refunded) / er1.rate AS NUMERIC(18,6)) AS net_eur FROM payments p1 JOIN exchange_rates er1 ON er1.base_code = 'EUR' AND er1.quote_code = p1.currency_code AND er1.as_of = CAST(p1.captured_at AS date) WHERE p1.status IN ('captured', 'refunded', 'partially_refunded') AND p1.order_id IN (SELECT t0.order_id AS order_id FROM t0) ), t2 AS ( SELECT t1.order_id AS order_id, SUM(t1.gross_eur) AS gross_eur, SUM(t1.net_eur) AS net_eur FROM t1 GROUP BY t1.order_id ), t3 AS ( SELECT t0.order_status AS order_status, COUNT(t0.order_id) AS order_count, SUM(t0.grand_total_eur) AS orders_grand_total_eur FROM t0 GROUP BY t0.order_status ), t4 AS ( SELECT t0.order_status AS order_status, COUNT(t2.order_id) AS orders_with_money_collected_count, SUM(COALESCE(t2.gross_eur, 0)) AS gross_collected_eur, SUM(COALESCE(t2.net_eur, 0)) AS net_collected_eur FROM t0 LEFT JOIN t2 ON t2.order_id = t0.order_id GROUP BY t0.order_status ) SELECT t3.order_status AS order_status, t3.order_count AS order_count, t3.orders_grand_total_eur AS orders_grand_total_eur, t4.orders_with_money_collected_count AS orders_with_money_collected_count, t4.gross_collected_eur AS gross_collected_eur, t4.net_collected_eur AS net_collected_eur, t3.orders_grand_total_eur - t4.gross_collected_eur AS gap_vs_gross_eur, (t3.orders_grand_total_eur - t4.gross_collected_eur) / NULLIF(t3.orders_grand_total_eur, 0) * 100 AS gap_vs_gross_pct FROM t3 JOIN t4 ON t4.order_status = t3.order_status ORDER BY t3.orders_grand_total_eur DESC NULLS FIRST, t3.order_status ASC NULLS LAST
The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted
Report request from Yuki Tanaka (analyst for the Head of Data):
Title: Order-to-payment gap re-measured with refund-inclusive payment status
Question: How much of the gap between order value and 'captured' payments is really just that refunded/partially-refunded payments were excluded by my status filter, versus a genuine hole where money was never collected?
Grain: order status
Columns:
- order_status
- order_count
- orders_grand_total_eur
- orders_with_money_collected_count
- gross_collected_eur
- net_collected_eur
- gap_vs_gross_eur
- gap_vs_gross_pct
Details: Same population as the prior order-status report: orders with placed_at between 2023-01-01 and 2025-06-30 inclusive. Group by orders.status. order_count and orders_grand_total_eur as before (grand_total converted to EUR via exchange_rates using orders.currency_code and placed_at date). orders_with_money_collected_count = count of those orders with at least one linked payments row (payments.order_id = orders.id) whose status is in ('captured','refunded','partially_refunded') - i.e. any payment that was successfully captured at some point, whether or not it was later refunded. gross_collected_eur = sum of payments.amount for those linked payments (the original captured amount, before any refund), converted to EUR via payments.currency_code and captured_at date. net_collected_eur = sum of (payments.amount - payments.amount_refunded) for those same linked payments, converted the same way - this is what the merchant actually kept after refunds. gap_vs_gross_eur = orders_grand_total_eur - gross_collected_eur. gap_vs_gross_pct = gap_vs_gross_eur / orders_grand_total_eur * 100, rounded to 1 decimal. Tables: orders, payments, exchange_rates. Order by orders_grand_total_eur descending.
Check your query with the warehouse's dry_run tool, then hand it in with submit.dry_runWITH t0 AS ( SELECT o0.id AS order_id, o0.status AS order_status, o0.grand_total / er0.rate AS grand_total_eur FROM orders o0 JOIN exchange_rates er0 ON er0.base_code = 'EUR' AND er0.quote_code = o0.currency_code AND er0.as_of = CAST(o0.placed_at AS date) WHERE o0.placed_at >= TIMESTAMP '2023-01-01 00:00:00' AND o0.placed_at < TIMESTAMP '2025-07-01 00:00:00' ), t1 AS ( SELECT p1.order_id AS order_id, p1.amount / er1.rate AS gross_eur, (p1.amount - p1.amount_refunded) / er1.rate AS net_eur FROM payments p1 JOIN exchange_rates er1 ON er1.base_code = 'EUR' AND er1.quote_code = p1.currency_code AND er1.as_of = CAST(p1.captured_at AS date) WHERE p1.status IN ('captured', 'refunded', 'partially_refunded') AND p1.order_id IN (SELECT t0.order_id FROM t0) ), t2 AS ( SELECT t1.order_id AS order_id, SUM(t1.gross_eur) AS gross_eur, SUM(t1.net_eur) AS net_eur FROM t1 GROUP BY t1.order_id ), t3 AS ( SELECT t0.order_status AS order_status, COUNT(t0.order_id) AS order_count, SUM(t0.grand_total_eur) AS orders_grand_total_eur FROM t0 GROUP BY t0.order_status ), t4 AS ( SELECT t0.order_status AS order_status, COUNT(t2.order_id) AS orders_with_money_collected_count, SUM(COALESCE(t2.gross_eur, 0)) AS gross_collected_eur, SUM(COALESCE(t2.net_eur, 0)) AS net_collected_eur FROM t0 LEFT JOIN t2 ON t2.order_id = t0.order_id GROUP BY t0.order_status ) SELECT t3.order_status AS order_status, t3.order_count AS order_count, t3.orders_grand_total_eur AS orders_grand_total_eur, t4.orders_with_money_collected_count AS orders_with_money_collected_count, t4.gross_collected_eur AS gross_collected_eur, t4.net_collected_eur AS net_collected_eur, t3.orders_grand_total_eur - t4.gross_collected_eur AS gap_vs_gross_eur, ROUND( (t3.orders_grand_total_eur - t4.gross_collected_eur) / NULLIF(t3.orders_grand_total_eur, 0) * 100, 1 ) AS gap_vs_gross_pct FROM t3 JOIN t4 ON t4.order_status = t3.order_status ORDER BY t3.orders_grand_total_eur DESC, t3.order_status ASC
STATUS: COMPILES (dry run on the empty schema, no data read) COLUMNS: order_status, order_count, orders_grand_total_eur, orders_with_money_collected_count, gross_collected_eur, net_collected_eur, gap_vs_gross_eur, gap_vs_gross_pct
submitWITH t0 AS ( SELECT o0.id AS order_id, o0.status AS order_status, o0.grand_total / er0.rate AS grand_total_eur FROM orders o0 JOIN exchange_rates er0 ON er0.base_code = 'EUR' AND er0.quote_code = o0.currency_code AND er0.as_of = CAST(o0.placed_at AS date) WHERE o0.placed_at >= TIMESTAMP '2023-01-01 00:00:00' AND o0.placed_at < TIMESTAMP '2025-07-01 00:00:00' ), t1 AS ( SELECT p1.order_id AS order_id, p1.amount / er1.rate AS gross_eur, (p1.amount - p1.amount_refunded) / er1.rate AS net_eur FROM payments p1 JOIN exchange_rates er1 ON er1.base_code = 'EUR' AND er1.quote_code = p1.currency_code AND er1.as_of = CAST(p1.captured_at AS date) WHERE p1.status IN ('captured', 'refunded', 'partially_refunded') AND p1.order_id IN (SELECT t0.order_id FROM t0) ), t2 AS ( SELECT t1.order_id AS order_id, SUM(t1.gross_eur) AS gross_eur, SUM(t1.net_eur) AS net_eur FROM t1 GROUP BY t1.order_id ), t3 AS ( SELECT t0.order_status AS order_status, COUNT(t0.order_id) AS order_count, SUM(t0.grand_total_eur) AS orders_grand_total_eur FROM t0 GROUP BY t0.order_status ), t4 AS ( SELECT t0.order_status AS order_status, COUNT(t2.order_id) AS orders_with_money_collected_count, SUM(COALESCE(t2.gross_eur, 0)) AS gross_collected_eur, SUM(COALESCE(t2.net_eur, 0)) AS net_collected_eur FROM t0 LEFT JOIN t2 ON t2.order_id = t0.order_id GROUP BY t0.order_status ) SELECT t3.order_status AS order_status, t3.order_count AS order_count, t3.orders_grand_total_eur AS orders_grand_total_eur, t4.orders_with_money_collected_count AS orders_with_money_collected_count, t4.gross_collected_eur AS gross_collected_eur, t4.net_collected_eur AS net_collected_eur, t3.orders_grand_total_eur - t4.gross_collected_eur AS gap_vs_gross_eur, ROUND( (t3.orders_grand_total_eur - t4.gross_collected_eur) / NULLIF(t3.orders_grand_total_eur, 0) * 100, 1 ) AS gap_vs_gross_pct FROM t3 JOIN t4 ON t4.order_status = t3.order_status ORDER BY t3.orders_grand_total_eur DESC, t3.order_status ASC
Submitted to tonight's batch. Your work on this request is done.