DB Olympics

Queries · q082 · VP Product · week 1, night 3

Where orders stall after checkout, Q4 2025 — retail merchants only (corrected)

“For the real one-off retail merchants (excluding subscription-box sellers), is the drop-off between order creation and payment a single cliff or a slow bleed, and at which status does it happen?”

Manager · asked the questionJonas FeldVP Product. Joined from a mobile-first retailer. Deciding the 2026 roadmap.
Analyst · planned the reportEmeka ObiCustomer-centric: follows cohorts of shoppers or merchants over time.
Engineer · wrote the SQLOmarPragmatic: the shortest correct query, aggregates first and joins dimensions afterwards, FILTER clauses for conditional counts.

Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told

  1. A business event lands on the VP Product's desk

    Week 1

    We are redesigning the shopper journey for 2026. We need to understand the whole path, from the first visit to checkout, delivery, returns and the next purchase, and where shoppers drop out and why.

  2. Jonas Feld writes to Emeka Obi

    The manager, at the start of week 1

    Subject: Shopper journey end-to-end — what I need for the 2026 roadmap

    Emeka,

    We're resetting the shopper journey for 2026, and I need a clear picture of the whole path — first visit through checkout, delivery, returns, and whether people come back — so we can decide where to invest design and engineering time first. This feeds the roadmap review I'm presenting in two weeks, so the Friday memo should leave me able to say "here's where shoppers are leaking out, and here's the one or two places a fix would move the most volume."

    I know you only get a handful of report slots this week, so please prioritize in the order below — stop earlier in the list if something forces a tradeoff, rather than spreading thin across all of it.

    1. Where do we lose people between landing on the storefront and placing an order? I want to see the shape of the funnel (visit → browse/search → cart → checkout → paid order) and which step is the biggest drop-off, ideally for a few of our bigger merchants so we're not averaging away real differences.
    2. Once someone reaches checkout, what's stopping them from finishing? Is it a one-step cliff or a slow bleed across the checkout flow, and does it differ by device or by how they arrived (e.g. ad vs. direct vs. returning)?
    3. After the order is placed, how is delivery experience affecting trust? I want to understand how often deliveries run late and whether that visibly connects to returns or to people not coming back.
    4. How painful are returns, and does a return kill the relationship? I'd like a sense of how often a return leads to the shopper buying again within a reasonable window versus going quiet.
    5. Who comes back, and how soon? A simple read on repeat purchase behavior — how many first-time buyers return, and roughly how long it takes — would help me know if we're building a roadmap for acquisition or for retention.

    If you can only land three of these this week, do 1, 2 and 5 — funnel, checkout, and repeat purchase are the ones I most need going into the roadmap conversation. Delivery and returns can follow next week if we run out of nights.

    Thanks — looking forward to Friday.

    Jonas

  3. Emeka Obi plans the night

    The analyst · week 1, night 3, request 1 of 3

    Reading night 2 critically:

    Report 1 (tracking coverage) confirms the Report-1/night-1 puzzle, but not quite the way I expected: the four Food & Coffee merchants have visits_per_order well under 1 (0.53-0.73) — far more orders than visits — while the retail merchants (Pet Supplies, Beauty, Apparel, Electronics) sit at a believable 1.0-3.3 visits per order. BUT subscription_linked_orders_pct is 0.0% everywhere, so the orders aren't literally tied to subscription invoices via orders.id — meaning subscription billing in this schema doesn't create a browsing/checkout event at all, it just produces order rows directly (automated renewal), with zero session footprint. That's the real explanation: for subscription merchants, "order" ≠ "someone browsed and checked out," so the storefront funnel simply doesn't apply to most of their volume. I'll segment subscription vs. retail merchants in the memo and scope the "where do we leak" funnel conclusion to the retail segment, where the numbers are coherent.

    Report 2 (status-transition funnel) has a methodology problem I caused: I told the engineer "same 10 tenants as the Q4 funnel report" without giving the explicit list, so they substituted a top-order-count proxy — which is almost certainly dominated by the high-volume subscription merchants (confirmed status never appears, pending→paid is ~98% instant at 0.0 median hours, consistent with automated billing that has no checkout UI to abandon). This result doesn't answer the retail checkout question I actually need — I have to rerun it with the explicit 6 retail tenant_ids.

    Report 3 (cart duration) is clean and tenant-matched correctly, and it's a real finding: only 29% of carts resolve within an hour (and those are ~100% converted — fast successful checkouts, not failures); the bulk, 67.7% combined, linger 1 hour to 7 days before mostly failing (1.8%-15.3% conversion). That's a genuine slow bleed, not a single cliff. I want to check this isn't an artifact of one tenant's cart-expiry policy before I put it in the memo as a general pattern.

    Tonight (night 3 of 4): fix the contaminated status-transition report with the explicit retail tenant list, check whether the cart-bleed pattern holds per-tenant or is driven by an outlier, and — since priorities 1/2/5 are now in good shape — spend the third slot starting priority 3 (delivery lateness vs. returns) so there's something to work with next week even if we run out of nights.

    Where orders stall after checkout, Q4 2025 — retail merchants only (corrected)
    Question
    For the real one-off retail merchants (excluding subscription-box sellers), is the drop-off between order creation and payment a single cliff or a slow bleed, and at which status does it happen?
    One row per
    order status
    Columns
    • status
    • orders reaching this status (count)
    • pct of total orders in window
    • median hours from order creation to reaching this status
  4. Omar writes the SQL

    The engineer · 1 dry run on the empty schema, then the submission

    Dry run 1 · STATUS: COMPILES (dry run on the empty schema, no data read)
    WITH target_tenants AS (
      SELECT id FROM tenants
      WHERE name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')
    ),
    base_orders AS (
      SELECT o.id, o.created_at
      FROM orders o
      JOIN target_tenants t ON t.id = o.tenant_id
      WHERE o.created_at >= TIMESTAMP '2025-10-01' AND o.created_at < TIMESTAMP '2026-01-01'
    ),
    total AS (
      SELECT count(*) AS total_orders FROM base_orders
    ),
    statuses AS (
      SELECT unnest(ARRAY['pending','confirmed','paid','partially_fulfilled','fulfilled','cancelled','refunded']) AS status
    ),
    first_reach AS (
      SELECT bo.id AS order_id, osh.to_status AS status, min(osh.created_at) AS first_at
      FROM base_orders bo
      JOIN order_status_history osh ON osh.order_id = bo.id
      WHERE osh.to_status IN ('pending','confirmed','paid','partially_fulfilled','fulfilled','cancelled','refunded')
      GROUP BY bo.id, osh.to_status
    ),
    joined AS (
      SELECT fr.order_id, fr.status, fr.first_at, bo.created_at,
             EXTRACT(EPOCH FROM (fr.first_at - bo.created_at)) / 3600.0 AS hours
      FROM first_reach fr
      JOIN base_orders bo ON bo.id = fr.order_id
    )
    SELECT
      s.status,
      count(j.order_id) AS orders_reaching_status,
      round(100.0 * count(j.order_id) / NULLIF(t.total_orders, 0), 1) AS pct_of_total_orders,
      round(CAST(percentile_cont(0.5) WITHIN GROUP (ORDER BY j.hours) AS numeric), 1) AS median_hours_from_order_creation
    FROM statuses s
    CROSS JOIN total t
    LEFT JOIN joined j ON j.status = s.status
    GROUP BY s.status, t.total_orders
    ORDER BY pct_of_total_orders DESC, s.status ASC

    “Query compiles and returns the expected columns. Submitting now.”

    Submitted · the SQL of the last dry run, unchanged · Submitted to tonight's batch. Your work on this request is done.

    “Submitted. The report gives one row per order status (all 7 included) for the six named retail tenants, with orders reaching each status, percent of total orders in the Q4 2025 window, and median hours from order creation to first reaching that status — ordered by percent descending so the cliff/bleed pattern is immediately visible.”

  5. The report comes back overnight

    As Emeka Obi saw it, from the small dataset the agents worked on

    status              | orders_reaching_status | pct_of_total_orders | median_hours_from_order_creation
    --------------------+------------------------+---------------------+---------------------------------
    pending             |                   2997 |               100.0 |                              0.0
    paid                |                   2924 |                97.6 |                              0.0
    fulfilled           |                   2843 |                94.9 |                             22.8
    refunded            |                    173 |                 5.8 |                            517.0
    cancelled           |                    127 |                 4.2 |                              0.1
    confirmed           |                      0 |                 0.0 |                             NULL
    partially_fulfilled |                      0 |                 0.0 |                             NULL
    (7 rows)
  6. What came of it

    The memo and the queries that followed; links found by title, so inferred

    It builds on:

  7. It becomes benchmark query q082

    Changed only where the benchmark needs the same result on every system

    • editunnest in a select list is a PostgreSQL extension: the statuses as a VALUES list, standard SQL.
    • numericsdocs/NUMERICS.md: rounding removed; results keep full precision; bare NUMERIC cast that only served the rounding removed
    • numericsdocs/NUMERICS.md: EXTRACT(EPOCH FROM (b - a)) cast to NUMERIC(18,6)
    • made explicita selected t.col gets AS col, its standard name made explicit (7×)
    • placeholderpercentile cont
    • placeholderseconds between
    • determinismthe standard leaves the place of NULLs in a sort to the implementation: NULLS LAST ascending, NULLS FIRST descending, as in the reference results (2×)
    The benchmark text, against the submitted SQL
    WITH target_tenants AS (  SELECT id FROM tenants  WHERE name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')),base_orders AS (  SELECT o.id, o.created_at  SELECT o.id AS id, o.created_at AS created_at  FROM orders o  JOIN target_tenants t ON t.id = o.tenant_id  WHERE o.created_at >= TIMESTAMP '2025-10-01' AND o.created_at < TIMESTAMP '2026-01-01'),total AS (  SELECT count(*) AS total_orders FROM base_orders),statuses AS (  SELECT unnest(ARRAY['pending','confirmed','paid','partially_fulfilled','fulfilled','cancelled','refunded']) AS status  SELECT status FROM (VALUES ('pending'), ('confirmed'), ('paid'), ('partially_fulfilled'), ('fulfilled'), ('cancelled'), ('refunded')) AS s(status)),first_reach AS (  SELECT bo.id AS order_id, osh.to_status AS status, min(osh.created_at) AS first_at  FROM base_orders bo  JOIN order_status_history osh ON osh.order_id = bo.id  WHERE osh.to_status IN ('pending','confirmed','paid','partially_fulfilled','fulfilled','cancelled','refunded')  GROUP BY bo.id, osh.to_status),joined AS (  SELECT fr.order_id, fr.status, fr.first_at, bo.created_at,         EXTRACT(EPOCH FROM (fr.first_at - bo.created_at)) / 3600.0 AS hours  SELECT fr.order_id AS order_id, fr.status AS status, fr.first_at AS first_at, bo.created_at AS created_at,         {{seconds_between(bo.created_at, fr.first_at)}} / 3600.0 AS hours  FROM first_reach fr  JOIN base_orders bo ON bo.id = fr.order_id)SELECT  s.status,  s.status AS status,  count(j.order_id) AS orders_reaching_status,  round(100.0 * count(j.order_id) / NULLIF(t.total_orders, 0), 1) AS pct_of_total_orders,  round(CAST(percentile_cont(0.5) WITHIN GROUP (ORDER BY j.hours) AS numeric), 1) AS median_hours_from_order_creation  100.0 * count(j.order_id) / NULLIF(t.total_orders, 0) AS pct_of_total_orders,  {{percentile_cont(0.5, j.hours)}} AS median_hours_from_order_creationFROM statuses sCROSS JOIN total tLEFT JOIN joined j ON j.status = s.statusGROUP BY s.status, t.total_ordersORDER BY pct_of_total_orders DESC, s.status ASCORDER BY pct_of_total_orders DESC NULLS FIRST, s.status ASC NULLS LAST
    WITH target_tenants AS (
      SELECT id FROM tenants
      WHERE name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')
    ),
    base_orders AS (
      SELECT o.id AS id, o.created_at AS created_at
      FROM orders o
      JOIN target_tenants t ON t.id = o.tenant_id
      WHERE o.created_at >= TIMESTAMP '2025-10-01' AND o.created_at < TIMESTAMP '2026-01-01'
    ),
    total AS (
      SELECT count(*) AS total_orders FROM base_orders
    ),
    statuses AS (
      SELECT status FROM (VALUES ('pending'), ('confirmed'), ('paid'), ('partially_fulfilled'), ('fulfilled'), ('cancelled'), ('refunded')) AS s(status)
    ),
    first_reach AS (
      SELECT bo.id AS order_id, osh.to_status AS status, min(osh.created_at) AS first_at
      FROM base_orders bo
      JOIN order_status_history osh ON osh.order_id = bo.id
      WHERE osh.to_status IN ('pending','confirmed','paid','partially_fulfilled','fulfilled','cancelled','refunded')
      GROUP BY bo.id, osh.to_status
    ),
    joined AS (
      SELECT fr.order_id AS order_id, fr.status AS status, fr.first_at AS first_at, bo.created_at AS created_at,
             CAST(EXTRACT(EPOCH FROM (fr.first_at - bo.created_at)) AS NUMERIC(18,6)) / 3600.0 AS hours
      FROM first_reach fr
      JOIN base_orders bo ON bo.id = fr.order_id
    )
    SELECT
      s.status AS status,
      count(j.order_id) AS orders_reaching_status,
      100.0 * count(j.order_id) / NULLIF(t.total_orders, 0) AS pct_of_total_orders,
      PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY j.hours) AS median_hours_from_order_creation
    FROM statuses s
    CROSS JOIN total t
    LEFT JOIN joined j ON j.status = s.status
    GROUP BY s.status, t.total_orders
    ORDER BY pct_of_total_orders DESC NULLS FIRST, s.status ASC NULLS LAST
    100 ms1 s10 sHyper243 msDuckDB648 msGizmoSQL618 msclickhouse_brew1.99 sFirebolt380 msDataFusion460 msClickHouse703 ms
  8. Everything, as recorded

    The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted

    Omar's whole session, 7 turns
    • The request, as Omar received it
      Report request from Emeka Obi (analyst for the VP Product):
      
      Title: Where orders stall after checkout, Q4 2025 — retail merchants only (corrected)
      Question: For the real one-off retail merchants (excluding subscription-box sellers), is the drop-off between order creation and payment a single cliff or a slow bleed, and at which status does it happen?
      Grain: one row per order status
      Columns:
      - status
      - orders reaching this status (count)
      - pct of total orders in window
      - median hours from order creation to reaching this status
      Details: This repeats the night-2 'status-transition funnel' request but with the explicit tenant list I should have given the first time (the earlier run substituted a top-order-count proxy that was dominated by automated subscription merchants and isn't usable). Restrict to tenant_id for exactly: Kibblewick, Studio Vrij, Petalcourt Beauty, Marrowby & Vane, Cragsley Outfitters, Ampfeld Elektronik (look up by name in tenants). Base population: orders with created_at in 2025-10-01 to 2025-12-31 for these tenants only. total_orders = count of distinct such orders. For each status in (pending, confirmed, paid, partially_fulfilled, fulfilled, cancelled, refunded): orders_reaching_status = count of distinct orders with at least one order_status_history row where to_status = that status; pct_of_total_orders = that count / total_orders * 100, rounded 1 decimal; median_hours_from_order_creation = median of (first order_status_history.created_at with that to_status) minus orders.created_at, in hours, rounded 1 decimal (NULL if no order ever reached it). Include all 7 statuses even if zero. Order rows by pct_of_total_orders descending, tie-break by status name.
      
      Check your query with the warehouse's dry_run tool, then hand it in with submit.
    • Omar calls dry_run
      WITH target_tenants AS (
        SELECT id FROM tenants
        WHERE name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')
      ),
      base_orders AS (
        SELECT o.id, o.created_at
        FROM orders o
        JOIN target_tenants t ON t.id = o.tenant_id
        WHERE o.created_at >= TIMESTAMP '2025-10-01' AND o.created_at < TIMESTAMP '2026-01-01'
      ),
      total AS (
        SELECT count(*) AS total_orders FROM base_orders
      ),
      statuses AS (
        SELECT unnest(ARRAY['pending','confirmed','paid','partially_fulfilled','fulfilled','cancelled','refunded']) AS status
      ),
      first_reach AS (
        SELECT bo.id AS order_id, osh.to_status AS status, min(osh.created_at) AS first_at
        FROM base_orders bo
        JOIN order_status_history osh ON osh.order_id = bo.id
        WHERE osh.to_status IN ('pending','confirmed','paid','partially_fulfilled','fulfilled','cancelled','refunded')
        GROUP BY bo.id, osh.to_status
      ),
      joined AS (
        SELECT fr.order_id, fr.status, fr.first_at, bo.created_at,
               EXTRACT(EPOCH FROM (fr.first_at - bo.created_at)) / 3600.0 AS hours
        FROM first_reach fr
        JOIN base_orders bo ON bo.id = fr.order_id
      )
      SELECT
        s.status,
        count(j.order_id) AS orders_reaching_status,
        round(100.0 * count(j.order_id) / NULLIF(t.total_orders, 0), 1) AS pct_of_total_orders,
        round(CAST(percentile_cont(0.5) WITHIN GROUP (ORDER BY j.hours) AS numeric), 1) AS median_hours_from_order_creation
      FROM statuses s
      CROSS JOIN total t
      LEFT JOIN joined j ON j.status = s.status
      GROUP BY s.status, t.total_orders
      ORDER BY pct_of_total_orders DESC, s.status ASC
    • The warehouse answers
      STATUS: COMPILES (dry run on the empty schema, no data read)
      COLUMNS: status, orders_reaching_status, pct_of_total_orders, median_hours_from_order_creation
    • OmarQuery compiles and returns the expected columns. Submitting now.
    • Omar calls submit
      WITH target_tenants AS (
        SELECT id FROM tenants
        WHERE name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')
      ),
      base_orders AS (
        SELECT o.id, o.created_at
        FROM orders o
        JOIN target_tenants t ON t.id = o.tenant_id
        WHERE o.created_at >= TIMESTAMP '2025-10-01' AND o.created_at < TIMESTAMP '2026-01-01'
      ),
      total AS (
        SELECT count(*) AS total_orders FROM base_orders
      ),
      statuses AS (
        SELECT unnest(ARRAY['pending','confirmed','paid','partially_fulfilled','fulfilled','cancelled','refunded']) AS status
      ),
      first_reach AS (
        SELECT bo.id AS order_id, osh.to_status AS status, min(osh.created_at) AS first_at
        FROM base_orders bo
        JOIN order_status_history osh ON osh.order_id = bo.id
        WHERE osh.to_status IN ('pending','confirmed','paid','partially_fulfilled','fulfilled','cancelled','refunded')
        GROUP BY bo.id, osh.to_status
      ),
      joined AS (
        SELECT fr.order_id, fr.status, fr.first_at, bo.created_at,
               EXTRACT(EPOCH FROM (fr.first_at - bo.created_at)) / 3600.0 AS hours
        FROM first_reach fr
        JOIN base_orders bo ON bo.id = fr.order_id
      )
      SELECT
        s.status,
        count(j.order_id) AS orders_reaching_status,
        round(100.0 * count(j.order_id) / NULLIF(t.total_orders, 0), 1) AS pct_of_total_orders,
        round(CAST(percentile_cont(0.5) WITHIN GROUP (ORDER BY j.hours) AS numeric), 1) AS median_hours_from_order_creation
      FROM statuses s
      CROSS JOIN total t
      LEFT JOIN joined j ON j.status = s.status
      GROUP BY s.status, t.total_orders
      ORDER BY pct_of_total_orders DESC, s.status ASC
    • The warehouse answers
      Submitted to tonight's batch. Your work on this request is done.
    • OmarSubmitted. The report gives one row per order status (all 7 included) for the six named retail tenants, with orders reaching each status, percent of total orders in the Q4 2025 window, and median hours from order creation to first reaching that status — ordered by percent descending so the cliff/bleed pattern is immediately visible.