DB Olympics

Queries · q293 · VP Product · week 2, night 3

Email nudge effect on stalled carts, controlling for guest vs. logged-in customers

“Once we separate carts that could never receive an email (guests) from logged-in customers who simply weren't emailed, does receiving an email still look like it drives recovery?”

Manager · asked the questionJonas FeldVP Product. Joined from a mobile-first retailer. Deciding the 2026 roadmap.
Analyst · planned the reportEmeka ObiCustomer-centric: follows cohorts of shoppers or merchants over time.
Engineer · wrote the SQLOmarPragmatic: the shortest correct query, aggregates first and joins dimensions afterwards, FILTER clauses for conditional counts.

Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told

  1. A business event lands on the VP Product's desk

    Week 1

    We are redesigning the shopper journey for 2026. We need to understand the whole path, from the first visit to checkout, delivery, returns and the next purchase, and where shoppers drop out and why.

  2. Jonas Feld writes to Emeka Obi

    The manager, at the start of week 2

    Subject: Good start — let's dig into the cart leak

    Emeka, this is exactly the kind of clarity I needed. The cart-stage bleed is the headline, and I agree we should lead with it — but a two-thirds drop-off that lingers for days, converting at under 20%, isn't something we can act on until we know why people stall. "Add cart recovery emails" is a guess until we understand the shape of the hesitation. That's my main ask for this week.

    I also want to flag one thing before we build a roadmap slide on repeat purchase and returns: the return-vs-repeat gap and the delivery/returns pattern are both flagged as not-yet-causal, confounded by vertical. I don't want to present an 8-point "returns hurt loyalty" number to the exec team if it's really just "apparel behaves differently." Let's firm that up before it goes in a deck, but it's not urgent this week.

    Questions for this week:

    1. What does a stalled cart actually contain, and does it differ from a fast-converting one? I'm thinking about basket size, price point, discount/coupon presence — anything that hints at why someone hesitates for hours or days instead of minutes.
    2. Is the cart bleed getting better, worse, or flat over the past year? If it's been stable at ~70% for two years, that's a structural product problem; if it's worsening recently, something changed (a checkout step, a fee, a release) that we should find and reverse.
    3. Of the carts that eventually do convert after sitting a while, is there a nudge already happening (a reminder, a return visit, a discount) that we could learn from and do more deliberately? I want to know if recovery is already happening organically before we invest in prompting it.
    4. On the subscription merchants' tracking gap — is this just a blind spot in how we instrument recurring orders, or is it telling us something about how those shoppers actually behave (e.g., they never browse again after the first signup)? A quick gut-check is enough this week; no need for a deep investigation yet.

    Keep the returns/delivery vertical-control work and the causal retention question on the list for whenever a slot frees up, but the cart story is what I want to walk into the roadmap review with real conviction on.

    Thanks, Emeka — strong week.

    Jonas

  3. Emeka Obi plans the night

    The analyst · week 2, night 3, request 1 of 3

    Reading last night's reports:

    Report 1 (Cragsley weekly conversion around the Nov 24 outage) undercuts the simple story: conversion didn't dip during the broken-reminder week — it hit 43.5%, the highest week in the whole window, up from a 29-41% range beforehand. The obvious explanation is that Nov 24-30, 2025 is Black Friday/Cyber Monday week — a massive demand spike that would swamp any negative effect from broken reminder emails. So this natural experiment is confounded by timing and can't cleanly tell us whether the reminders matter; I don't think it's worth forcing further at the weekly grain, but a daily cut around the exact outage start (before Black Friday really kicks in on Nov 28) might still show something, so I'll try that once.

    Report 2 (price tier vs. conversion) reinforces the Q1 story on a different cut: higher-priced carts (which are also higher-item-count: 1.3 items at <€50 vs 3.2 items at €200+) convert better and faster (51.4% / 44.1% within 1hr for €200+ vs 23.3% / 16.1% for <€50) — consistent with last night's finding that small, likely single-item, lower-value carts are where hesitation concentrates, not pricier "considered" purchases. Two independent cuts now agree, so I consider Q1 well-answered. Before I put "small single-item carts stall" in the memo as a general pattern, I want one quick robustness check that it's not just one merchant (e.g. Cragsley, which dominates volume) driving the aggregate.

    Report 3 (email nudge) shows a striking gap — 56.0% eventual conversion for carts whose user got an email vs. 8.1% for no-email — but it's likely confounded: 7,540 of the 12,280 "no_email" carts are guest carts that structurally can never receive an email (no user_id to target), and guests plausibly convert worse regardless of email. I want to split "no_email" into guest vs. logged-in before trusting this as evidence the nudge itself works, since the fair comparison is received_email vs. logged-in-but-not-emailed.

    Tonight's 3 requests: (1) the cleaner 3-way email-nudge split, (2) a daily drill-down at Cragsley right at the outage start to see if Black Friday is really masking a real dip, (3) a per-tenant robustness check that the single-item-cart-stalls pattern isn't just one merchant's effect.

    Email nudge effect on stalled carts, controlling for guest vs. logged-in customers
    Question
    Once we separate carts that could never receive an email (guests) from logged-in customers who simply weren't emailed, does receiving an email still look like it drives recovery?
    One row per
    group: received an email (logged-in) / logged-in but no email / guest (no user to email)
    Columns
    • group
    • cart count
    • pct eventually converted
    • median days to convert (converted carts only)
  4. Omar writes the SQL

    The engineer · 1 dry run on the empty schema, then the submission

    Dry run 1 · STATUS: COMPILES (dry run on the empty schema, no data read)
    WITH base_carts AS (
      SELECT c.tenant_id, c.user_id, c.status, c.created_at, c.converted_at,
        EXISTS (
          SELECT 1 FROM email_log e
          WHERE e.tenant_id = c.tenant_id
            AND e.user_id = c.user_id
            AND e.sent_at >= c.created_at + INTERVAL '1 hour'
            AND e.sent_at <= c.created_at + INTERVAL '48 hour'
        ) AS has_email
      FROM carts c
      JOIN tenants t ON t.id = c.tenant_id
      WHERE t.name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')
        AND c.created_at >= DATE '2025-01-01'
        AND c.created_at < DATE '2025-12-01'
        AND NOT (c.status = 'converted' AND c.converted_at IS NOT NULL AND c.converted_at <= c.created_at + INTERVAL '1 hour')
    ),
    grouped AS (
      SELECT
        CASE
          WHEN user_id IS NULL THEN 'guest'
          WHEN has_email THEN 'received_email'
          ELSE 'logged_in_no_email'
        END AS group_name,
        status,
        converted_at,
        created_at
      FROM base_carts
    )
    SELECT
      group_name AS "group",
      COUNT(*) AS cart_count,
      ROUND(100.0 * COUNT(*) FILTER (WHERE status = 'converted') / COUNT(*), 1) AS pct_eventually_converted,
      ROUND(
        PERCENTILE_CONT(0.5) WITHIN GROUP (
          ORDER BY EXTRACT(EPOCH FROM (converted_at - created_at)) / 86400.0
        ) FILTER (WHERE status = 'converted')
      , 1) AS median_days_to_convert
    FROM grouped
    GROUP BY group_name
    ORDER BY CASE group_name WHEN 'received_email' THEN 1 WHEN 'logged_in_no_email' THEN 2 WHEN 'guest' THEN 3 END

    “The query compiles correctly with all expected columns. Submitting now.”

    Submitted · the SQL of the last dry run, unchanged · Submitted to tonight's batch. Your work on this request is done.

    “Submitted. The report groups the 6-retailer stalled-cart population (2025-01-01 to 2025-11-30, excluding carts converted within the first hour) into received_email / logged_in_no_email / guest, using the same "eventually converted" and "median days to convert" definitions as the prior report, so the VP Product can see whether the email effect holds once guests are separated out from un-emailed logged-in users.”

  5. The report comes back overnight

    As Emeka Obi saw it, from the small dataset the agents worked on

    group              | cart_count | pct_eventually_converted | median_days_to_convert
    -------------------+------------+--------------------------+-----------------------
    received_email     |        587 |                     56.0 |                    1.4
    logged_in_no_email |       4740 |                     20.9 |                    4.8
    guest              |       7540 |                      0.0 |                   NULL
    (3 rows)
  6. What came of it

    The memo and the queries that followed; links found by title, so inferred

    Later queries that build on this report:

    It builds on:

  7. It becomes benchmark query q293

    Changed only where the benchmark needs the same result on every system

    • numericsdocs/NUMERICS.md: rounding removed; results keep full precision
    • numericsdocs/NUMERICS.md: EXTRACT(EPOCH FROM (b - a)) cast to NUMERIC(18,6)
    • made explicita selected t.col gets AS col, its standard name made explicit (5×)
    • placeholderpercentile cont
    • placeholderseconds between
    • determinismthe standard leaves the place of NULLs in a sort to the implementation: NULLS LAST ascending, NULLS FIRST descending, as in the reference results
    The benchmark text, against the submitted SQL
    WITH base_carts AS (  SELECT c.tenant_id, c.user_id, c.status, c.created_at, c.converted_at,  SELECT c.tenant_id AS tenant_id, c.user_id AS user_id, c.status AS status, c.created_at AS created_at, c.converted_at AS converted_at,    EXISTS (      SELECT 1 FROM email_log e      WHERE e.tenant_id = c.tenant_id        AND e.user_id = c.user_id        AND e.sent_at >= c.created_at + INTERVAL '1 hour'        AND e.sent_at <= c.created_at + INTERVAL '48 hour'    ) AS has_email  FROM carts c  JOIN tenants t ON t.id = c.tenant_id  WHERE t.name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')    AND c.created_at >= DATE '2025-01-01'    AND c.created_at < DATE '2025-12-01'    AND NOT (c.status = 'converted' AND c.converted_at IS NOT NULL AND c.converted_at <= c.created_at + INTERVAL '1 hour')),grouped AS (  SELECT    CASE      WHEN user_id IS NULL THEN 'guest'      WHEN has_email THEN 'received_email'      ELSE 'logged_in_no_email'    END AS group_name,    status,    converted_at,    created_at  FROM base_carts)SELECT  group_name AS "group",  COUNT(*) AS cart_count,  ROUND(100.0 * COUNT(*) FILTER (WHERE status = 'converted') / COUNT(*), 1) AS pct_eventually_converted,  ROUND(    PERCENTILE_CONT(0.5) WITHIN GROUP (      ORDER BY EXTRACT(EPOCH FROM (converted_at - created_at)) / 86400.0    ) FILTER (WHERE status = 'converted')  , 1) AS median_days_to_convert  100.0 * COUNT(*) FILTER (WHERE status = 'converted') / COUNT(*) AS pct_eventually_converted,  {{percentile_cont_filter(0.5, {{seconds_between(created_at, converted_at)}} / 86400.0, status = 'converted')}} AS median_days_to_convertFROM groupedGROUP BY group_nameORDER BY CASE group_name WHEN 'received_email' THEN 1 WHEN 'logged_in_no_email' THEN 2 WHEN 'guest' THEN 3 ENDORDER BY CASE group_name WHEN 'received_email' THEN 1 WHEN 'logged_in_no_email' THEN 2 WHEN 'guest' THEN 3 END NULLS LAST
    WITH base_carts AS (
      SELECT c.tenant_id AS tenant_id, c.user_id AS user_id, c.status AS status, c.created_at AS created_at, c.converted_at AS converted_at,
        EXISTS (
          SELECT 1 FROM email_log e
          WHERE e.tenant_id = c.tenant_id
            AND e.user_id = c.user_id
            AND e.sent_at >= c.created_at + INTERVAL '1 hour'
            AND e.sent_at <= c.created_at + INTERVAL '48 hour'
        ) AS has_email
      FROM carts c
      JOIN tenants t ON t.id = c.tenant_id
      WHERE t.name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')
        AND c.created_at >= DATE '2025-01-01'
        AND c.created_at < DATE '2025-12-01'
        AND NOT (c.status = 'converted' AND c.converted_at IS NOT NULL AND c.converted_at <= c.created_at + INTERVAL '1 hour')
    ),
    grouped AS (
      SELECT
        CASE
          WHEN user_id IS NULL THEN 'guest'
          WHEN has_email THEN 'received_email'
          ELSE 'logged_in_no_email'
        END AS group_name,
        status,
        converted_at,
        created_at
      FROM base_carts
    )
    SELECT
      group_name AS "group",
      COUNT(*) AS cart_count,
      100.0 * COUNT(*) FILTER (WHERE status = 'converted') / COUNT(*) AS pct_eventually_converted,
      PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY CAST(EXTRACT(EPOCH FROM (converted_at - created_at)) AS NUMERIC(18,6)) / 86400.0) FILTER (WHERE status = 'converted') AS median_days_to_convert
    FROM grouped
    GROUP BY group_name
    ORDER BY CASE group_name WHEN 'received_email' THEN 1 WHEN 'logged_in_no_email' THEN 2 WHEN 'guest' THEN 3 END NULLS LAST
    100 ms1 s10 sHyper725 msDuckDB271 msGizmoSQL270 msclickhouse_brew5.26 sFirebolt1.62 sDataFusionerrorClickHouseerror
    • DataFusionThis feature is not implemented: Physical plan does not support logical expression Exists(Exists { subquery: <subquery>, negated: false })
    • ClickHouseCode: 48. DB::Exception: Current query is not supported yet, because can't find correlated column '__table6.tenant_id' in current header: __table6.user_id, __table6.status, __table6.created_at, __table6.converted_at. (NOT_IMPLEMENTED)
  8. Everything, as recorded

    The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted

    Omar's whole session, 7 turns
    • The request, as Omar received it
      Report request from Emeka Obi (analyst for the VP Product):
      
      Title: Email nudge effect on stalled carts, controlling for guest vs. logged-in customers
      Question: Once we separate carts that could never receive an email (guests) from logged-in customers who simply weren't emailed, does receiving an email still look like it drives recovery?
      Grain: one row per group: received an email (logged-in) / logged-in but no email / guest (no user to email)
      Columns:
      - group
      - cart count
      - pct eventually converted
      - median days to convert (converted carts only)
      Details: Same population and definitions as the prior 'stalled carts that receive an email' report: the 6 retail tenants (Kibblewick, Studio Vrij, Petalcourt Beauty, Marrowby & Vane, Cragsley Outfitters, Ampfeld Elektronik), carts with created_at in 2025-01-01 to 2025-11-30, restricted to carts not already converted within the first hour of creation. 'received_email' = non-null user_id with at least one email_log row (same tenant_id and user_id) with sent_at between created_at+1h and created_at+48h. 'logged_in_no_email' = non-null user_id with no such email_log row. 'guest' = null user_id (can never be matched to an email). pct_eventually_converted and median_days_to_convert use the same definitions as before. Order rows: received_email, logged_in_no_email, guest.
      
      Check your query with the warehouse's dry_run tool, then hand it in with submit.
    • Omar calls dry_run
      WITH base_carts AS (
        SELECT c.tenant_id, c.user_id, c.status, c.created_at, c.converted_at,
          EXISTS (
            SELECT 1 FROM email_log e
            WHERE e.tenant_id = c.tenant_id
              AND e.user_id = c.user_id
              AND e.sent_at >= c.created_at + INTERVAL '1 hour'
              AND e.sent_at <= c.created_at + INTERVAL '48 hour'
          ) AS has_email
        FROM carts c
        JOIN tenants t ON t.id = c.tenant_id
        WHERE t.name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')
          AND c.created_at >= DATE '2025-01-01'
          AND c.created_at < DATE '2025-12-01'
          AND NOT (c.status = 'converted' AND c.converted_at IS NOT NULL AND c.converted_at <= c.created_at + INTERVAL '1 hour')
      ),
      grouped AS (
        SELECT
          CASE
            WHEN user_id IS NULL THEN 'guest'
            WHEN has_email THEN 'received_email'
            ELSE 'logged_in_no_email'
          END AS group_name,
          status,
          converted_at,
          created_at
        FROM base_carts
      )
      SELECT
        group_name AS "group",
        COUNT(*) AS cart_count,
        ROUND(100.0 * COUNT(*) FILTER (WHERE status = 'converted') / COUNT(*), 1) AS pct_eventually_converted,
        ROUND(
          PERCENTILE_CONT(0.5) WITHIN GROUP (
            ORDER BY EXTRACT(EPOCH FROM (converted_at - created_at)) / 86400.0
          ) FILTER (WHERE status = 'converted')
        , 1) AS median_days_to_convert
      FROM grouped
      GROUP BY group_name
      ORDER BY CASE group_name WHEN 'received_email' THEN 1 WHEN 'logged_in_no_email' THEN 2 WHEN 'guest' THEN 3 END
    • The warehouse answers
      STATUS: COMPILES (dry run on the empty schema, no data read)
      COLUMNS: group, cart_count, pct_eventually_converted, median_days_to_convert
    • OmarThe query compiles correctly with all expected columns. Submitting now.
    • Omar calls submit
      WITH base_carts AS (
        SELECT c.tenant_id, c.user_id, c.status, c.created_at, c.converted_at,
          EXISTS (
            SELECT 1 FROM email_log e
            WHERE e.tenant_id = c.tenant_id
              AND e.user_id = c.user_id
              AND e.sent_at >= c.created_at + INTERVAL '1 hour'
              AND e.sent_at <= c.created_at + INTERVAL '48 hour'
          ) AS has_email
        FROM carts c
        JOIN tenants t ON t.id = c.tenant_id
        WHERE t.name IN ('Kibblewick','Studio Vrij','Petalcourt Beauty','Marrowby & Vane','Cragsley Outfitters','Ampfeld Elektronik')
          AND c.created_at >= DATE '2025-01-01'
          AND c.created_at < DATE '2025-12-01'
          AND NOT (c.status = 'converted' AND c.converted_at IS NOT NULL AND c.converted_at <= c.created_at + INTERVAL '1 hour')
      ),
      grouped AS (
        SELECT
          CASE
            WHEN user_id IS NULL THEN 'guest'
            WHEN has_email THEN 'received_email'
            ELSE 'logged_in_no_email'
          END AS group_name,
          status,
          converted_at,
          created_at
        FROM base_carts
      )
      SELECT
        group_name AS "group",
        COUNT(*) AS cart_count,
        ROUND(100.0 * COUNT(*) FILTER (WHERE status = 'converted') / COUNT(*), 1) AS pct_eventually_converted,
        ROUND(
          PERCENTILE_CONT(0.5) WITHIN GROUP (
            ORDER BY EXTRACT(EPOCH FROM (converted_at - created_at)) / 86400.0
          ) FILTER (WHERE status = 'converted')
        , 1) AS median_days_to_convert
      FROM grouped
      GROUP BY group_name
      ORDER BY CASE group_name WHEN 'received_email' THEN 1 WHEN 'logged_in_no_email' THEN 2 WHEN 'guest' THEN 3 END
    • The warehouse answers
      Submitted to tonight's batch. Your work on this request is done.
    • OmarSubmitted. The report groups the 6-retailer stalled-cart population (2025-01-01 to 2025-11-30, excluding carts converted within the first hour) into received_email / logged_in_no_email / guest, using the same "eventually converted" and "median days to convert" definitions as the prior report, so the VP Product can see whether the email effect holds once guests are separated out from un-emailed logged-in users.