Queries · q167 · Head of Lifecycle Marketing · week 1, night 3
“What share of our active customer base is enrolled in loyalty or has ever used a referral code, and is that share growing, flat, or shrinking over time?”
Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told
Week 1; shared with Tobias Lindqvist, Chief Financial Officer
Is the loyalty and referral programme worth it? Do members and referred customers behave differently from others, and what does the programme cost?
The manager, at the start of week 1
Subject: Loyalty & referral programme — what I need for Friday's memo
Hi Ingrid,
Welcome aboard — thank you for jumping on this in week one. Tobias and I need to give the leadership team a joint, evidence-based view on whether the loyalty and referral programme is paying for itself. The deliverable is your Friday memo, which Tobias and I will use to decide whether we expand the programme, leave it as is, or rework it. Given the slot budget, let's prioritize breadth of the right questions over depth on any one — we can always follow up next week.
Here's what I'm trying to answer, in my own words:
Given we only have a few report slots a night, I'd rank these in order 3, 1, 2 as the must-haves for Friday; 4 and 5 are nice-to-haves if slots allow. Please loop in whatever Tobias's analyst is already pulling on the finance side so we're not duplicating report requests.
Thanks, Ingrid — happy to talk through any of this before you start drafting requests.
Best, Aisha
Subject: Finance angle on the loyalty/referral review
Hi Ingrid,
A few things I'd like us to cover as you scope this:
Happy to look at a first cut together before it goes wider.
Tobias
The analyst · week 1, night 3, request 2 of 3
Night 2 sharpened the picture. The tenure-controlled cut (Report 1) rules out the 'members are just newer accounts' explanation: at every tenure band members order less, spend less, and are refunded 2x more than non-members - and the refund gap actually widens with tenure (27% to 34% for members vs 20% down to 14% for non-members). So this isn't a cohort artifact; it's a persistent, worsening pattern. The before/after enrollment report (Report 2) came back with zero pre-enrollment orders for everyone - meaning enrollment essentially happens at or before a member's very first order (likely at signup), so there is no 'pre' baseline to diff against; that test was a dead end for isolating causality but confirms enrollment is immediate, not something people do after building a track record. Report 3 shows no smoking-gun concentration: redemption counts per referrer are almost all 1 (occasional 2), spread across dozens of tenants and referrers with normal-looking order histories (from near-zero to tens of thousands of EUR) - this looks like genuine light word-of-mouth, not gaming, though volume is tiny. For the last full night before the memo, I want to: (1) directly test Aisha's 'rewards redeemed without real repeat purchase' hypothesis by checking whether orders where points were actually redeemed show elevated refund rates versus other member orders - this could explain the mechanism behind the refund gap; (2) quantify participation and its trend for Aisha's #4; (3) close out Tobias's B2B payment-terms question, which is still untouched. I'm leaving one night in reserve for any necessary follow-up before Friday.
The engineer · 1 dry run on the empty schema, then the submission
SELECT q.quarter AS quarter, (SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end) AS active_customers_cumulative, (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) AS loyalty_members_cumulative, ROUND(100.0 * (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) , 2) AS pct_customers_loyalty_members, (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) AS referred_customers_cumulative, ROUND(100.0 * (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) , 2) AS pct_customers_referred FROM ( SELECT year_num, quarter_num, CONCAT(CAST(year_num AS VARCHAR), '-Q', CAST(quarter_num AS VARCHAR)) AS quarter, MAX(calendar_date) AS quarter_end FROM calendar_days WHERE year_num BETWEEN 2023 AND 2025 GROUP BY year_num, quarter_num ) q ORDER BY q.year_num ASC, q.quarter_num ASC
“Compiles cleanly with the expected columns. Submitting.”
“Submitted. The report computes, for each of the 12 quarter-ends from 2023-Q1 through 2025-Q4, the cumulative distinct non-staff customers with an order, cumulative loyalty enrollees, and cumulative referred customers, plus their shares of the active base — aggregated platform-wide across all tenants since the question wasn't merchant-specific.”
As Ingrid Solberg saw it, from the small dataset the agents worked on
quarter | active_customers_cumulative | loyalty_members_cumulative | pct_customers_loyalty_members | referred_customers_cumulative | pct_customers_referred --------+-----------------------------+----------------------------+-------------------------------+-------------------------------+----------------------- 2023-Q1 | 1047 | 279 | 26.65 | 0 | 0.0 2023-Q2 | 1795 | 358 | 19.94 | 2 | 0.11 2023-Q3 | 2421 | 442 | 18.26 | 7 | 0.29 2023-Q4 | 3401 | 559 | 16.44 | 10 | 0.29 2024-Q1 | 4004 | 636 | 15.88 | 16 | 0.4 2024-Q2 | 4654 | 727 | 15.62 | 20 | 0.43 2024-Q3 | 5269 | 813 | 15.43 | 25 | 0.47 2024-Q4 | 6314 | 975 | 15.44 | 38 | 0.6 2025-Q1 | 6963 | 1092 | 15.68 | 43 | 0.62 2025-Q2 | 7632 | 1217 | 15.95 | 50 | 0.66 2025-Q3 | 8377 | 1344 | 16.04 | 57 | 0.68 2025-Q4 | 9640 | 1546 | 16.04 | 67 | 0.7 (12 rows)
The memo and the queries that followed; links found by title, so inferred
Later queries that build on this report:
It builds on:
Changed only where the benchmark needs the same result on every system
SELECT q.quarter AS quarter, (SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end) AS active_customers_cumulative, (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) AS loyalty_members_cumulative, ROUND(100.0 * (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) , 2) AS pct_customers_loyalty_members, 100.0 *(SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_idWHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end)/ NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) AS pct_customers_loyalty_members, (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) AS referred_customers_cumulative, ROUND(100.0 * (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) , 2) AS pct_customers_referred 100.0 *(SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_idWHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end)/ NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) AS pct_customers_referredFROM ( SELECT year_num, quarter_num, CONCAT(CAST(year_num AS VARCHAR), '-Q', CAST(quarter_num AS VARCHAR)) AS quarter, MAX(calendar_date) AS quarter_end FROM calendar_days WHERE year_num BETWEEN 2023 AND 2025 GROUP BY year_num, quarter_num) qORDER BY q.year_num ASC, q.quarter_num ASCORDER BY q.year_num ASC NULLS LAST, q.quarter_num ASC NULLS LAST
SELECT q.quarter AS quarter, (SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end) AS active_customers_cumulative, (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) AS loyalty_members_cumulative, 100.0 * (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) AS pct_customers_loyalty_members, (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) AS referred_customers_cumulative, 100.0 * (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) AS pct_customers_referred FROM ( SELECT year_num, quarter_num, CONCAT(CAST(year_num AS VARCHAR), '-Q', CAST(quarter_num AS VARCHAR)) AS quarter, MAX(calendar_date) AS quarter_end FROM calendar_days WHERE year_num BETWEEN 2023 AND 2025 GROUP BY year_num, quarter_num ) q ORDER BY q.year_num ASC NULLS LAST, q.quarter_num ASC NULLS LAST
The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted
Report request from Ingrid Solberg (analyst for the Head of Lifecycle Marketing): Title: Loyalty and referral programme participation rate over time Question: What share of our active customer base is enrolled in loyalty or has ever used a referral code, and is that share growing, flat, or shrinking over time? Grain: one row per calendar quarter from 2023-Q1 through 2025-Q4 Columns: - quarter - active_customers_cumulative - loyalty_members_cumulative - pct_customers_loyalty_members - referred_customers_cumulative - pct_customers_referred Details: For each quarter end date (2023-03-31, 2023-06-30, ... 2025-12-31), compute cumulative counts as of that date: active_customers_cumulative = distinct users (is_staff = false) with at least one order placed_at on or before the quarter end; loyalty_members_cumulative = distinct users with a loyalty_accounts.enrolled_at on or before the quarter end; pct_customers_loyalty_members = loyalty_members_cumulative / active_customers_cumulative as a percentage; referred_customers_cumulative = distinct referred_user_id in referral_redemptions with redeemed_at on or before the quarter end; pct_customers_referred = referred_customers_cumulative / active_customers_cumulative as a percentage. Use calendar_days to get correct quarter-end dates. Order by quarter ascending. Check your query with the warehouse's dry_run tool, then hand it in with submit.
dry_runSELECT q.quarter AS quarter, (SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end) AS active_customers_cumulative, (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) AS loyalty_members_cumulative, ROUND(100.0 * (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) , 2) AS pct_customers_loyalty_members, (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) AS referred_customers_cumulative, ROUND(100.0 * (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) , 2) AS pct_customers_referred FROM ( SELECT year_num, quarter_num, CONCAT(CAST(year_num AS VARCHAR), '-Q', CAST(quarter_num AS VARCHAR)) AS quarter, MAX(calendar_date) AS quarter_end FROM calendar_days WHERE year_num BETWEEN 2023 AND 2025 GROUP BY year_num, quarter_num ) q ORDER BY q.year_num ASC, q.quarter_num ASC
STATUS: COMPILES (dry run on the empty schema, no data read) COLUMNS: quarter, active_customers_cumulative, loyalty_members_cumulative, pct_customers_loyalty_members, referred_customers_cumulative, pct_customers_referred
submitSELECT q.quarter AS quarter, (SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end) AS active_customers_cumulative, (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) AS loyalty_members_cumulative, ROUND(100.0 * (SELECT COUNT(DISTINCT la.user_id) FROM loyalty_accounts la JOIN users u ON u.id = la.user_id WHERE u.is_staff = false AND CAST(la.enrolled_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) , 2) AS pct_customers_loyalty_members, (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) AS referred_customers_cumulative, ROUND(100.0 * (SELECT COUNT(DISTINCT rr.referred_user_id) FROM referral_redemptions rr JOIN users u ON u.id = rr.referred_user_id WHERE u.is_staff = false AND CAST(rr.redeemed_at AS DATE) <= q.quarter_end) / NULLIF((SELECT COUNT(DISTINCT o.user_id) FROM orders o JOIN users u ON u.id = o.user_id WHERE u.is_staff = false AND o.placed_at IS NOT NULL AND CAST(o.placed_at AS DATE) <= q.quarter_end), 0) , 2) AS pct_customers_referred FROM ( SELECT year_num, quarter_num, CONCAT(CAST(year_num AS VARCHAR), '-Q', CAST(quarter_num AS VARCHAR)) AS quarter, MAX(calendar_date) AS quarter_end FROM calendar_days WHERE year_num BETWEEN 2023 AND 2025 GROUP BY year_num, quarter_num ) q ORDER BY q.year_num ASC, q.quarter_num ASC
Submitted to tonight's batch. Your work on this request is done.