Queries · q161 · Head of Lifecycle Marketing · week 1, night 1
“Do enrolled loyalty members actually order more often, spend more, stick around longer, and return items less than non-member customers, as of today?”
Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told
Week 1; shared with Tobias Lindqvist, Chief Financial Officer
Is the loyalty and referral programme worth it? Do members and referred customers behave differently from others, and what does the programme cost?
The manager, at the start of week 1
Subject: Loyalty & referral programme — what I need for Friday's memo
Hi Ingrid,
Welcome aboard — thank you for jumping on this in week one. Tobias and I need to give the leadership team a joint, evidence-based view on whether the loyalty and referral programme is paying for itself. The deliverable is your Friday memo, which Tobias and I will use to decide whether we expand the programme, leave it as is, or rework it. Given the slot budget, let's prioritize breadth of the right questions over depth on any one — we can always follow up next week.
Here's what I'm trying to answer, in my own words:
Given we only have a few report slots a night, I'd rank these in order 3, 1, 2 as the must-haves for Friday; 4 and 5 are nice-to-haves if slots allow. Please loop in whatever Tobias's analyst is already pulling on the finance side so we're not duplicating report requests.
Thanks, Ingrid — happy to talk through any of this before you start drafting requests.
Best, Aisha
Subject: Finance angle on the loyalty/referral review
Hi Ingrid,
A few things I'd like us to cover as you scope this:
Happy to look at a first cut together before it goes wider.
Tobias
The analyst · week 1, night 1, request 2 of 3
Prioritizing Aisha's must-haves (3,1,2) which also cover most of Tobias's asks (net cost, refund profile). Night 1 establishes three independent baselines: (a) program cost/liability trend, (b) member vs non-member behavior, (c) referred vs non-referred early behavior. I'm folding Tobias's refund-rate question into (b) and (c) as extra columns rather than separate reports, since it's a natural measure at the same grain. Deliberately NOT yet controlling for tenure/cohort matching in (b) - that's the obvious follow-up once I see whether raw numbers show a gap (my risk-focused read: if members already look no better on raw numbers, that itself is a red flag worth flagging Friday). Saving participation-rate trend (#4) and gaming/concentration red flags (#5) for night 2 once I've seen these results and know where to dig. Will flag to Aisha/Tobias to confirm their analyst isn't pulling the same cost report before night 2.
The engineer · 1 dry run on the empty schema, then the submission
WITH orders_base AS ( SELECT o.id AS order_id, o.tenant_id, o.user_id, o.status, o.currency_code, o.grand_total, COALESCE(o.placed_at, o.created_at) AS order_date FROM orders o JOIN users u ON u.id = o.user_id AND u.tenant_id = o.tenant_id WHERE u.is_staff = false AND COALESCE(o.placed_at, o.created_at) >= DATE '2023-01-01' AND COALESCE(o.placed_at, o.created_at) < DATE '2026-01-01' ), orders_enriched AS ( SELECT ob.*, CASE WHEN ob.currency_code = 'EUR' THEN ob.grand_total ELSE ob.grand_total / er.rate END AS eur_amount, EXISTS (SELECT 1 FROM payments p WHERE p.order_id = ob.order_id AND p.amount_refunded > 0) AS has_refund, EXISTS (SELECT 1 FROM order_returns r WHERE r.order_id = ob.order_id) AS has_return FROM orders_base ob LEFT JOIN exchange_rates er ON er.base_code = 'EUR' AND er.quote_code = ob.currency_code AND er.as_of = CAST(ob.order_date AS DATE) ), customer_agg AS ( SELECT user_id, COUNT(*) AS total_orders, COUNT(*) FILTER (WHERE status <> 'cancelled') AS noncancelled_orders, SUM(eur_amount) FILTER (WHERE status <> 'cancelled') AS noncancelled_revenue, MIN(order_date) AS first_order_date, MAX(order_date) AS last_order_date, COUNT(*) FILTER (WHERE has_refund OR has_return) AS refunded_orders FROM orders_enriched GROUP BY user_id ), segmented AS ( SELECT ca.*, CASE WHEN EXISTS (SELECT 1 FROM loyalty_accounts la WHERE la.user_id = ca.user_id) THEN 'loyalty member' ELSE 'non-member' END AS segment FROM customer_agg ca ) SELECT segment, COUNT(*) AS customer_count, ROUND(AVG(total_orders)::numeric, 2) AS avg_lifetime_orders, ROUND((SUM(noncancelled_revenue) / NULLIF(COUNT(*),0))::numeric, 2) AS avg_lifetime_revenue_eur, ROUND((SUM(noncancelled_revenue) / NULLIF(SUM(noncancelled_orders),0))::numeric, 2) AS avg_order_value_eur, ROUND((AVG(EXTRACT(EPOCH FROM (last_order_date - first_order_date)) / 86400.0))::numeric, 1) AS avg_tenure_days, ROUND((100.0 * COUNT(*) FILTER (WHERE total_orders >= 2) / NULLIF(COUNT(*),0))::numeric, 1) AS repeat_purchase_rate_pct, ROUND((100.0 * SUM(refunded_orders) / NULLIF(SUM(total_orders),0))::numeric, 1) AS refund_rate_pct FROM segmented GROUP BY segment ORDER BY CASE WHEN segment = 'loyalty member' THEN 0 ELSE 1 END, segment
“Query compiles cleanly with the expected 8 columns, one row per segment. Submitting.”
“Submitted. Summary of key choices baked into the query, also noted for the analyst:
- Population/date filter: orders where COALESCE(placed_at, created_at) falls in 2023‑01‑01..2025‑12‑31, for non-staff users.
- Segment: loyalty member if the user has any loyalty_accounts row, else non-member.
- Avg lifetime orders / repeat rate / refund rate: computed over all orders regardless of status (matches "any status" population wording).
- Avg lifetime revenue & avg order value: numerator/denominator restricted to non-cancelled orders, per the spec.
- EUR conversion: grand_total / rate where rate is the EUR→order-currency rate as of the order date (1:1 when already EUR).
- Tenure: days between first and last order date per customer (0 for single-order customers), computed via epoch seconds/86400 for dialect-safety.”
As Ingrid Solberg saw it, from the small dataset the agents worked on
segment | customer_count | avg_lifetime_orders | avg_lifetime_revenue_eur | avg_order_value_eur | avg_tenure_days | repeat_purchase_rate_pct | refund_rate_pct ---------------+----------------+---------------------+--------------------------+---------------------+-----------------+--------------------------+---------------- loyalty member | 1330 | 3.72 | 687.99 | 194.40 | 199.7 | 67.7 | 32.4 non-member | 8310 | 5.55 | 1511.84 | 281.57 | 234.1 | 71.7 | 15.5 (2 rows)
The memo and the queries that followed; links found by title, so inferred
Later queries that build on this report:
Changed only where the benchmark needs the same result on every system
WITH orders_base AS ( SELECT o.id AS order_id, o.tenant_id, o.user_id, o.status, o.currency_code, o.grand_total, o.tenant_id AS tenant_id, o.user_id AS user_id, o.status AS status, o.currency_code AS currency_code, o.grand_total AS grand_total, COALESCE(o.placed_at, o.created_at) AS order_date FROM orders o JOIN users u ON u.id = o.user_id AND u.tenant_id = o.tenant_id WHERE u.is_staff = false AND COALESCE(o.placed_at, o.created_at) >= DATE '2023-01-01' AND COALESCE(o.placed_at, o.created_at) < DATE '2026-01-01'),orders_enriched AS ( SELECT ob.*, CASE WHEN ob.currency_code = 'EUR' THEN ob.grand_total ELSE ob.grand_total / er.rate END AS eur_amount, ELSE CAST(ob.grand_total / er.rate AS NUMERIC(18,6)) END AS eur_amount, EXISTS (SELECT 1 FROM payments p WHERE p.order_id = ob.order_id AND p.amount_refunded > 0) AS has_refund, EXISTS (SELECT 1 FROM order_returns r WHERE r.order_id = ob.order_id) AS has_return FROM orders_base ob LEFT JOIN exchange_rates er ON er.base_code = 'EUR' AND er.quote_code = ob.currency_code AND er.as_of = CAST(ob.order_date AS DATE)),customer_agg AS ( SELECT user_id, COUNT(*) AS total_orders, COUNT(*) FILTER (WHERE status <> 'cancelled') AS noncancelled_orders, SUM(eur_amount) FILTER (WHERE status <> 'cancelled') AS noncancelled_revenue, MIN(order_date) AS first_order_date, MAX(order_date) AS last_order_date, COUNT(*) FILTER (WHERE has_refund OR has_return) AS refunded_orders FROM orders_enriched GROUP BY user_id),segmented AS ( SELECT ca.*, CASE WHEN EXISTS (SELECT 1 FROM loyalty_accounts la WHERE la.user_id = ca.user_id) THEN 'loyalty member' ELSE 'non-member' END AS segment FROM customer_agg ca)SELECT segment, COUNT(*) AS customer_count, ROUND(AVG(total_orders)::numeric, 2) AS avg_lifetime_orders, ROUND((SUM(noncancelled_revenue) / NULLIF(COUNT(*),0))::numeric, 2) AS avg_lifetime_revenue_eur, ROUND((SUM(noncancelled_revenue) / NULLIF(SUM(noncancelled_orders),0))::numeric, 2) AS avg_order_value_eur, ROUND((AVG(EXTRACT(EPOCH FROM (last_order_date - first_order_date)) / 86400.0))::numeric, 1) AS avg_tenure_days, ROUND((100.0 * COUNT(*) FILTER (WHERE total_orders >= 2) / NULLIF(COUNT(*),0))::numeric, 1) AS repeat_purchase_rate_pct, ROUND((100.0 * SUM(refunded_orders) / NULLIF(SUM(total_orders),0))::numeric, 1) AS refund_rate_pct AVG(CAST(total_orders AS NUMERIC(18,6))) AS avg_lifetime_orders, (SUM(noncancelled_revenue) / NULLIF(COUNT(*),0)) AS avg_lifetime_revenue_eur, (SUM(noncancelled_revenue) / NULLIF(SUM(noncancelled_orders),0)) AS avg_order_value_eur, (AVG({{seconds_between(first_order_date, last_order_date)}} / 86400.0)) AS avg_tenure_days, (100.0 * COUNT(*) FILTER (WHERE total_orders >= 2) / NULLIF(COUNT(*),0)) AS repeat_purchase_rate_pct, (100.0 * SUM(refunded_orders) / NULLIF(SUM(total_orders),0)) AS refund_rate_pctFROM segmentedGROUP BY segmentORDER BY CASE WHEN segment = 'loyalty member' THEN 0 ELSE 1 END, segmentORDER BY CASE WHEN segment = 'loyalty member' THEN 0 ELSE 1 END NULLS LAST, segment NULLS LAST
WITH orders_base AS ( SELECT o.id AS order_id, o.tenant_id AS tenant_id, o.user_id AS user_id, o.status AS status, o.currency_code AS currency_code, o.grand_total AS grand_total, COALESCE(o.placed_at, o.created_at) AS order_date FROM orders o JOIN users u ON u.id = o.user_id AND u.tenant_id = o.tenant_id WHERE u.is_staff = false AND COALESCE(o.placed_at, o.created_at) >= DATE '2023-01-01' AND COALESCE(o.placed_at, o.created_at) < DATE '2026-01-01' ), orders_enriched AS ( SELECT ob.*, CASE WHEN ob.currency_code = 'EUR' THEN ob.grand_total ELSE CAST(ob.grand_total / er.rate AS NUMERIC(18,6)) END AS eur_amount, EXISTS (SELECT 1 FROM payments p WHERE p.order_id = ob.order_id AND p.amount_refunded > 0) AS has_refund, EXISTS (SELECT 1 FROM order_returns r WHERE r.order_id = ob.order_id) AS has_return FROM orders_base ob LEFT JOIN exchange_rates er ON er.base_code = 'EUR' AND er.quote_code = ob.currency_code AND er.as_of = CAST(ob.order_date AS DATE) ), customer_agg AS ( SELECT user_id, COUNT(*) AS total_orders, COUNT(*) FILTER (WHERE status <> 'cancelled') AS noncancelled_orders, SUM(eur_amount) FILTER (WHERE status <> 'cancelled') AS noncancelled_revenue, MIN(order_date) AS first_order_date, MAX(order_date) AS last_order_date, COUNT(*) FILTER (WHERE has_refund OR has_return) AS refunded_orders FROM orders_enriched GROUP BY user_id ), segmented AS ( SELECT ca.*, CASE WHEN EXISTS (SELECT 1 FROM loyalty_accounts la WHERE la.user_id = ca.user_id) THEN 'loyalty member' ELSE 'non-member' END AS segment FROM customer_agg ca ) SELECT segment, COUNT(*) AS customer_count, AVG(CAST(total_orders AS NUMERIC(18,6))) AS avg_lifetime_orders, (SUM(noncancelled_revenue) / NULLIF(COUNT(*),0)) AS avg_lifetime_revenue_eur, (SUM(noncancelled_revenue) / NULLIF(SUM(noncancelled_orders),0)) AS avg_order_value_eur, (AVG(CAST(EXTRACT(EPOCH FROM (last_order_date - first_order_date)) AS NUMERIC(18,6)) / 86400.0)) AS avg_tenure_days, (100.0 * COUNT(*) FILTER (WHERE total_orders >= 2) / NULLIF(COUNT(*),0)) AS repeat_purchase_rate_pct, (100.0 * SUM(refunded_orders) / NULLIF(SUM(total_orders),0)) AS refund_rate_pct FROM segmented GROUP BY segment ORDER BY CASE WHEN segment = 'loyalty member' THEN 0 ELSE 1 END NULLS LAST, segment NULLS LAST
WITH orders_base AS ( SELECT o.id AS order_id, o.tenant_id AS tenant_id, o.user_id AS user_id, o.status AS status, o.currency_code AS currency_code, o.grand_total AS grand_total, COALESCE(o.placed_at, o.created_at) AS order_date FROM orders o JOIN users u ON u.id = o.user_id AND u.tenant_id = o.tenant_id WHERE u.is_staff = false AND COALESCE(o.placed_at, o.created_at) >= DATE '2023-01-01' AND COALESCE(o.placed_at, o.created_at) < DATE '2026-01-01' ), orders_enriched AS ( SELECT ob.*, CASE WHEN ob.currency_code = 'EUR' THEN ob.grand_total ELSE CAST(ob.grand_total / er.rate AS NUMERIC(18,6)) END AS eur_amount, EXISTS (SELECT 1 FROM payments p WHERE p.order_id = ob.order_id AND p.amount_refunded > 0) AS has_refund, EXISTS (SELECT 1 FROM order_returns r WHERE r.order_id = ob.order_id) AS has_return FROM orders_base ob LEFT JOIN exchange_rates er ON er.base_code = 'EUR' AND er.quote_code = ob.currency_code AND er.as_of = CAST(ob.order_date AS DATE) ), customer_agg AS ( SELECT user_id, COUNT(*) AS total_orders, COUNT(*) FILTER (WHERE status <> 'cancelled') AS noncancelled_orders, SUM(eur_amount) FILTER (WHERE status <> 'cancelled') AS noncancelled_revenue, MIN(order_date) AS first_order_date, MAX(order_date) AS last_order_date, COUNT(*) FILTER (WHERE has_refund OR has_return) AS refunded_orders FROM orders_enriched GROUP BY user_id ), segmented AS ( SELECT ca.*, CASE WHEN EXISTS (SELECT 1 FROM loyalty_accounts la WHERE la.user_id = ca.user_id) THEN 'loyalty member' ELSE 'non-member' END AS segment FROM customer_agg ca ) SELECT segment, COUNT(*) AS customer_count, AVG(CAST(total_orders AS NUMERIC(18,6))) AS avg_lifetime_orders, (SUM(noncancelled_revenue) / NULLIF(COUNT(*),0)) AS avg_lifetime_revenue_eur, (SUM(noncancelled_revenue) / NULLIF(SUM(noncancelled_orders),0)) AS avg_order_value_eur, (AVG(CAST(divideDecimal(toDecimal128(dateDiff('microsecond', first_order_date, last_order_date), 0), toDecimal128(1000000, 0), 6) AS Nullable(Decimal(18, 6))) / 86400.0)) AS avg_tenure_days, (100.0 * COUNT(*) FILTER (WHERE total_orders >= 2) / NULLIF(COUNT(*),0)) AS repeat_purchase_rate_pct, (100.0 * SUM(refunded_orders) / NULLIF(SUM(total_orders),0)) AS refund_rate_pct FROM segmented GROUP BY segment ORDER BY CASE WHEN segment = 'loyalty member' THEN 0 ELSE 1 END NULLS LAST, segment NULLS LAST
The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted
Report request from Ingrid Solberg (analyst for the Head of Lifecycle Marketing): Title: Loyalty member vs non-member customer behavior Question: Do enrolled loyalty members actually order more often, spend more, stick around longer, and return items less than non-member customers, as of today? Grain: one row per customer segment: 'loyalty member' vs 'non-member' Columns: - segment - customer count - avg lifetime orders per customer - avg lifetime revenue per customer (EUR) - avg order value (EUR) - avg customer tenure days (first order to last order) - repeat purchase rate (% with 2+ orders) - refund rate (% of orders with any amount refunded or an associated return) Details: Population: all users (is_staff = false) across all tenants who have placed at least one order (any status) between 2023-01-01 and 2025-12-31. Segment = 'loyalty member' if the user has a row in loyalty_accounts (any program, any tenant), else 'non-member'. Revenue and order value: use orders.grand_total converted to EUR via exchange_rates using orders.currency_code and the rate as_of orders.placed_at (fall back to created_at if placed_at is null). Avg lifetime revenue per customer = total EUR revenue from non-cancelled orders / customer count in segment. Avg order value = total EUR revenue / total order count in segment. Tenure = days between each customer's first and last placed order (0 for single-order customers). Repeat purchase rate = share of customers in the segment with 2 or more orders. Refund rate = (orders with payments.amount_refunded > 0 OR an order_returns row) / total orders in the segment. Order rows with 'loyalty member' first, then 'non-member'. Check your query with the warehouse's dry_run tool, then hand it in with submit.
dry_runWITH orders_base AS ( SELECT o.id AS order_id, o.tenant_id, o.user_id, o.status, o.currency_code, o.grand_total, COALESCE(o.placed_at, o.created_at) AS order_date FROM orders o JOIN users u ON u.id = o.user_id AND u.tenant_id = o.tenant_id WHERE u.is_staff = false AND COALESCE(o.placed_at, o.created_at) >= DATE '2023-01-01' AND COALESCE(o.placed_at, o.created_at) < DATE '2026-01-01' ), orders_enriched AS ( SELECT ob.*, CASE WHEN ob.currency_code = 'EUR' THEN ob.grand_total ELSE ob.grand_total / er.rate END AS eur_amount, EXISTS (SELECT 1 FROM payments p WHERE p.order_id = ob.order_id AND p.amount_refunded > 0) AS has_refund, EXISTS (SELECT 1 FROM order_returns r WHERE r.order_id = ob.order_id) AS has_return FROM orders_base ob LEFT JOIN exchange_rates er ON er.base_code = 'EUR' AND er.quote_code = ob.currency_code AND er.as_of = CAST(ob.order_date AS DATE) ), customer_agg AS ( SELECT user_id, COUNT(*) AS total_orders, COUNT(*) FILTER (WHERE status <> 'cancelled') AS noncancelled_orders, SUM(eur_amount) FILTER (WHERE status <> 'cancelled') AS noncancelled_revenue, MIN(order_date) AS first_order_date, MAX(order_date) AS last_order_date, COUNT(*) FILTER (WHERE has_refund OR has_return) AS refunded_orders FROM orders_enriched GROUP BY user_id ), segmented AS ( SELECT ca.*, CASE WHEN EXISTS (SELECT 1 FROM loyalty_accounts la WHERE la.user_id = ca.user_id) THEN 'loyalty member' ELSE 'non-member' END AS segment FROM customer_agg ca ) SELECT segment, COUNT(*) AS customer_count, ROUND(AVG(total_orders)::numeric, 2) AS avg_lifetime_orders, ROUND((SUM(noncancelled_revenue) / NULLIF(COUNT(*),0))::numeric, 2) AS avg_lifetime_revenue_eur, ROUND((SUM(noncancelled_revenue) / NULLIF(SUM(noncancelled_orders),0))::numeric, 2) AS avg_order_value_eur, ROUND((AVG(EXTRACT(EPOCH FROM (last_order_date - first_order_date)) / 86400.0))::numeric, 1) AS avg_tenure_days, ROUND((100.0 * COUNT(*) FILTER (WHERE total_orders >= 2) / NULLIF(COUNT(*),0))::numeric, 1) AS repeat_purchase_rate_pct, ROUND((100.0 * SUM(refunded_orders) / NULLIF(SUM(total_orders),0))::numeric, 1) AS refund_rate_pct FROM segmented GROUP BY segment ORDER BY CASE WHEN segment = 'loyalty member' THEN 0 ELSE 1 END, segment
STATUS: COMPILES (dry run on the empty schema, no data read) COLUMNS: segment, customer_count, avg_lifetime_orders, avg_lifetime_revenue_eur, avg_order_value_eur, avg_tenure_days, repeat_purchase_rate_pct, refund_rate_pct
submitWITH orders_base AS ( SELECT o.id AS order_id, o.tenant_id, o.user_id, o.status, o.currency_code, o.grand_total, COALESCE(o.placed_at, o.created_at) AS order_date FROM orders o JOIN users u ON u.id = o.user_id AND u.tenant_id = o.tenant_id WHERE u.is_staff = false AND COALESCE(o.placed_at, o.created_at) >= DATE '2023-01-01' AND COALESCE(o.placed_at, o.created_at) < DATE '2026-01-01' ), orders_enriched AS ( SELECT ob.*, CASE WHEN ob.currency_code = 'EUR' THEN ob.grand_total ELSE ob.grand_total / er.rate END AS eur_amount, EXISTS (SELECT 1 FROM payments p WHERE p.order_id = ob.order_id AND p.amount_refunded > 0) AS has_refund, EXISTS (SELECT 1 FROM order_returns r WHERE r.order_id = ob.order_id) AS has_return FROM orders_base ob LEFT JOIN exchange_rates er ON er.base_code = 'EUR' AND er.quote_code = ob.currency_code AND er.as_of = CAST(ob.order_date AS DATE) ), customer_agg AS ( SELECT user_id, COUNT(*) AS total_orders, COUNT(*) FILTER (WHERE status <> 'cancelled') AS noncancelled_orders, SUM(eur_amount) FILTER (WHERE status <> 'cancelled') AS noncancelled_revenue, MIN(order_date) AS first_order_date, MAX(order_date) AS last_order_date, COUNT(*) FILTER (WHERE has_refund OR has_return) AS refunded_orders FROM orders_enriched GROUP BY user_id ), segmented AS ( SELECT ca.*, CASE WHEN EXISTS (SELECT 1 FROM loyalty_accounts la WHERE la.user_id = ca.user_id) THEN 'loyalty member' ELSE 'non-member' END AS segment FROM customer_agg ca ) SELECT segment, COUNT(*) AS customer_count, ROUND(AVG(total_orders)::numeric, 2) AS avg_lifetime_orders, ROUND((SUM(noncancelled_revenue) / NULLIF(COUNT(*),0))::numeric, 2) AS avg_lifetime_revenue_eur, ROUND((SUM(noncancelled_revenue) / NULLIF(SUM(noncancelled_orders),0))::numeric, 2) AS avg_order_value_eur, ROUND((AVG(EXTRACT(EPOCH FROM (last_order_date - first_order_date)) / 86400.0))::numeric, 1) AS avg_tenure_days, ROUND((100.0 * COUNT(*) FILTER (WHERE total_orders >= 2) / NULLIF(COUNT(*),0))::numeric, 1) AS repeat_purchase_rate_pct, ROUND((100.0 * SUM(refunded_orders) / NULLIF(SUM(total_orders),0))::numeric, 1) AS refund_rate_pct FROM segmented GROUP BY segment ORDER BY CASE WHEN segment = 'loyalty member' THEN 0 ELSE 1 END, segment
Submitted to tonight's batch. Your work on this request is done.