Queries · q279 · Head of Risk and Payments · week 2, night 2
“Among the accounts with 5+ failed logins in February 2025, how many went on to log in successfully afterward, and how soon — i.e. was this a near-miss or just noise?”
Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told
Week 1; shared with Rosa Van den Berg, Head of Data
An enterprise prospect sent a security and compliance questionnaire covering account security, access control, privacy requests, data retention and audit trails. Sales wants answers backed by evidence from our own systems, not by policy documents.
The manager, at the start of week 2
Subject: Re: Questionnaire memo — good work, a few threads to pull
Claire,
This is a strong memo, and the gaps you flagged are exactly the kind of thing I'd rather surface now than have a prospect's security team find later. A few things stand out and I want to understand them better before we finalize anything.
The February 2025 login spike is the one that worries me most. A 10x jump in failures with no corresponding rise in flagged suspicious activity is a real detection gap, not just a reporting curiosity — if that was a credential-stuffing attempt, I need to know whether it actually got anywhere.
Questions for this week:
Keep the role/permission logging gap and the broken query as noted — no need to keep spending slots chasing those; I'll raise the first with engineering myself and we can note the second as a known limitation in the memo.
Thanks, Aylin
The analyst · week 2, night 2, request 1 of 3
Night 1 results need a careful read before I pass anything to Aylin:
Report 1 (concentration by merchant) is clean and striking: 3 merchants — Cragsley Outfitters (4,407 failed, 88.1% fail rate), Marrowby & Vane (1,978 failed, 86.1%), Studio Vrij (783 failed, 84.2%) — account for ~7,168 of the ~7,429 failed attempts we found last week (96.5%), while the next-highest merchant had just 17 failed attempts. These are the exact same three merchants Support Ops called out last week as having both the heaviest ticket volume and worst response times. But the pattern doesn't look like a credential-stuffing attack: distinct_emails_with_any_failed_attempt is nearly as high as total failed attempts (e.g. Cragsley: 2,786 distinct emails for 4,407 fails, max 8 fails on any single email, max 9 from any single IP, spread across 1,903 distinct IPs). That's wide, shallow failure across many accounts and many sources at once — much more consistent with a broken login flow/client bug affecting those 3 merchants specifically than with an external attack.
Report 2 (near-miss outcomes) is internally broken and I will not pass these numbers on: percentages are computed against a fixed denominator of 102, but the bucket counts (42+12+89+452=595) don't come close to summing to 102, and the engineer's note confirms email was matched globally rather than scoped per tenant (login_attempts.email is only unique per tenant) — so the "cohort" silently grew across tenants mid-query while the percentage base didn't update. This is exactly the kind of actor/identity mismatch Rosa flagged in week 1, now showing up in a different table. I'm retrying tonight with tenant-scoped (tenant_id, email) pairs and mutually-exclusive buckets that must sum to the true cohort size.
Report 3 (DSAR via updated_at) looks solid and resolves Aylin's question #2: all 3 rejected DSARs show updated_at well inside their 30-day due_at window (8.7, 3.8, and 23.7 days after receipt respectively) — plausibly within deadline, with the usual caveat that updated_at isn't a verified "decision" timestamp.
Tonight (3 slots): (1) retry the near-miss report correctly; (2) incidents/compliance_events volume, severity, and resolution time (question #3, deferred from last week); (3) how far back the five retention-policy-less tables actually go, to size the exposure for question #4.
The engineer · 1 dry run on the empty schema, then the submission
WITH failed_feb AS ( SELECT tenant_id, email, occurred_at FROM login_attempts WHERE succeeded = false AND occurred_at >= DATE '2025-02-01' AND occurred_at < DATE '2025-03-01' ), cohort AS ( SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at FROM failed_feb GROUP BY tenant_id, email HAVING COUNT(*) >= 5 ), success_after AS ( SELECT c.tenant_id, c.email, c.failed_count, c.last_failed_at, MIN(la.occurred_at) AS success_at FROM cohort c LEFT JOIN login_attempts la ON la.tenant_id = c.tenant_id AND la.email = c.email AND la.succeeded = true AND la.occurred_at > c.last_failed_at AND la.occurred_at < DATE '2025-03-01' GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at ), classified AS ( SELECT tenant_id, email, failed_count, CASE WHEN success_at IS NULL THEN 'never_succeeded_in_month' WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h' WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h' ELSE 'succeeded_later_in_month' END AS outcome_bucket FROM success_after ), cohort_size AS ( SELECT COUNT(*) AS n FROM cohort ), bucket_list AS ( SELECT * FROM (VALUES ('succeeded_within_1h', 1), ('succeeded_within_24h', 2), ('succeeded_later_in_month', 3), ('never_succeeded_in_month', 4) ) AS b(outcome_bucket, sort_order) ) SELECT b.outcome_bucket, COUNT(c.tenant_id) AS pair_count, ROUND(100.0 * COUNT(c.tenant_id) / cs.n, 1) AS pct_of_cohort, CASE WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL ELSE ROUND(AVG(c.failed_count), 2) END AS avg_failed_attempts_before_success, COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affected FROM bucket_list b LEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucket CROSS JOIN cohort_size cs GROUP BY b.outcome_bucket, b.sort_order, cs.n ORDER BY b.sort_order
“Query compiles with the expected columns. Submitting.”
“Submitted. Summary of the approach:
- Cohort: distinct (tenant_id, email) pairs with ≥5 succeeded=false rows in Feb 2025, keyed correctly by tenant+email (not email alone) per the correction.
- Matching success: for each pair, the first succeeded=true row for that same (tenant_id, email) strictly after the pair's last Feb failure, still within February.
- Buckets: classified by elapsed time (≤1h, ≤24h, else later-in-month, else never), applied with the requested precedence, and always printed as all 4 rows (zero-filled) so pair_count sums exactly to the cohort size and pct_of_cohort sums to 100%.
- avg_failed_attempts_before_success: average of each pair's total Feb failed-attempt count (left blank for the never-succeeded bucket), since the matching success is always after the last failure, making every Feb failure "before success" by construction.
- distinct_tenants_affected: count of distinct tenants contributing to each bucket.”
As Claire Dubois saw it, from the small dataset the agents worked on
outcome_bucket | pair_count | pct_of_cohort | avg_failed_attempts_before_success | distinct_tenants_affected -------------------------+------------+---------------+------------------------------------+-------------------------- succeeded_within_1h | 6 | 6.1 | 5.5 | 3 succeeded_within_24h | 2 | 2.0 | 5.0 | 2 succeeded_later_in_month | 12 | 12.2 | 5.5 | 2 never_succeeded_in_month | 78 | 79.6 | NULL | 3 (4 rows)
The memo and the queries that followed; links found by title, so inferred
It builds on:
Changed only where the benchmark needs the same result on every system
WITH failed_feb AS ( SELECT tenant_id, email, occurred_at FROM login_attempts WHERE succeeded = false AND occurred_at >= DATE '2025-02-01' AND occurred_at < DATE '2025-03-01'),cohort AS ( SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at FROM failed_feb GROUP BY tenant_id, email HAVING COUNT(*) >= 5),success_after AS ( SELECT c.tenant_id, c.email, c.failed_count, c.last_failed_at, c.tenant_id AS tenant_id, c.email AS email, c.failed_count AS failed_count, c.last_failed_at AS last_failed_at, MIN(la.occurred_at) AS success_at FROM cohort c LEFT JOIN login_attempts la ON la.tenant_id = c.tenant_id AND la.email = c.email AND la.succeeded = true AND la.occurred_at > c.last_failed_at AND la.occurred_at < DATE '2025-03-01' GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at),classified AS ( SELECT tenant_id, email, failed_count, CASE WHEN success_at IS NULL THEN 'never_succeeded_in_month' WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h' WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h' ELSE 'succeeded_later_in_month' END AS outcome_bucket FROM success_after),cohort_size AS ( SELECT COUNT(*) AS n FROM cohort),bucket_list AS ( SELECT * FROM (VALUES ('succeeded_within_1h', 1), ('succeeded_within_24h', 2), ('succeeded_later_in_month', 3), ('never_succeeded_in_month', 4) ) AS b(outcome_bucket, sort_order))SELECT b.outcome_bucket, b.outcome_bucket AS outcome_bucket, COUNT(c.tenant_id) AS pair_count, ROUND(100.0 * COUNT(c.tenant_id) / cs.n, 1) AS pct_of_cohort, 100.0 * COUNT(c.tenant_id) / NULLIF(cs.n, 0) AS pct_of_cohort, CASE WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL ELSE ROUND(AVG(c.failed_count), 2) ELSE AVG(CAST(c.failed_count AS NUMERIC(18,6))) END AS avg_failed_attempts_before_success, COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affectedFROM bucket_list bLEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucketCROSS JOIN cohort_size csGROUP BY b.outcome_bucket, b.sort_order, cs.nORDER BY b.sort_orderORDER BY b.sort_order NULLS LAST
WITH failed_feb AS ( SELECT tenant_id, email, occurred_at FROM login_attempts WHERE succeeded = false AND occurred_at >= DATE '2025-02-01' AND occurred_at < DATE '2025-03-01' ), cohort AS ( SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at FROM failed_feb GROUP BY tenant_id, email HAVING COUNT(*) >= 5 ), success_after AS ( SELECT c.tenant_id AS tenant_id, c.email AS email, c.failed_count AS failed_count, c.last_failed_at AS last_failed_at, MIN(la.occurred_at) AS success_at FROM cohort c LEFT JOIN login_attempts la ON la.tenant_id = c.tenant_id AND la.email = c.email AND la.succeeded = true AND la.occurred_at > c.last_failed_at AND la.occurred_at < DATE '2025-03-01' GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at ), classified AS ( SELECT tenant_id, email, failed_count, CASE WHEN success_at IS NULL THEN 'never_succeeded_in_month' WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h' WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h' ELSE 'succeeded_later_in_month' END AS outcome_bucket FROM success_after ), cohort_size AS ( SELECT COUNT(*) AS n FROM cohort ), bucket_list AS ( SELECT * FROM (VALUES ('succeeded_within_1h', 1), ('succeeded_within_24h', 2), ('succeeded_later_in_month', 3), ('never_succeeded_in_month', 4) ) AS b(outcome_bucket, sort_order) ) SELECT b.outcome_bucket AS outcome_bucket, COUNT(c.tenant_id) AS pair_count, 100.0 * COUNT(c.tenant_id) / NULLIF(cs.n, 0) AS pct_of_cohort, CASE WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL ELSE AVG(CAST(c.failed_count AS NUMERIC(18,6))) END AS avg_failed_attempts_before_success, COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affected FROM bucket_list b LEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucket CROSS JOIN cohort_size cs GROUP BY b.outcome_bucket, b.sort_order, cs.n ORDER BY b.sort_order NULLS LAST
The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted
Report request from Claire Dubois (analyst for the Head of Risk and Payments): Title: February 2025 repeat-failure accounts: did they succeed afterward? (corrected) Question: Among the accounts with 5+ failed logins in February 2025, how many went on to log in successfully afterward, and how soon — i.e. was this a near-miss or just noise? Grain: one row per outcome bucket, computed over the cohort of distinct (tenant_id, email) pairs with >=5 failed login_attempts rows in February 2025 Columns: - outcome_bucket (succeeded_within_1h / succeeded_within_24h / succeeded_later_in_month / never_succeeded_in_month) - pair_count - pct_of_cohort - avg_failed_attempts_before_success (blank for never_succeeded) - distinct_tenants_affected Details: Fix for last night's broken version: scope the cohort to (tenant_id, email) pairs, not email alone, since login_attempts.email is only unique per tenant and matching on email alone silently conflated unrelated users across merchants. For each pair in the cohort, find the first succeeded=true login_attempts row for that same (tenant_id, email) occurring after that pair's last Feb-2025 failed attempt, within February 2025 only. Classify each pair into exactly one bucket using this precedence: succeeded_within_1h, else succeeded_within_24h, else succeeded_later_in_month, else never_succeeded_in_month. Buckets must be mutually exclusive and exhaustive: pair_count across all 4 rows must sum exactly to the cohort size, and pct_of_cohort must sum to 100%. Do not print individual emails. Order buckets in the sequence listed above. Check your query with the warehouse's dry_run tool, then hand it in with submit.
dry_runWITH failed_feb AS ( SELECT tenant_id, email, occurred_at FROM login_attempts WHERE succeeded = false AND occurred_at >= DATE '2025-02-01' AND occurred_at < DATE '2025-03-01' ), cohort AS ( SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at FROM failed_feb GROUP BY tenant_id, email HAVING COUNT(*) >= 5 ), success_after AS ( SELECT c.tenant_id, c.email, c.failed_count, c.last_failed_at, MIN(la.occurred_at) AS success_at FROM cohort c LEFT JOIN login_attempts la ON la.tenant_id = c.tenant_id AND la.email = c.email AND la.succeeded = true AND la.occurred_at > c.last_failed_at AND la.occurred_at < DATE '2025-03-01' GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at ), classified AS ( SELECT tenant_id, email, failed_count, CASE WHEN success_at IS NULL THEN 'never_succeeded_in_month' WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h' WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h' ELSE 'succeeded_later_in_month' END AS outcome_bucket FROM success_after ), cohort_size AS ( SELECT COUNT(*) AS n FROM cohort ), bucket_list AS ( SELECT * FROM (VALUES ('succeeded_within_1h', 1), ('succeeded_within_24h', 2), ('succeeded_later_in_month', 3), ('never_succeeded_in_month', 4) ) AS b(outcome_bucket, sort_order) ) SELECT b.outcome_bucket, COUNT(c.tenant_id) AS pair_count, ROUND(100.0 * COUNT(c.tenant_id) / cs.n, 1) AS pct_of_cohort, CASE WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL ELSE ROUND(AVG(c.failed_count), 2) END AS avg_failed_attempts_before_success, COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affected FROM bucket_list b LEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucket CROSS JOIN cohort_size cs GROUP BY b.outcome_bucket, b.sort_order, cs.n ORDER BY b.sort_order
STATUS: COMPILES (dry run on the empty schema, no data read) COLUMNS: outcome_bucket, pair_count, pct_of_cohort, avg_failed_attempts_before_success, distinct_tenants_affected
submitWITH failed_feb AS ( SELECT tenant_id, email, occurred_at FROM login_attempts WHERE succeeded = false AND occurred_at >= DATE '2025-02-01' AND occurred_at < DATE '2025-03-01' ), cohort AS ( SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at FROM failed_feb GROUP BY tenant_id, email HAVING COUNT(*) >= 5 ), success_after AS ( SELECT c.tenant_id, c.email, c.failed_count, c.last_failed_at, MIN(la.occurred_at) AS success_at FROM cohort c LEFT JOIN login_attempts la ON la.tenant_id = c.tenant_id AND la.email = c.email AND la.succeeded = true AND la.occurred_at > c.last_failed_at AND la.occurred_at < DATE '2025-03-01' GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at ), classified AS ( SELECT tenant_id, email, failed_count, CASE WHEN success_at IS NULL THEN 'never_succeeded_in_month' WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h' WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h' ELSE 'succeeded_later_in_month' END AS outcome_bucket FROM success_after ), cohort_size AS ( SELECT COUNT(*) AS n FROM cohort ), bucket_list AS ( SELECT * FROM (VALUES ('succeeded_within_1h', 1), ('succeeded_within_24h', 2), ('succeeded_later_in_month', 3), ('never_succeeded_in_month', 4) ) AS b(outcome_bucket, sort_order) ) SELECT b.outcome_bucket, COUNT(c.tenant_id) AS pair_count, ROUND(100.0 * COUNT(c.tenant_id) / cs.n, 1) AS pct_of_cohort, CASE WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL ELSE ROUND(AVG(c.failed_count), 2) END AS avg_failed_attempts_before_success, COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affected FROM bucket_list b LEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucket CROSS JOIN cohort_size cs GROUP BY b.outcome_bucket, b.sort_order, cs.n ORDER BY b.sort_order
Submitted to tonight's batch. Your work on this request is done.