Queries · q286 · Head of Risk and Payments · week 2, night 4
“Did the February 2025 failure spike hit the same calendar days across all three affected merchants (pointing to one shared platform cause), and was access-denied logging silent specifically on those spike days?”
Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told
Week 1; shared with Rosa Van den Berg, Head of Data
An enterprise prospect sent a security and compliance questionnaire covering account security, access control, privacy requests, data retention and audit trails. Sales wants answers backed by evidence from our own systems, not by policy documents.
The manager, at the start of week 2
Subject: Re: Questionnaire memo — good work, a few threads to pull
Claire,
This is a strong memo, and the gaps you flagged are exactly the kind of thing I'd rather surface now than have a prospect's security team find later. A few things stand out and I want to understand them better before we finalize anything.
The February 2025 login spike is the one that worries me most. A 10x jump in failures with no corresponding rise in flagged suspicious activity is a real detection gap, not just a reporting curiosity — if that was a credential-stuffing attempt, I need to know whether it actually got anywhere.
Questions for this week:
Keep the role/permission logging gap and the broken query as noted — no need to keep spending slots chasing those; I'll raise the first with engineering myself and we can note the second as a known limitation in the memo.
Thanks, Aylin
The analyst · week 2, night 4, request 2 of 2
Night 3 results round out the picture nicely:
Report 1 (pre-2023 anomaly): change_history (2 outlier rows out of ~31,099) and compliance_events (1 outlier row out of 9) are clearly negligible noise. data_subject_requests is different: 12 of 60 total rows (20%) predate 2023-01-01 (1 in 2020, 4 in 2021, 7 in 2022) — not negligible. Reassuringly, the 2023-2025 rows sum to exactly 48 (10+23+15), matching the "~48 requests since 2023" figure I've already reported, so that number is internally consistent and unaffected — but I want to quickly check whether any of those 12 pre-2023 rows represent unresolved/backlog risk before I close this out, since they're sitting entirely outside the window I've been reporting on.
Reports 2 and 3 (Feb 2025 pattern + quarterly history) make the spike's character much clearer: 17/11/8 distinct user agents at the three merchants respectively with no single user agent dominating (~25-27% share) — not bot-like; and the quarterly history shows every other quarter at all three merchants sits at a stable 4.8-7.3% fail rate, with Q1 2025 alone spiking to 61-68% — a clean, isolated one-quarter event, not a recurring problem. One more thing worth nailing down before I write the memo: at each merchant individually, ~74-75% of failures cluster in just 3 days — I want to check whether it's the same 3 days across all three merchants, which would point to one shared platform-level cause rather than three coincidentally-timed separate issues, and whether access_denied logging was silent specifically on those days (not just the month overall).
This is the last night, so I'm keeping it tight: (1) a quick status check on the 12 pre-2023 DSAR rows, to close out any residual backlog question; (2) a day-by-day breakdown for February 2025 across the three merchants (failed logins per merchant plus access_denied events) to test the shared-cause hypothesis precisely. I'm leaving the third slot unused — I have what I need to write a confident memo and don't have a well-justified third question tonight.
The engineer · 2 dry runs on the empty schema, then the submission
SELECT id, name FROM tenants WHERE name ILIKE '%cragsley%' OR name ILIKE '%marrowby%' OR name ILIKE '%studio vrij%'
SELECT id, name FROM tenants WHERE name ILIKE '%cragsley%' OR name ILIKE '%marrowby%' OR name ILIKE '%studio vrij%'WITH target_tenants AS ( SELECT id, name FROM tenants WHERE name ILIKE '%cragsley%' OR name ILIKE '%marrowby%' OR name ILIKE '%studio vrij%'),days AS ( SELECT calendar_date FROM calendar_days WHERE calendar_date >= DATE '2025-02-01' AND calendar_date <= DATE '2025-02-28'),failed_logins AS ( SELECT la.tenant_id, DATE(la.occurred_at) AS login_date, COUNT(*) AS failed_count FROM login_attempts la JOIN target_tenants tt ON tt.id = la.tenant_id WHERE la.succeeded = false AND la.occurred_at >= TIMESTAMP '2025-02-01' AND la.occurred_at < TIMESTAMP '2025-03-01' GROUP BY la.tenant_id, DATE(la.occurred_at)),access_denied_all AS ( SELECT tenant_id, occurred_at FROM audit_log WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01' UNION ALL SELECT tenant_id, occurred_at FROM audit_log_archive WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01'),access_denied_daily AS ( SELECT DATE(ad.occurred_at) AS event_date, COUNT(*) AS access_denied_count FROM access_denied_all ad JOIN target_tenants tt ON tt.id = ad.tenant_id GROUP BY DATE(ad.occurred_at))SELECT d.calendar_date AS date, COALESCE(SUM(CASE WHEN tt.name ILIKE '%cragsley%' THEN fl.failed_count END), 0) AS cragsley_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%marrowby%' THEN fl.failed_count END), 0) AS marrowby_vane_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%studio vrij%' THEN fl.failed_count END), 0) AS studio_vrij_failed_logins, COALESCE(SUM(fl.failed_count), 0) AS total_failed_logins_3_merchants, COALESCE(MAX(adn.access_denied_count), 0) AS access_denied_events_3_merchantsFROM days dLEFT JOIN failed_logins fl ON fl.login_date = d.calendar_dateLEFT JOIN target_tenants tt ON tt.id = fl.tenant_idLEFT JOIN access_denied_daily adn ON adn.event_date = d.calendar_dateGROUP BY d.calendar_dateORDER BY d.calendar_date ASC
“This matches the requested grain (one row per day, Feb 2025) and columns. Submitting.”
“Submitted. One note for awareness: since the request named merchants by name rather than tenant_id, I matched them via ILIKE on tenants.name ('cragsley', 'marrowby', 'studio vrij') — flagged this in the submit note so the analyst can confirm those patterns uniquely resolve to the intended three tenants.”
As Claire Dubois saw it, from the small dataset the agents worked on
date | cragsley_failed_logins | marrowby_vane_failed_logins | studio_vrij_failed_logins | total_failed_logins_3_merchants | access_denied_events_3_merchants -----------+------------------------+-----------------------------+---------------------------+---------------------------------+--------------------------------- 2025-02-01 | 2 | 0 | 0 | 2 | 0 2025-02-02 | 4 | 2 | 0 | 6 | 0 2025-02-03 | 1 | 1 | 0 | 2 | 0 2025-02-04 | 4 | 0 | 0 | 4 | 0 2025-02-05 | 2 | 0 | 0 | 2 | 0 2025-02-06 | 4 | 0 | 0 | 4 | 0 2025-02-07 | 6 | 2 | 0 | 8 | 0 2025-02-08 | 1091 | 491 | 195 | 1777 | 0 2025-02-09 | 1090 | 493 | 195 | 1778 | 0 2025-02-10 | 1091 | 491 | 194 | 1776 | 0 2025-02-11 | 1085 | 491 | 196 | 1772 | 0 2025-02-12 | 5 | 0 | 0 | 5 | 0 2025-02-13 | 1 | 0 | 0 | 1 | 0 2025-02-14 | 0 | 0 | 0 | 0 | 0 2025-02-15 | 4 | 0 | 0 | 4 | 0 2025-02-16 | 1 | 0 | 0 | 1 | 0 2025-02-17 | 0 | 0 | 0 | 0 | 0 2025-02-18 | 3 | 0 | 1 | 4 | 1 2025-02-19 | 1 | 3 | 2 | 6 | 0 2025-02-20 | 0 | 0 | 0 | 0 | 0 2025-02-21 | 4 | 0 | 0 | 4 | 0 2025-02-22 | 0 | 1 | 0 | 1 | 0 2025-02-23 | 4 | 0 | 0 | 4 | 0 2025-02-24 | 0 | 0 | 0 | 0 | 0 2025-02-25 | 0 | 1 | 0 | 1 | 0 2025-02-26 | 2 | 0 | 0 | 2 | 0 2025-02-27 | 2 | 2 | 0 | 4 | 0 2025-02-28 | 0 | 0 | 0 | 0 | 0 (28 rows)
The memo and the queries that followed; links found by title, so inferred
It builds on:
Changed only where the benchmark needs the same result on every system
WITH target_tenants AS ( SELECT id, name FROM tenants WHERE name ILIKE '%cragsley%' OR name ILIKE '%marrowby%' OR name ILIKE '%studio vrij%'),days AS ( SELECT calendar_date FROM calendar_days WHERE calendar_date >= DATE '2025-02-01' AND calendar_date <= DATE '2025-02-28'),failed_logins AS ( SELECT la.tenant_id, DATE(la.occurred_at) AS login_date, la.tenant_id AS tenant_id, CAST(la.occurred_at AS DATE) AS login_date, COUNT(*) AS failed_count FROM login_attempts la JOIN target_tenants tt ON tt.id = la.tenant_id WHERE la.succeeded = false AND la.occurred_at >= TIMESTAMP '2025-02-01' AND la.occurred_at < TIMESTAMP '2025-03-01' GROUP BY la.tenant_id, DATE(la.occurred_at) GROUP BY la.tenant_id, CAST(la.occurred_at AS DATE)),access_denied_all AS ( SELECT tenant_id, occurred_at FROM audit_log WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01' UNION ALL SELECT tenant_id, occurred_at FROM audit_log_archive WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01'),access_denied_daily AS ( SELECT DATE(ad.occurred_at) AS event_date, CAST(ad.occurred_at AS DATE) AS event_date, COUNT(*) AS access_denied_count FROM access_denied_all ad JOIN target_tenants tt ON tt.id = ad.tenant_id GROUP BY DATE(ad.occurred_at) GROUP BY CAST(ad.occurred_at AS DATE))SELECT d.calendar_date AS date, COALESCE(SUM(CASE WHEN tt.name ILIKE '%cragsley%' THEN fl.failed_count END), 0) AS cragsley_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%marrowby%' THEN fl.failed_count END), 0) AS marrowby_vane_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%studio vrij%' THEN fl.failed_count END), 0) AS studio_vrij_failed_logins, COALESCE(SUM(fl.failed_count), 0) AS total_failed_logins_3_merchants, COALESCE(MAX(adn.access_denied_count), 0) AS access_denied_events_3_merchantsFROM days dLEFT JOIN failed_logins fl ON fl.login_date = d.calendar_dateLEFT JOIN target_tenants tt ON tt.id = fl.tenant_idLEFT JOIN access_denied_daily adn ON adn.event_date = d.calendar_dateGROUP BY d.calendar_dateORDER BY d.calendar_date ASCORDER BY d.calendar_date ASC NULLS LAST
WITH target_tenants AS ( SELECT id, name FROM tenants WHERE name ILIKE '%cragsley%' OR name ILIKE '%marrowby%' OR name ILIKE '%studio vrij%' ), days AS ( SELECT calendar_date FROM calendar_days WHERE calendar_date >= DATE '2025-02-01' AND calendar_date <= DATE '2025-02-28' ), failed_logins AS ( SELECT la.tenant_id AS tenant_id, CAST(la.occurred_at AS DATE) AS login_date, COUNT(*) AS failed_count FROM login_attempts la JOIN target_tenants tt ON tt.id = la.tenant_id WHERE la.succeeded = false AND la.occurred_at >= TIMESTAMP '2025-02-01' AND la.occurred_at < TIMESTAMP '2025-03-01' GROUP BY la.tenant_id, CAST(la.occurred_at AS DATE) ), access_denied_all AS ( SELECT tenant_id, occurred_at FROM audit_log WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01' UNION ALL SELECT tenant_id, occurred_at FROM audit_log_archive WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01' ), access_denied_daily AS ( SELECT CAST(ad.occurred_at AS DATE) AS event_date, COUNT(*) AS access_denied_count FROM access_denied_all ad JOIN target_tenants tt ON tt.id = ad.tenant_id GROUP BY CAST(ad.occurred_at AS DATE) ) SELECT d.calendar_date AS date, COALESCE(SUM(CASE WHEN tt.name ILIKE '%cragsley%' THEN fl.failed_count END), 0) AS cragsley_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%marrowby%' THEN fl.failed_count END), 0) AS marrowby_vane_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%studio vrij%' THEN fl.failed_count END), 0) AS studio_vrij_failed_logins, COALESCE(SUM(fl.failed_count), 0) AS total_failed_logins_3_merchants, COALESCE(MAX(adn.access_denied_count), 0) AS access_denied_events_3_merchants FROM days d LEFT JOIN failed_logins fl ON fl.login_date = d.calendar_date LEFT JOIN target_tenants tt ON tt.id = fl.tenant_id LEFT JOIN access_denied_daily adn ON adn.event_date = d.calendar_date GROUP BY d.calendar_date ORDER BY d.calendar_date ASC NULLS LAST
The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted
Report request from Claire Dubois (analyst for the Head of Risk and Payments): Title: Daily failed-login and access-denied alignment for the 3 merchants in February 2025 Question: Did the February 2025 failure spike hit the same calendar days across all three affected merchants (pointing to one shared platform cause), and was access-denied logging silent specifically on those spike days? Grain: one row per calendar day in February 2025 (28 rows) Columns: - date - cragsley_failed_logins - marrowby_vane_failed_logins - studio_vrij_failed_logins - total_failed_logins_3_merchants - access_denied_events_3_merchants (audit_log UNION audit_log_archive, action='access_denied', tenant_id in these 3 tenants, occurred_at on this date) Details: Source: login_attempts (succeeded=false, Feb 2025, split by tenant into the three named columns) and audit_log/audit_log_archive (action='access_denied', same 3 tenant_ids, same days). Purpose: last night's report showed each merchant independently has ~74-75% of its failures in its own top-3 days; this checks whether those days coincide across merchants (shared root cause) and whether access_denied events ticked up at all on those specific days or stayed flat (confirming the detection gap at daily, not just monthly, granularity). Order by date ascending. Check your query with the warehouse's dry_run tool, then hand it in with submit.
dry_runSELECT id, name FROM tenants WHERE name ILIKE '%cragsley%' OR name ILIKE '%marrowby%' OR name ILIKE '%studio vrij%'
STATUS: COMPILES (dry run on the empty schema, no data read) COLUMNS: id, name
dry_runWITH target_tenants AS ( SELECT id, name FROM tenants WHERE name ILIKE '%cragsley%' OR name ILIKE '%marrowby%' OR name ILIKE '%studio vrij%' ), days AS ( SELECT calendar_date FROM calendar_days WHERE calendar_date >= DATE '2025-02-01' AND calendar_date <= DATE '2025-02-28' ), failed_logins AS ( SELECT la.tenant_id, DATE(la.occurred_at) AS login_date, COUNT(*) AS failed_count FROM login_attempts la JOIN target_tenants tt ON tt.id = la.tenant_id WHERE la.succeeded = false AND la.occurred_at >= TIMESTAMP '2025-02-01' AND la.occurred_at < TIMESTAMP '2025-03-01' GROUP BY la.tenant_id, DATE(la.occurred_at) ), access_denied_all AS ( SELECT tenant_id, occurred_at FROM audit_log WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01' UNION ALL SELECT tenant_id, occurred_at FROM audit_log_archive WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01' ), access_denied_daily AS ( SELECT DATE(ad.occurred_at) AS event_date, COUNT(*) AS access_denied_count FROM access_denied_all ad JOIN target_tenants tt ON tt.id = ad.tenant_id GROUP BY DATE(ad.occurred_at) ) SELECT d.calendar_date AS date, COALESCE(SUM(CASE WHEN tt.name ILIKE '%cragsley%' THEN fl.failed_count END), 0) AS cragsley_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%marrowby%' THEN fl.failed_count END), 0) AS marrowby_vane_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%studio vrij%' THEN fl.failed_count END), 0) AS studio_vrij_failed_logins, COALESCE(SUM(fl.failed_count), 0) AS total_failed_logins_3_merchants, COALESCE(MAX(adn.access_denied_count), 0) AS access_denied_events_3_merchants FROM days d LEFT JOIN failed_logins fl ON fl.login_date = d.calendar_date LEFT JOIN target_tenants tt ON tt.id = fl.tenant_id LEFT JOIN access_denied_daily adn ON adn.event_date = d.calendar_date GROUP BY d.calendar_date ORDER BY d.calendar_date ASC
STATUS: COMPILES (dry run on the empty schema, no data read) COLUMNS: date, cragsley_failed_logins, marrowby_vane_failed_logins, studio_vrij_failed_logins, total_failed_logins_3_merchants, access_denied_events_3_merchants
submitWITH target_tenants AS ( SELECT id, name FROM tenants WHERE name ILIKE '%cragsley%' OR name ILIKE '%marrowby%' OR name ILIKE '%studio vrij%' ), days AS ( SELECT calendar_date FROM calendar_days WHERE calendar_date >= DATE '2025-02-01' AND calendar_date <= DATE '2025-02-28' ), failed_logins AS ( SELECT la.tenant_id, DATE(la.occurred_at) AS login_date, COUNT(*) AS failed_count FROM login_attempts la JOIN target_tenants tt ON tt.id = la.tenant_id WHERE la.succeeded = false AND la.occurred_at >= TIMESTAMP '2025-02-01' AND la.occurred_at < TIMESTAMP '2025-03-01' GROUP BY la.tenant_id, DATE(la.occurred_at) ), access_denied_all AS ( SELECT tenant_id, occurred_at FROM audit_log WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01' UNION ALL SELECT tenant_id, occurred_at FROM audit_log_archive WHERE action = 'access_denied' AND occurred_at >= TIMESTAMP '2025-02-01' AND occurred_at < TIMESTAMP '2025-03-01' ), access_denied_daily AS ( SELECT DATE(ad.occurred_at) AS event_date, COUNT(*) AS access_denied_count FROM access_denied_all ad JOIN target_tenants tt ON tt.id = ad.tenant_id GROUP BY DATE(ad.occurred_at) ) SELECT d.calendar_date AS date, COALESCE(SUM(CASE WHEN tt.name ILIKE '%cragsley%' THEN fl.failed_count END), 0) AS cragsley_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%marrowby%' THEN fl.failed_count END), 0) AS marrowby_vane_failed_logins, COALESCE(SUM(CASE WHEN tt.name ILIKE '%studio vrij%' THEN fl.failed_count END), 0) AS studio_vrij_failed_logins, COALESCE(SUM(fl.failed_count), 0) AS total_failed_logins_3_merchants, COALESCE(MAX(adn.access_denied_count), 0) AS access_denied_events_3_merchants FROM days d LEFT JOIN failed_logins fl ON fl.login_date = d.calendar_date LEFT JOIN target_tenants tt ON tt.id = fl.tenant_id LEFT JOIN access_denied_daily adn ON adn.event_date = d.calendar_date GROUP BY d.calendar_date ORDER BY d.calendar_date ASC
Submitted to tonight's batch. Your work on this request is done.