Queries · q074 · Head of Risk and Payments · week 1, night 4
“How much login activity fails, is there visible evidence of brute-force-style patterns, and are access-denied events actually being captured as suspicious activity over time?”
Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told
Week 1; shared with Rosa Van den Berg, Head of Data
An enterprise prospect sent a security and compliance questionnaire covering account security, access control, privacy requests, data retention and audit trails. Sales wants answers backed by evidence from our own systems, not by policy documents.
The manager, at the start of week 1
Subject: Evidence we need for the security/compliance questionnaire (this week's priority)
Hi Claire,
This questionnaire is going to a serious enterprise prospect, and Sales wants it backed by evidence from our own systems, not by our policy wording. Rosa and I are co-owning this, so your reports will feed both the questionnaire answers themselves and a broader readiness review Rosa's side is doing. Given you only get a handful of report slots a night, let's be selective and prioritize what a security reviewer will actually probe on. In my own words, here's what I need to know:
Given the slot constraints, please treat questions 3 and 1 as the top two for this week — those are the ones Sales says the prospect calls out by name. If we only get through three of the five, drop 5 first and fold it into next week.
One more thing: since this report touches account security and personal data, please keep the output at a safe aggregate level — no need to name individual customers or staff unless I specifically ask.
Let's sync briefly before you lock in Thursday night's batch so nothing gets wasted on a bounced or bundled request.
Thanks, Aylin
Hi Claire,
Before we hand evidence to this prospect, I'd want us to be confident the underlying records are actually solid enough to stand behind in writing. A few things from my side:
If any of these raise flags on your end, I'd rather we know before it goes into a customer-facing answer than after.
Thanks, Rosa
The analyst · week 1, night 4, request 2 of 3
Night 3 take-aways — one genuinely important finding, one bug, one solid result:
Report 1 (audit coverage sweep): decisive and concerning. Across all 13 identity/access-control tables I checked (roles, user_roles, role_permissions, permissions, teams, team_members, api_keys, invitations, mfa_devices, sessions, service_accounts, personal_access_tokens — plus singular/plural variants), only the users table shows ANY audit_log/audit_log_archive activity at all (delete, export, login, logout, update — no create, oddly). Role and permission grant/revocation events are simply not captured in the audit trail, at any point in the 3-year history. This confirms last night's all-zero result wasn't a query bug — it's a real gap. This has to go into the memo plainly for both Aylin (we cannot evidence who granted/revoked a given access right, only today's snapshot) and Rosa (this is exactly the kind of audit-trail incompleteness she asked me to flag before it goes into a customer-facing answer).
Report 2 (DSAR rejection timing) failed on execution: a cast error pulling in an unrelated product_variant row's after_data, which tells me the query matched change_history on entity_id alone without also constraining entity_table — entity_id isn't globally unique across tables, so it collided with an unrelated row. This is a fixable query bug, not a data problem, and worth one retry since it's the last loose end on our top priority (#3) — the only thing standing between us and a clean, fully-evidenced DSAR story.
Report 3 (MFA adoption): first real evidence for priority #2. Confirmed-MFA coverage for active staff is 54.9% (57.8% any device) — material but not universal, worth citing with the caveat that ~45% of merchant admin accounts are not MFA-protected. Customer-facing accounts show 0% MFA entirely (not even attempted) — expected if MFA is currently a staff-only feature, but worth stating plainly rather than silently.
No sync reply from Aylin arrived before lock-in, so I'm proceeding on her original stated priority order: #1 and #3 are solid for the memo (with the audit-gap and rejection-timestamp caveats to close out), so tonight's last 3 slots go to: (1) a defensively-rewritten retry of the DSAR rejection-timing query, (2) the other half of priority #2 — failed-login and suspicious-access patterns over time, since we've only covered MFA adoption so far, and (3) a quick retention-policy inventory for the audit/compliance tables, which gives us a toehold on priority #4 (ahead of #5, which Aylin said to drop first) and directly tests Rosa's "are we quietly losing history early" concern given what we now know about the audit-trail gap.
The engineer · 1 dry run on the empty schema, then the submission
WITH months AS ( SELECT DISTINCT month_label, month_start FROM calendar_days WHERE calendar_date BETWEEN DATE '2023-01-01' AND DATE '2025-12-31' ), login_month AS ( SELECT date_trunc('month', occurred_at)::date AS month_start, email, succeeded FROM login_attempts ), login_agg AS ( SELECT month_start, COUNT(*) AS total_login_attempts, COUNT(*) FILTER (WHERE succeeded = false) AS failed_login_attempts FROM login_month GROUP BY month_start ), brute_force AS ( SELECT month_start, COUNT(DISTINCT email) AS emails_with_5plus_failed FROM ( SELECT month_start, email, COUNT(*) AS cnt FROM login_month WHERE succeeded = false AND email IS NOT NULL GROUP BY month_start, email HAVING COUNT(*) >= 5 ) t GROUP BY month_start ), audit_combined AS ( SELECT occurred_at, action FROM audit_log WHERE action = 'access_denied' UNION ALL SELECT occurred_at, action FROM audit_log_archive WHERE action = 'access_denied' ), audit_agg AS ( SELECT date_trunc('month', occurred_at)::date AS month_start, COUNT(*) AS audit_access_denied_events FROM audit_combined GROUP BY month_start ) SELECT m.month_label AS month, COALESCE(la.total_login_attempts, 0) AS total_login_attempts, COALESCE(la.failed_login_attempts, 0) AS failed_login_attempts, ROUND(CASE WHEN COALESCE(la.total_login_attempts,0) = 0 THEN 0 ELSE 100.0 * la.failed_login_attempts / la.total_login_attempts END, 2) AS pct_failed, COALESCE(bf.emails_with_5plus_failed, 0) AS emails_with_5plus_failed_in_month, COALESCE(aa.audit_access_denied_events, 0) AS audit_access_denied_events FROM months m LEFT JOIN login_agg la ON la.month_start = m.month_start LEFT JOIN brute_force bf ON bf.month_start = m.month_start LEFT JOIN audit_agg aa ON aa.month_start = m.month_start ORDER BY m.month_start
“Query compiles and returns the expected columns. Submitting now.”
“Submitted. The report gives one row per month (2023-01 through 2025-12) with: total and failed login attempts, pct_failed, a distinct-email brute-force proxy (≥5 failed logins by the same email in that month), and the parallel access_denied audit signal from audit_log+audit_log_archive. Months with no activity still appear with zeros, thanks to the calendar join, so Aylin can see any spikes or gaps across the full history.”
As Claire Dubois saw it, from the small dataset the agents worked on
month | total_login_attempts | failed_login_attempts | pct_failed | emails_with_5plus_failed_in_month | audit_access_denied_events --------+----------------------+-----------------------+------------+-----------------------------------+--------------------------- 2023-01 | 3491 | 213 | 6.1 | 0 | 11 2023-02 | 2949 | 176 | 5.97 | 0 | 3 2023-03 | 3368 | 185 | 5.49 | 1 | 6 2023-04 | 3307 | 208 | 6.29 | 0 | 5 2023-05 | 3614 | 223 | 6.17 | 1 | 7 2023-06 | 3611 | 250 | 6.92 | 0 | 3 2023-07 | 3637 | 208 | 5.72 | 1 | 3 2023-08 | 3905 | 260 | 6.66 | 0 | 3 2023-09 | 3779 | 244 | 6.46 | 1 | 6 2023-10 | 4071 | 242 | 5.94 | 0 | 8 2023-11 | 5038 | 302 | 5.99 | 2 | 5 2023-12 | 4551 | 288 | 6.33 | 0 | 4 2024-01 | 4590 | 263 | 5.73 | 1 | 3 2024-02 | 4070 | 233 | 5.72 | 0 | 7 2024-03 | 4654 | 291 | 6.25 | 1 | 5 2024-04 | 4676 | 286 | 6.12 | 0 | 4 2024-05 | 4977 | 317 | 6.37 | 1 | 5 2024-06 | 4559 | 314 | 6.89 | 0 | 5 2024-07 | 5051 | 289 | 5.72 | 0 | 5 2024-08 | 4990 | 289 | 5.79 | 0 | 5 2024-09 | 4952 | 286 | 5.78 | 0 | 3 2024-10 | 5453 | 312 | 5.72 | 1 | 5 2024-11 | 6131 | 360 | 5.87 | 0 | 4 2024-12 | 6198 | 360 | 5.81 | 1 | 5 2025-01 | 5933 | 377 | 6.35 | 0 | 5 2025-02 | 12211 | 7429 | 60.84 | 102 | 6 2025-03 | 5627 | 373 | 6.63 | 0 | 10 2025-04 | 5636 | 323 | 5.73 | 0 | 2 2025-05 | 5886 | 376 | 6.39 | 0 | 2 2025-06 | 5439 | 306 | 5.63 | 1 | 5 2025-07 | 6110 | 349 | 5.71 | 1 | 5 2025-08 | 5877 | 393 | 6.69 | 0 | 3 2025-09 | 6125 | 377 | 6.16 | 0 | 1 2025-10 | 6711 | 411 | 6.12 | 0 | 7 2025-11 | 7532 | 477 | 6.33 | 0 | 5 2025-12 | 7598 | 443 | 5.83 | 1 | 5 (36 rows)
The memo and the queries that followed; links found by title, so inferred
Later queries that build on this report:
It builds on:
Changed only where the benchmark needs the same result on every system
WITH months AS ( SELECT DISTINCT month_label, month_start FROM calendar_days WHERE calendar_date BETWEEN DATE '2023-01-01' AND DATE '2025-12-31'),login_month AS ( SELECT date_trunc('month', occurred_at)::date AS month_start, SELECT CAST(date_trunc('month', occurred_at) AS date) AS month_start, email, succeeded FROM login_attempts),login_agg AS ( SELECT month_start, COUNT(*) AS total_login_attempts, COUNT(*) FILTER (WHERE succeeded = false) AS failed_login_attempts FROM login_month GROUP BY month_start),brute_force AS ( SELECT month_start, COUNT(DISTINCT email) AS emails_with_5plus_failed FROM ( SELECT month_start, email, COUNT(*) AS cnt FROM login_month WHERE succeeded = false AND email IS NOT NULL GROUP BY month_start, email HAVING COUNT(*) >= 5 ) t GROUP BY month_start),audit_combined AS ( SELECT occurred_at, action FROM audit_log WHERE action = 'access_denied' UNION ALL SELECT occurred_at, action FROM audit_log_archive WHERE action = 'access_denied'),audit_agg AS ( SELECT date_trunc('month', occurred_at)::date AS month_start, SELECT CAST(date_trunc('month', occurred_at) AS date) AS month_start, COUNT(*) AS audit_access_denied_events FROM audit_combined GROUP BY month_start)SELECT m.month_label AS month, COALESCE(la.total_login_attempts, 0) AS total_login_attempts, COALESCE(la.failed_login_attempts, 0) AS failed_login_attempts, ROUND(CASE WHEN COALESCE(la.total_login_attempts,0) = 0 THEN 0 ELSE 100.0 * la.failed_login_attempts / la.total_login_attempts END, 2) AS pct_failed, CASE WHEN COALESCE(la.total_login_attempts,0) = 0 THEN 0 ELSE 100.0 * la.failed_login_attempts / la.total_login_attempts END AS pct_failed, COALESCE(bf.emails_with_5plus_failed, 0) AS emails_with_5plus_failed_in_month, COALESCE(aa.audit_access_denied_events, 0) AS audit_access_denied_eventsFROM months mLEFT JOIN login_agg la ON la.month_start = m.month_startLEFT JOIN brute_force bf ON bf.month_start = m.month_startLEFT JOIN audit_agg aa ON aa.month_start = m.month_startORDER BY m.month_startORDER BY m.month_start NULLS LAST, m.month_label NULLS LAST
WITH months AS ( SELECT DISTINCT month_label, month_start FROM calendar_days WHERE calendar_date BETWEEN DATE '2023-01-01' AND DATE '2025-12-31' ), login_month AS ( SELECT CAST(date_trunc('month', occurred_at) AS date) AS month_start, email, succeeded FROM login_attempts ), login_agg AS ( SELECT month_start, COUNT(*) AS total_login_attempts, COUNT(*) FILTER (WHERE succeeded = false) AS failed_login_attempts FROM login_month GROUP BY month_start ), brute_force AS ( SELECT month_start, COUNT(DISTINCT email) AS emails_with_5plus_failed FROM ( SELECT month_start, email, COUNT(*) AS cnt FROM login_month WHERE succeeded = false AND email IS NOT NULL GROUP BY month_start, email HAVING COUNT(*) >= 5 ) t GROUP BY month_start ), audit_combined AS ( SELECT occurred_at, action FROM audit_log WHERE action = 'access_denied' UNION ALL SELECT occurred_at, action FROM audit_log_archive WHERE action = 'access_denied' ), audit_agg AS ( SELECT CAST(date_trunc('month', occurred_at) AS date) AS month_start, COUNT(*) AS audit_access_denied_events FROM audit_combined GROUP BY month_start ) SELECT m.month_label AS month, COALESCE(la.total_login_attempts, 0) AS total_login_attempts, COALESCE(la.failed_login_attempts, 0) AS failed_login_attempts, CASE WHEN COALESCE(la.total_login_attempts,0) = 0 THEN 0 ELSE 100.0 * la.failed_login_attempts / la.total_login_attempts END AS pct_failed, COALESCE(bf.emails_with_5plus_failed, 0) AS emails_with_5plus_failed_in_month, COALESCE(aa.audit_access_denied_events, 0) AS audit_access_denied_events FROM months m LEFT JOIN login_agg la ON la.month_start = m.month_start LEFT JOIN brute_force bf ON bf.month_start = m.month_start LEFT JOIN audit_agg aa ON aa.month_start = m.month_start ORDER BY m.month_start NULLS LAST, m.month_label NULLS LAST
The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted
Report request from Claire Dubois (analyst for the Head of Risk and Payments): Title: Failed login attempts and suspicious access patterns by month Question: How much login activity fails, is there visible evidence of brute-force-style patterns, and are access-denied events actually being captured as suspicious activity over time? Grain: one row per calendar month (occurred_at), platform-wide, from 2023-01 through 2025-12 Columns: - month (YYYY-MM) - total_login_attempts (from login_attempts) - failed_login_attempts (succeeded = false) - pct_failed - emails_with_5plus_failed_in_month (distinct email values with >=5 failed login_attempts rows in the month — a brute-force proxy) - audit_access_denied_events (count from audit_log UNION audit_log_archive where action = 'access_denied', same month) Details: Sources: login_attempts for the login outcome series; audit_log + audit_log_archive (action='access_denied') as a separate parallel signal of flagged suspicious access, same month grain, not joined row-by-row. Purpose: give Aylin real evidence that failed logins and suspicious access attempts are visible in the data (not just policy), and whether volumes/patterns look stable or show spikes worth a follow-up. Order by month ascending. Check your query with the warehouse's dry_run tool, then hand it in with submit.
dry_runWITH months AS ( SELECT DISTINCT month_label, month_start FROM calendar_days WHERE calendar_date BETWEEN DATE '2023-01-01' AND DATE '2025-12-31' ), login_month AS ( SELECT date_trunc('month', occurred_at)::date AS month_start, email, succeeded FROM login_attempts ), login_agg AS ( SELECT month_start, COUNT(*) AS total_login_attempts, COUNT(*) FILTER (WHERE succeeded = false) AS failed_login_attempts FROM login_month GROUP BY month_start ), brute_force AS ( SELECT month_start, COUNT(DISTINCT email) AS emails_with_5plus_failed FROM ( SELECT month_start, email, COUNT(*) AS cnt FROM login_month WHERE succeeded = false AND email IS NOT NULL GROUP BY month_start, email HAVING COUNT(*) >= 5 ) t GROUP BY month_start ), audit_combined AS ( SELECT occurred_at, action FROM audit_log WHERE action = 'access_denied' UNION ALL SELECT occurred_at, action FROM audit_log_archive WHERE action = 'access_denied' ), audit_agg AS ( SELECT date_trunc('month', occurred_at)::date AS month_start, COUNT(*) AS audit_access_denied_events FROM audit_combined GROUP BY month_start ) SELECT m.month_label AS month, COALESCE(la.total_login_attempts, 0) AS total_login_attempts, COALESCE(la.failed_login_attempts, 0) AS failed_login_attempts, ROUND(CASE WHEN COALESCE(la.total_login_attempts,0) = 0 THEN 0 ELSE 100.0 * la.failed_login_attempts / la.total_login_attempts END, 2) AS pct_failed, COALESCE(bf.emails_with_5plus_failed, 0) AS emails_with_5plus_failed_in_month, COALESCE(aa.audit_access_denied_events, 0) AS audit_access_denied_events FROM months m LEFT JOIN login_agg la ON la.month_start = m.month_start LEFT JOIN brute_force bf ON bf.month_start = m.month_start LEFT JOIN audit_agg aa ON aa.month_start = m.month_start ORDER BY m.month_start
STATUS: COMPILES (dry run on the empty schema, no data read) COLUMNS: month, total_login_attempts, failed_login_attempts, pct_failed, emails_with_5plus_failed_in_month, audit_access_denied_events
submitWITH months AS ( SELECT DISTINCT month_label, month_start FROM calendar_days WHERE calendar_date BETWEEN DATE '2023-01-01' AND DATE '2025-12-31' ), login_month AS ( SELECT date_trunc('month', occurred_at)::date AS month_start, email, succeeded FROM login_attempts ), login_agg AS ( SELECT month_start, COUNT(*) AS total_login_attempts, COUNT(*) FILTER (WHERE succeeded = false) AS failed_login_attempts FROM login_month GROUP BY month_start ), brute_force AS ( SELECT month_start, COUNT(DISTINCT email) AS emails_with_5plus_failed FROM ( SELECT month_start, email, COUNT(*) AS cnt FROM login_month WHERE succeeded = false AND email IS NOT NULL GROUP BY month_start, email HAVING COUNT(*) >= 5 ) t GROUP BY month_start ), audit_combined AS ( SELECT occurred_at, action FROM audit_log WHERE action = 'access_denied' UNION ALL SELECT occurred_at, action FROM audit_log_archive WHERE action = 'access_denied' ), audit_agg AS ( SELECT date_trunc('month', occurred_at)::date AS month_start, COUNT(*) AS audit_access_denied_events FROM audit_combined GROUP BY month_start ) SELECT m.month_label AS month, COALESCE(la.total_login_attempts, 0) AS total_login_attempts, COALESCE(la.failed_login_attempts, 0) AS failed_login_attempts, ROUND(CASE WHEN COALESCE(la.total_login_attempts,0) = 0 THEN 0 ELSE 100.0 * la.failed_login_attempts / la.total_login_attempts END, 2) AS pct_failed, COALESCE(bf.emails_with_5plus_failed, 0) AS emails_with_5plus_failed_in_month, COALESCE(aa.audit_access_denied_events, 0) AS audit_access_denied_events FROM months m LEFT JOIN login_agg la ON la.month_start = m.month_start LEFT JOIN brute_force bf ON bf.month_start = m.month_start LEFT JOIN audit_agg aa ON aa.month_start = m.month_start ORDER BY m.month_start
Submitted to tonight's batch. Your work on this request is done.