DB Olympics

Queries · q279 · Head of Risk and Payments · week 2, night 2

February 2025 repeat-failure accounts: did they succeed afterward? (corrected)

“Among the accounts with 5+ failed logins in February 2025, how many went on to log in successfully afterward, and how soon — i.e. was this a near-miss or just noise?”

Manager · asked the questionAylin DemirHead of Risk and Payments. Previously in fraud operations at a bank. Responsible for payment performance and for keeping bad actors out.
Analyst · planned the reportClaire DuboisTime-series minded: weekly and daily trends, before and after, seasonality.
Engineer · wrote the SQLInesThinks in window functions: ranks, running totals, LAG/LEAD for comparisons, percent-of-total.

Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told

  1. A business event lands on the Head of Risk and Payments's desk

    Week 1; shared with Rosa Van den Berg, Head of Data

    An enterprise prospect sent a security and compliance questionnaire covering account security, access control, privacy requests, data retention and audit trails. Sales wants answers backed by evidence from our own systems, not by policy documents.

  2. Aylin Demir writes to Claire Dubois

    The manager, at the start of week 2

    Subject: Re: Questionnaire memo — good work, a few threads to pull

    Claire,

    This is a strong memo, and the gaps you flagged are exactly the kind of thing I'd rather surface now than have a prospect's security team find later. A few things stand out and I want to understand them better before we finalize anything.

    The February 2025 login spike is the one that worries me most. A 10x jump in failures with no corresponding rise in flagged suspicious activity is a real detection gap, not just a reporting curiosity — if that was a credential-stuffing attempt, I need to know whether it actually got anywhere.

    Questions for this week:

    1. On the February 2025 spike — did it cluster around a handful of accounts, merchants, or sources, or was it spread evenly? And did any of the accounts that failed repeatedly go on to log in successfully shortly after? That tells us whether this was noise or an actual near-miss.
    2. On the three unresolved rejection dates — rather than retrying the same technical path, can we establish, using whatever reliable timestamp we do have, roughly when each decision was made relative to the request date? I don't need the exact figure if it's not retrievable — I need to know whether we were plausibly within the deadline or not, so we can decide how to word that answer.
    3. On flagged incidents (the one we deferred) — now that we're another week in, I want this one: how many incidents have been flagged, how severe, how long they typically take to close, and whether anything has sat open a long time. This is likely to come up in the same breath as the login spike.
    4. On the tables with no retention policy — before I escalate this as a gap, I want to know how exposed we actually are: roughly how far back does that data go today? If it's a few months, this is a minor cleanup; if it's three years, it's a bigger conversation with Rosa.

    Keep the role/permission logging gap and the broken query as noted — no need to keep spending slots chasing those; I'll raise the first with engineering myself and we can note the second as a known limitation in the memo.

    Thanks, Aylin

  3. Claire Dubois plans the night

    The analyst · week 2, night 2, request 1 of 3

    Night 1 results need a careful read before I pass anything to Aylin:

    Report 1 (concentration by merchant) is clean and striking: 3 merchants — Cragsley Outfitters (4,407 failed, 88.1% fail rate), Marrowby & Vane (1,978 failed, 86.1%), Studio Vrij (783 failed, 84.2%) — account for ~7,168 of the ~7,429 failed attempts we found last week (96.5%), while the next-highest merchant had just 17 failed attempts. These are the exact same three merchants Support Ops called out last week as having both the heaviest ticket volume and worst response times. But the pattern doesn't look like a credential-stuffing attack: distinct_emails_with_any_failed_attempt is nearly as high as total failed attempts (e.g. Cragsley: 2,786 distinct emails for 4,407 fails, max 8 fails on any single email, max 9 from any single IP, spread across 1,903 distinct IPs). That's wide, shallow failure across many accounts and many sources at once — much more consistent with a broken login flow/client bug affecting those 3 merchants specifically than with an external attack.

    Report 2 (near-miss outcomes) is internally broken and I will not pass these numbers on: percentages are computed against a fixed denominator of 102, but the bucket counts (42+12+89+452=595) don't come close to summing to 102, and the engineer's note confirms email was matched globally rather than scoped per tenant (login_attempts.email is only unique per tenant) — so the "cohort" silently grew across tenants mid-query while the percentage base didn't update. This is exactly the kind of actor/identity mismatch Rosa flagged in week 1, now showing up in a different table. I'm retrying tonight with tenant-scoped (tenant_id, email) pairs and mutually-exclusive buckets that must sum to the true cohort size.

    Report 3 (DSAR via updated_at) looks solid and resolves Aylin's question #2: all 3 rejected DSARs show updated_at well inside their 30-day due_at window (8.7, 3.8, and 23.7 days after receipt respectively) — plausibly within deadline, with the usual caveat that updated_at isn't a verified "decision" timestamp.

    Tonight (3 slots): (1) retry the near-miss report correctly; (2) incidents/compliance_events volume, severity, and resolution time (question #3, deferred from last week); (3) how far back the five retention-policy-less tables actually go, to size the exposure for question #4.

    February 2025 repeat-failure accounts: did they succeed afterward? (corrected)
    Question
    Among the accounts with 5+ failed logins in February 2025, how many went on to log in successfully afterward, and how soon — i.e. was this a near-miss or just noise?
    One row per
    outcome bucket, computed over the cohort of distinct (tenant_id, email) pairs with >=5 failed login_attempts rows in February 2025
    Columns
    • outcome_bucket (succeeded_within_1h / succeeded_within_24h / succeeded_later_in_month / never_succeeded_in_month)
    • pair_count
    • pct_of_cohort
    • avg_failed_attempts_before_success (blank for never_succeeded)
    • distinct_tenants_affected
  4. Ines writes the SQL

    The engineer · 1 dry run on the empty schema, then the submission

    Dry run 1 · STATUS: COMPILES (dry run on the empty schema, no data read)
    WITH failed_feb AS (
      SELECT tenant_id, email, occurred_at
      FROM login_attempts
      WHERE succeeded = false
        AND occurred_at >= DATE '2025-02-01'
        AND occurred_at < DATE '2025-03-01'
    ),
    cohort AS (
      SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at
      FROM failed_feb
      GROUP BY tenant_id, email
      HAVING COUNT(*) >= 5
    ),
    success_after AS (
      SELECT
        c.tenant_id,
        c.email,
        c.failed_count,
        c.last_failed_at,
        MIN(la.occurred_at) AS success_at
      FROM cohort c
      LEFT JOIN login_attempts la
        ON la.tenant_id = c.tenant_id
        AND la.email = c.email
        AND la.succeeded = true
        AND la.occurred_at > c.last_failed_at
        AND la.occurred_at < DATE '2025-03-01'
      GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at
    ),
    classified AS (
      SELECT
        tenant_id,
        email,
        failed_count,
        CASE
          WHEN success_at IS NULL THEN 'never_succeeded_in_month'
          WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h'
          WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h'
          ELSE 'succeeded_later_in_month'
        END AS outcome_bucket
      FROM success_after
    ),
    cohort_size AS (
      SELECT COUNT(*) AS n FROM cohort
    ),
    bucket_list AS (
      SELECT * FROM (VALUES
        ('succeeded_within_1h', 1),
        ('succeeded_within_24h', 2),
        ('succeeded_later_in_month', 3),
        ('never_succeeded_in_month', 4)
      ) AS b(outcome_bucket, sort_order)
    )
    SELECT
      b.outcome_bucket,
      COUNT(c.tenant_id) AS pair_count,
      ROUND(100.0 * COUNT(c.tenant_id) / cs.n, 1) AS pct_of_cohort,
      CASE
        WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL
        ELSE ROUND(AVG(c.failed_count), 2)
      END AS avg_failed_attempts_before_success,
      COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affected
    FROM bucket_list b
    LEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucket
    CROSS JOIN cohort_size cs
    GROUP BY b.outcome_bucket, b.sort_order, cs.n
    ORDER BY b.sort_order

    “Query compiles with the expected columns. Submitting.”

    Submitted · the SQL of the last dry run, unchanged · Submitted to tonight's batch. Your work on this request is done.

    “Submitted. Summary of the approach: - Cohort: distinct (tenant_id, email) pairs with ≥5 succeeded=false rows in Feb 2025, keyed correctly by tenant+email (not email alone) per the correction. - Matching success: for each pair, the first succeeded=true row for that same (tenant_id, email) strictly after the pair's last Feb failure, still within February. - Buckets: classified by elapsed time (≤1h, ≤24h, else later-in-month, else never), applied with the requested precedence, and always printed as all 4 rows (zero-filled) so pair_count sums exactly to the cohort size and pct_of_cohort sums to 100%. - avg_failed_attempts_before_success: average of each pair's total Feb failed-attempt count (left blank for the never-succeeded bucket), since the matching success is always after the last failure, making every Feb failure "before success" by construction. - distinct_tenants_affected: count of distinct tenants contributing to each bucket.”

  5. The report comes back overnight

    As Claire Dubois saw it, from the small dataset the agents worked on

    outcome_bucket           | pair_count | pct_of_cohort | avg_failed_attempts_before_success | distinct_tenants_affected
    -------------------------+------------+---------------+------------------------------------+--------------------------
    succeeded_within_1h      |          6 |           6.1 |                                5.5 |                         3
    succeeded_within_24h     |          2 |           2.0 |                                5.0 |                         2
    succeeded_later_in_month |         12 |          12.2 |                                5.5 |                         2
    never_succeeded_in_month |         78 |          79.6 |                               NULL |                         3
    (4 rows)
  6. What came of it

    The memo and the queries that followed; links found by title, so inferred

    It builds on:

  7. It becomes benchmark query q279

    Changed only where the benchmark needs the same result on every system

    • numericsdocs/NUMERICS.md: rounding removed; results keep full precision
    • numericsdocs/NUMERICS.md: rounding removed; results keep full precision; AVG over an integer computed as a decimal: AVG(CAST(x AS NUMERIC(18,6)))
    • numericsdocs/NUMERICS.md: divisor guarded with NULLIF(..., 0); it is zero when only part of the data is loaded (an empty period), where engines raise an error or return inf/nan
    • made explicita selected t.col gets AS col, its standard name made explicit (5×)
    • determinismthe standard leaves the place of NULLs in a sort to the implementation: NULLS LAST ascending, NULLS FIRST descending, as in the reference results
    The benchmark text, against the submitted SQL
    WITH failed_feb AS (  SELECT tenant_id, email, occurred_at  FROM login_attempts  WHERE succeeded = false    AND occurred_at >= DATE '2025-02-01'    AND occurred_at < DATE '2025-03-01'),cohort AS (  SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at  FROM failed_feb  GROUP BY tenant_id, email  HAVING COUNT(*) >= 5),success_after AS (  SELECT    c.tenant_id,    c.email,    c.failed_count,    c.last_failed_at,    c.tenant_id AS tenant_id,    c.email AS email,    c.failed_count AS failed_count,    c.last_failed_at AS last_failed_at,    MIN(la.occurred_at) AS success_at  FROM cohort c  LEFT JOIN login_attempts la    ON la.tenant_id = c.tenant_id    AND la.email = c.email    AND la.succeeded = true    AND la.occurred_at > c.last_failed_at    AND la.occurred_at < DATE '2025-03-01'  GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at),classified AS (  SELECT    tenant_id,    email,    failed_count,    CASE      WHEN success_at IS NULL THEN 'never_succeeded_in_month'      WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h'      WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h'      ELSE 'succeeded_later_in_month'    END AS outcome_bucket  FROM success_after),cohort_size AS (  SELECT COUNT(*) AS n FROM cohort),bucket_list AS (  SELECT * FROM (VALUES    ('succeeded_within_1h', 1),    ('succeeded_within_24h', 2),    ('succeeded_later_in_month', 3),    ('never_succeeded_in_month', 4)  ) AS b(outcome_bucket, sort_order))SELECT  b.outcome_bucket,  b.outcome_bucket AS outcome_bucket,  COUNT(c.tenant_id) AS pair_count,  ROUND(100.0 * COUNT(c.tenant_id) / cs.n, 1) AS pct_of_cohort,  100.0 * COUNT(c.tenant_id) / NULLIF(cs.n, 0) AS pct_of_cohort,  CASE    WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL    ELSE ROUND(AVG(c.failed_count), 2)    ELSE AVG(CAST(c.failed_count AS NUMERIC(18,6)))  END AS avg_failed_attempts_before_success,  COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affectedFROM bucket_list bLEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucketCROSS JOIN cohort_size csGROUP BY b.outcome_bucket, b.sort_order, cs.nORDER BY b.sort_orderORDER BY b.sort_order NULLS LAST
    WITH failed_feb AS (
      SELECT tenant_id, email, occurred_at
      FROM login_attempts
      WHERE succeeded = false
        AND occurred_at >= DATE '2025-02-01'
        AND occurred_at < DATE '2025-03-01'
    ),
    cohort AS (
      SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at
      FROM failed_feb
      GROUP BY tenant_id, email
      HAVING COUNT(*) >= 5
    ),
    success_after AS (
      SELECT
        c.tenant_id AS tenant_id,
        c.email AS email,
        c.failed_count AS failed_count,
        c.last_failed_at AS last_failed_at,
        MIN(la.occurred_at) AS success_at
      FROM cohort c
      LEFT JOIN login_attempts la
        ON la.tenant_id = c.tenant_id
        AND la.email = c.email
        AND la.succeeded = true
        AND la.occurred_at > c.last_failed_at
        AND la.occurred_at < DATE '2025-03-01'
      GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at
    ),
    classified AS (
      SELECT
        tenant_id,
        email,
        failed_count,
        CASE
          WHEN success_at IS NULL THEN 'never_succeeded_in_month'
          WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h'
          WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h'
          ELSE 'succeeded_later_in_month'
        END AS outcome_bucket
      FROM success_after
    ),
    cohort_size AS (
      SELECT COUNT(*) AS n FROM cohort
    ),
    bucket_list AS (
      SELECT * FROM (VALUES
        ('succeeded_within_1h', 1),
        ('succeeded_within_24h', 2),
        ('succeeded_later_in_month', 3),
        ('never_succeeded_in_month', 4)
      ) AS b(outcome_bucket, sort_order)
    )
    SELECT
      b.outcome_bucket AS outcome_bucket,
      COUNT(c.tenant_id) AS pair_count,
      100.0 * COUNT(c.tenant_id) / NULLIF(cs.n, 0) AS pct_of_cohort,
      CASE
        WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL
        ELSE AVG(CAST(c.failed_count AS NUMERIC(18,6)))
      END AS avg_failed_attempts_before_success,
      COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affected
    FROM bucket_list b
    LEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucket
    CROSS JOIN cohort_size cs
    GROUP BY b.outcome_bucket, b.sort_order, cs.n
    ORDER BY b.sort_order NULLS LAST
    10 ms100 ms1 sHyper55.3 msDuckDB78.0 msGizmoSQL86.2 msclickhouse_brew304 msFirebolt292 msDataFusion571 msClickHouse358 ms
  8. Everything, as recorded

    The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted

    Ines's whole session, 7 turns
    • The request, as Ines received it
      Report request from Claire Dubois (analyst for the Head of Risk and Payments):
      
      Title: February 2025 repeat-failure accounts: did they succeed afterward? (corrected)
      Question: Among the accounts with 5+ failed logins in February 2025, how many went on to log in successfully afterward, and how soon — i.e. was this a near-miss or just noise?
      Grain: one row per outcome bucket, computed over the cohort of distinct (tenant_id, email) pairs with >=5 failed login_attempts rows in February 2025
      Columns:
      - outcome_bucket (succeeded_within_1h / succeeded_within_24h / succeeded_later_in_month / never_succeeded_in_month)
      - pair_count
      - pct_of_cohort
      - avg_failed_attempts_before_success (blank for never_succeeded)
      - distinct_tenants_affected
      Details: Fix for last night's broken version: scope the cohort to (tenant_id, email) pairs, not email alone, since login_attempts.email is only unique per tenant and matching on email alone silently conflated unrelated users across merchants. For each pair in the cohort, find the first succeeded=true login_attempts row for that same (tenant_id, email) occurring after that pair's last Feb-2025 failed attempt, within February 2025 only. Classify each pair into exactly one bucket using this precedence: succeeded_within_1h, else succeeded_within_24h, else succeeded_later_in_month, else never_succeeded_in_month. Buckets must be mutually exclusive and exhaustive: pair_count across all 4 rows must sum exactly to the cohort size, and pct_of_cohort must sum to 100%. Do not print individual emails. Order buckets in the sequence listed above.
      
      Check your query with the warehouse's dry_run tool, then hand it in with submit.
    • Ines calls dry_run
      WITH failed_feb AS (
        SELECT tenant_id, email, occurred_at
        FROM login_attempts
        WHERE succeeded = false
          AND occurred_at >= DATE '2025-02-01'
          AND occurred_at < DATE '2025-03-01'
      ),
      cohort AS (
        SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at
        FROM failed_feb
        GROUP BY tenant_id, email
        HAVING COUNT(*) >= 5
      ),
      success_after AS (
        SELECT
          c.tenant_id,
          c.email,
          c.failed_count,
          c.last_failed_at,
          MIN(la.occurred_at) AS success_at
        FROM cohort c
        LEFT JOIN login_attempts la
          ON la.tenant_id = c.tenant_id
          AND la.email = c.email
          AND la.succeeded = true
          AND la.occurred_at > c.last_failed_at
          AND la.occurred_at < DATE '2025-03-01'
        GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at
      ),
      classified AS (
        SELECT
          tenant_id,
          email,
          failed_count,
          CASE
            WHEN success_at IS NULL THEN 'never_succeeded_in_month'
            WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h'
            WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h'
            ELSE 'succeeded_later_in_month'
          END AS outcome_bucket
        FROM success_after
      ),
      cohort_size AS (
        SELECT COUNT(*) AS n FROM cohort
      ),
      bucket_list AS (
        SELECT * FROM (VALUES
          ('succeeded_within_1h', 1),
          ('succeeded_within_24h', 2),
          ('succeeded_later_in_month', 3),
          ('never_succeeded_in_month', 4)
        ) AS b(outcome_bucket, sort_order)
      )
      SELECT
        b.outcome_bucket,
        COUNT(c.tenant_id) AS pair_count,
        ROUND(100.0 * COUNT(c.tenant_id) / cs.n, 1) AS pct_of_cohort,
        CASE
          WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL
          ELSE ROUND(AVG(c.failed_count), 2)
        END AS avg_failed_attempts_before_success,
        COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affected
      FROM bucket_list b
      LEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucket
      CROSS JOIN cohort_size cs
      GROUP BY b.outcome_bucket, b.sort_order, cs.n
      ORDER BY b.sort_order
    • The warehouse answers
      STATUS: COMPILES (dry run on the empty schema, no data read)
      COLUMNS: outcome_bucket, pair_count, pct_of_cohort, avg_failed_attempts_before_success, distinct_tenants_affected
    • InesQuery compiles with the expected columns. Submitting.
    • Ines calls submit
      WITH failed_feb AS (
        SELECT tenant_id, email, occurred_at
        FROM login_attempts
        WHERE succeeded = false
          AND occurred_at >= DATE '2025-02-01'
          AND occurred_at < DATE '2025-03-01'
      ),
      cohort AS (
        SELECT tenant_id, email, COUNT(*) AS failed_count, MAX(occurred_at) AS last_failed_at
        FROM failed_feb
        GROUP BY tenant_id, email
        HAVING COUNT(*) >= 5
      ),
      success_after AS (
        SELECT
          c.tenant_id,
          c.email,
          c.failed_count,
          c.last_failed_at,
          MIN(la.occurred_at) AS success_at
        FROM cohort c
        LEFT JOIN login_attempts la
          ON la.tenant_id = c.tenant_id
          AND la.email = c.email
          AND la.succeeded = true
          AND la.occurred_at > c.last_failed_at
          AND la.occurred_at < DATE '2025-03-01'
        GROUP BY c.tenant_id, c.email, c.failed_count, c.last_failed_at
      ),
      classified AS (
        SELECT
          tenant_id,
          email,
          failed_count,
          CASE
            WHEN success_at IS NULL THEN 'never_succeeded_in_month'
            WHEN success_at <= last_failed_at + INTERVAL '1 hour' THEN 'succeeded_within_1h'
            WHEN success_at <= last_failed_at + INTERVAL '24 hour' THEN 'succeeded_within_24h'
            ELSE 'succeeded_later_in_month'
          END AS outcome_bucket
        FROM success_after
      ),
      cohort_size AS (
        SELECT COUNT(*) AS n FROM cohort
      ),
      bucket_list AS (
        SELECT * FROM (VALUES
          ('succeeded_within_1h', 1),
          ('succeeded_within_24h', 2),
          ('succeeded_later_in_month', 3),
          ('never_succeeded_in_month', 4)
        ) AS b(outcome_bucket, sort_order)
      )
      SELECT
        b.outcome_bucket,
        COUNT(c.tenant_id) AS pair_count,
        ROUND(100.0 * COUNT(c.tenant_id) / cs.n, 1) AS pct_of_cohort,
        CASE
          WHEN b.outcome_bucket = 'never_succeeded_in_month' THEN NULL
          ELSE ROUND(AVG(c.failed_count), 2)
        END AS avg_failed_attempts_before_success,
        COUNT(DISTINCT c.tenant_id) AS distinct_tenants_affected
      FROM bucket_list b
      LEFT JOIN classified c ON c.outcome_bucket = b.outcome_bucket
      CROSS JOIN cohort_size cs
      GROUP BY b.outcome_bucket, b.sort_order, cs.n
      ORDER BY b.sort_order
    • The warehouse answers
      Submitted to tonight's batch. Your work on this request is done.
    • InesSubmitted. Summary of the approach: - **Cohort**: distinct `(tenant_id, email)` pairs with ≥5 `succeeded=false` rows in Feb 2025, keyed correctly by tenant+email (not email alone) per the correction. - **Matching success**: for each pair, the first `succeeded=true` row for that same `(tenant_id, email)` strictly after the pair's last Feb failure, still within February. - **Buckets**: classified by elapsed time (≤1h, ≤24h, else later-in-month, else never), applied with the requested precedence, and always printed as all 4 rows (zero-filled) so `pair_count` sums exactly to the cohort size and `pct_of_cohort` sums to 100%. - **avg_failed_attempts_before_success**: average of each pair's total Feb failed-attempt count (left blank for the never-succeeded bucket), since the matching success is always after the last failure, making every Feb failure "before success" by construction. - **distinct_tenants_affected**: count of distinct tenants contributing to each bucket.