Queries · q132 · VP Platform Engineering · week 1, night 3
“How reliable were webhook deliveries to merchants' own systems across 2025, and what time window does the data actually cover?”
Fictional people at a fictional company, each played by a language-model agent. Who they are and what they are told
Week 1; shared with Samuel Whitcombe, Head of Merchant Success
Leadership wants a review of platform reliability in 2025: failed and slow background jobs, health check failures, rate limiting, webhook delivery problems, and which merchants felt them.
The manager, at the start of week 1
Subject: 2025 platform reliability review — what I need by Friday
Lucas,
Leadership wants a year-in-review on platform reliability for 2025, and I'm leading it jointly with Samuel Whitcombe (Merchant Success). The deliverable is a memo that tells leadership how reliable the platform actually was last year, where it hurt merchants most, and what that implies for our 2026 investment priorities (reliability engineering headcount, which integrations/carriers to fix or replace, whether we need stricter rate limiting or SLAs). Samuel's side will map our findings to specific merchant relationships and churn risk, so where possible, surface things in a way that can be tied back to individual merchants rather than only platform-wide averages.
Given you only get a handful of report slots a night, here's what matters most to me, roughly in priority order:
If you have to cut scope to fit your nightly slots, protect #1 and #2 — those are what the memo absolutely must answer. The rest can wait or be addressed at a higher level if time runs short.
Let me know if anything above is ambiguous before you spend a slot on it.
Diego
Subject: Platform reliability review — what I need from my side
Hi Lucas,
As you pull together the 2025 reliability picture, here's what matters from the merchant-health side:
Happy to flag specific merchant names if that helps scope the analysis.
Best, Samuel
The analyst · week 1, night 3, request 3 of 3
Night 2 reframed the whole job-reliability story: queue delay and run duration are flat and tiny every single week (no "lateness" problem exists at all), and ALL 20 failures in the entire Oct-Dec window happened in one week (2025-11-24) in one handler (carts.remind), hitting exactly 10 tenants. So this isn't a chronic reliability problem - it's one contained one-week incident tied to the abandoned-cart-reminder job. Report 3 showed something genuinely useful: the health-check unhealthy rate spiked (2.44% vs a ~1% baseline) in that exact same week, suggesting monitoring did catch it. Given this, tonight I pivot to what Samuel needs about those 10 tenants specifically (are they top-tier/large accounts, did it track with order volume, did it coincide with ticket pain) since that's now answerable precisely rather than generically, and I use my third slot to open up webhook delivery reliability (Diego's #5), which I haven't touched yet and which is a more directly merchant-facing failure mode than internal jobs turned out to be.
The engineer · 1 dry run on the empty schema, then the submission
SELECT date_trunc('month', created_at)::date AS month, count(*) AS total_deliveries, count(*) FILTER (WHERE status = 'success') AS success_count, round(100.0 * count(*) FILTER (WHERE status = 'success') / count(*), 2) AS success_rate_pct, count(*) FILTER (WHERE status = 'failed') AS failed_count, round(100.0 * count(*) FILTER (WHERE status = 'failed') / count(*), 2) AS failure_rate_pct, round(avg(duration_ms) FILTER (WHERE duration_ms IS NOT NULL)) AS avg_duration_ms, count(DISTINCT webhook_id) AS distinct_webhooks FROM webhook_deliveries WHERE created_at >= DATE '2025-01-01' AND created_at < DATE '2026-01-01' GROUP BY date_trunc('month', created_at) ORDER BY month ASC
“Query compiles and returns exactly the requested columns. Submitting.”
“Submitted. The report groups webhook_deliveries by calendar month of created_at for 2025, computing total/success/failed counts, success and failure rate percentages, average delivery duration, and distinct webhooks touched — one row per month, ordered ascending, with only months that actually have data appearing (so the row count itself reveals the true coverage window within 2025).”
As Lucas Moreau saw it, from the small dataset the agents worked on
month | total_deliveries | success_count | success_rate_pct | failed_count | failure_rate_pct | avg_duration_ms | distinct_webhooks -----------+------------------+---------------+------------------+--------------+------------------+-----------------+------------------ 2025-01-01 | 4438 | 4181 | 94.21 | 1 | 0.02 | 1329.0 | 61 2025-02-01 | 3088 | 2912 | 94.3 | 0 | 0.0 | 1355.0 | 60 2025-03-01 | 3551 | 3317 | 93.41 | 0 | 0.0 | 1531.0 | 60 2025-04-01 | 3434 | 3231 | 94.09 | 0 | 0.0 | 1422.0 | 63 2025-05-01 | 3810 | 3581 | 93.99 | 0 | 0.0 | 1443.0 | 63 2025-06-01 | 3413 | 3231 | 94.67 | 0 | 0.0 | 1233.0 | 63 2025-07-01 | 3702 | 3495 | 94.41 | 0 | 0.0 | 1285.0 | 65 2025-08-01 | 3308 | 3135 | 94.77 | 0 | 0.0 | 1185.0 | 65 2025-09-01 | 3724 | 3502 | 94.04 | 0 | 0.0 | 1429.0 | 65 2025-10-01 | 4110 | 3893 | 94.72 | 0 | 0.0 | 1267.0 | 68 2025-11-01 | 5720 | 5343 | 93.41 | 1 | 0.02 | 1332.0 | 70 2025-12-01 | 7454 | 7101 | 95.26 | 0 | 0.0 | 1181.0 | 71 (12 rows)
The memo and the queries that followed; links found by title, so inferred
Lucas Moreau's memo to Diego Morales at the end of the week:
6. Webhook deliveries are stable overall but a chronic ~15% "never resolves" rate sits in ERP/WMS/Accounting sync integrations. Platform-wide success rate held 93–95% all 12 months, no trend. Source: "2025 Webhook Delivery Reliability Trend." But the worst 20 individual endpoints are dominated by ERP/WMS/Accounting sync webhooks stuck in pending/retrying 13–20% of the time, at real volume (e.g., Brennquell Kaffee's accounting sync: ~770 stuck deliveries). Several of these tenants (Brennquell Kaffee, Cragsley Outfitters, Ampfeld Elektronik, Marrowby & Vane) also appear in the carts.remind-affected list. Source: "2025 Webhook Delivery Reliability by Endpoint." This looks like a bug in our own retry logic for sync job classes, not a third-party provider problem.
Later queries that build on this report:
Changed only where the benchmark needs the same result on every system
SELECT date_trunc('month', created_at)::date AS month, CAST(date_trunc('month', created_at) AS date) AS month, count(*) AS total_deliveries, count(*) FILTER (WHERE status = 'success') AS success_count, round(100.0 * count(*) FILTER (WHERE status = 'success') / count(*), 2) AS success_rate_pct, 100.0 * count(*) FILTER (WHERE status = 'success') / count(*) AS success_rate_pct, count(*) FILTER (WHERE status = 'failed') AS failed_count, round(100.0 * count(*) FILTER (WHERE status = 'failed') / count(*), 2) AS failure_rate_pct, round(avg(duration_ms) FILTER (WHERE duration_ms IS NOT NULL)) AS avg_duration_ms, 100.0 * count(*) FILTER (WHERE status = 'failed') / count(*) AS failure_rate_pct, avg(cast(duration_ms as numeric(18,6))) FILTER (WHERE duration_ms IS NOT NULL) AS avg_duration_ms, count(DISTINCT webhook_id) AS distinct_webhooksFROM webhook_deliveriesWHERE created_at >= DATE '2025-01-01' AND created_at < DATE '2026-01-01'GROUP BY date_trunc('month', created_at)ORDER BY month ASCORDER BY month ASC NULLS LAST
SELECT CAST(date_trunc('month', created_at) AS date) AS month, count(*) AS total_deliveries, count(*) FILTER (WHERE status = 'success') AS success_count, 100.0 * count(*) FILTER (WHERE status = 'success') / count(*) AS success_rate_pct, count(*) FILTER (WHERE status = 'failed') AS failed_count, 100.0 * count(*) FILTER (WHERE status = 'failed') / count(*) AS failure_rate_pct, avg(cast(duration_ms as numeric(18,6))) FILTER (WHERE duration_ms IS NOT NULL) AS avg_duration_ms, count(DISTINCT webhook_id) AS distinct_webhooks FROM webhook_deliveries WHERE created_at >= DATE '2025-01-01' AND created_at < DATE '2026-01-01' GROUP BY date_trunc('month', created_at) ORDER BY month ASC NULLS LAST
The agents' own records, as the role play wrote them; only the personal data of the person who ran it is redacted
Report request from Lucas Moreau (analyst for the VP Platform Engineering): Title: 2025 Webhook Delivery Reliability Trend Question: How reliable were webhook deliveries to merchants' own systems across 2025, and what time window does the data actually cover? Grain: one row per calendar month Columns: - month - total_deliveries - success_count - success_rate_pct - failed_count - failure_rate_pct - avg_duration_ms - distinct_webhooks Details: Table webhook_deliveries. Filter created_at >= 2025-01-01 and < 2026-01-01 (requesting the full year; per the pattern seen with job_runs and system_health_checks, report back whatever months actually have rows rather than erroring). Group by month of created_at. success_rate_pct = success_count/total_deliveries*100 rounded 2dp (success_count = status='success'); failure_rate_pct similarly for status='failed'. avg_duration_ms over non-null duration_ms, rounded to whole number. distinct_webhooks = count distinct webhook_id. Platform-wide, all tenants. Order by month ascending. Check your query with the warehouse's dry_run tool, then hand it in with submit.
dry_runSELECT date_trunc('month', created_at)::date AS month, count(*) AS total_deliveries, count(*) FILTER (WHERE status = 'success') AS success_count, round(100.0 * count(*) FILTER (WHERE status = 'success') / count(*), 2) AS success_rate_pct, count(*) FILTER (WHERE status = 'failed') AS failed_count, round(100.0 * count(*) FILTER (WHERE status = 'failed') / count(*), 2) AS failure_rate_pct, round(avg(duration_ms) FILTER (WHERE duration_ms IS NOT NULL)) AS avg_duration_ms, count(DISTINCT webhook_id) AS distinct_webhooks FROM webhook_deliveries WHERE created_at >= DATE '2025-01-01' AND created_at < DATE '2026-01-01' GROUP BY date_trunc('month', created_at) ORDER BY month ASC
STATUS: COMPILES (dry run on the empty schema, no data read) COLUMNS: month, total_deliveries, success_count, success_rate_pct, failed_count, failure_rate_pct, avg_duration_ms, distinct_webhooks
submitSELECT date_trunc('month', created_at)::date AS month, count(*) AS total_deliveries, count(*) FILTER (WHERE status = 'success') AS success_count, round(100.0 * count(*) FILTER (WHERE status = 'success') / count(*), 2) AS success_rate_pct, count(*) FILTER (WHERE status = 'failed') AS failed_count, round(100.0 * count(*) FILTER (WHERE status = 'failed') / count(*), 2) AS failure_rate_pct, round(avg(duration_ms) FILTER (WHERE duration_ms IS NOT NULL)) AS avg_duration_ms, count(DISTINCT webhook_id) AS distinct_webhooks FROM webhook_deliveries WHERE created_at >= DATE '2025-01-01' AND created_at < DATE '2026-01-01' GROUP BY date_trunc('month', created_at) ORDER BY month ASC
Submitted to tonight's batch. Your work on this request is done.