=== Report 1: Merchants Affected by the Sync-Webhook Stuck-Retry Pattern and Their Size (written by Grace) === Engineer's note: Dry runs never expose data, so I could not inspect the exact distinct webhooks.name values; I matched any webhook whose name contains 'erp'+'sync', 'wms'+'sync', or 'accounting'+'sync' (case-insensitive) to capture the three sync types and close variants. Tenants with a qualifying webhook but no 2025 deliveries, orders, or EUR rate match show 0/NULL rather than being dropped; revenue_rank is NULL if the tenant had no 2025 orders. No row limit was applied since the analyst asked for all qualifying tenants. REPORT: Merchants Affected by the Sync-Webhook Stuck-Retry Pattern and Their Size QUESTION: How many merchants have ERP/WMS/Accounting sync webhooks stuck in retry, and is this concentrated in our biggest accounts or spread across the long tail? tenant_name | total_sync_deliveries_2025 | success_rate_pct | pending_or_retrying_rate_pct | total_orders_2025 | total_revenue_eur_2025 | revenue_rank ------------------------------+----------------------------+------------------+------------------------------+-------------------+------------------------+------------- Teapot Society | 2 | 50.0 | 50.0 | 1 | 78.81 | 112 Kaffeekontor Nord | 13 | 61.54 | 38.46 | 8 | 405.82 | 105 Hautnah | 120 | 80.0 | 20.0 | 92 | 9190.62 | 51 Lumen Collective | 89 | 82.02 | 17.98 | 52 | 6391.23 | 62 Hanse Industriebedarf | 97 | 83.51 | 16.49 | 72 | 113320.38 | 13 Seilschaft | 25 | 84.0 | 16.0 | 17 | 6759.28 | 60 Thistlecombe | 140 | 84.29 | 15.71 | 92 | 12570.49 | 44 Folio & Fern | 58 | 86.21 | 13.79 | 46 | 2148.33 | 88 Brennquell Kaffee | 14149 | 88.87 | 11.12 | 4195 | 114554.65 | 11 Ampfeld Elektronik | 1583 | 89.83 | 10.17 | 449 | 512006.37 | 4 Linden & Loom | 614 | 91.21 | 8.79 | 263 | 38475.39 | 20 Wohnglück | 341 | 91.79 | 8.21 | 151 | 22859.8 | 32 Kibble Crate | 19 | 94.74 | 5.26 | 18 | 1211.91 | 95 Cragsley Outfitters | 14104 | 95.04 | 4.96 | 3635 | 724835.93 | 2 Basecamp Nord | 49 | 95.92 | 4.08 | 43 | 10657.51 | 46 Purrfect Pantry | 81 | 96.3 | 3.7 | 74 | 3583.86 | 74 Kettle & Key | 68 | 97.06 | 2.94 | 60 | 7581.57 | 54 Velo Noord | 39 | 97.44 | 2.56 | 31 | 9780.02 | 49 Noordzee Surfwear | 88 | 97.73 | 2.27 | 65 | 10894.01 | 45 Wiesgrund Wohnen | 642 | 97.98 | 2.02 | 296 | 38433.09 | 21 Sorella Skin | 105 | 98.1 | 1.9 | 99 | 10495.53 | 47 Mirelle | 68 | 98.53 | 1.47 | 63 | 2920.13 | 78 Studio Vrij | 2396 | 98.62 | 1.38 | 631 | 113540.14 | 12 Gratlinie Outdoor | 291 | 98.63 | 1.37 | 232 | 96277.59 | 14 Marrowby & Vane | 6425 | 98.63 | 1.37 | 1758 | 288909.93 | 8 Kibblewick | 1349 | 98.74 | 1.26 | 672 | 23801.41 | 30 Fjordline Wear | 81 | 98.77 | 1.23 | 57 | 8615.11 | 52 Nordic Bloom Cosmetics | 163 | 98.77 | 1.23 | 151 | 19074.7 | 36 Ostwind Mode | 162 | 98.77 | 1.23 | 122 | 21621.0 | 33 Atelier Nouveau | 87 | 98.85 | 1.15 | 64 | 9406.81 | 50 Grind Collective | 1136 | 98.94 | 1.06 | 1111 | 25610.52 | 27 Kontor24 Bürobedarf | 534 | 99.06 | 0.94 | 264 | 840409.53 | 1 Chocolaterie Mireille | 108 | 99.07 | 0.93 | 104 | 4566.99 | 67 Spice Route Box | 796 | 99.12 | 0.88 | 780 | 17113.71 | 39 Ballycorran Industrial Supply | 380 | 99.21 | 0.79 | 344 | 559149.73 | 3 Cinder Peak Sports Co. | 748 | 99.33 | 0.67 | 338 | 124756.84 | 10 Coastline Interiors | 179 | 99.44 | 0.56 | 155 | 21468.31 | 34 Fennel & Flint | 814 | 99.51 | 0.49 | 389 | 65764.13 | 16 OfficeHub Nordic | 350 | 99.71 | 0.29 | 170 | 428472.93 | 5 Petalcourt Beauty | 775 | 99.74 | 0.26 | 706 | 63508.22 | 17 Bijoux Lune | 23 | 100.0 | 0.0 | 21 | 3794.91 | 72 Copper Lane | 8 | 100.0 | 0.0 | 6 | 1003.56 | 98 Courtside Collective | 23 | 100.0 | 0.0 | 20 | 8514.48 | 53 Dewdrop Studio | 5 | 100.0 | 0.0 | 5 | 271.34 | 108 Fern & Fable | 48 | 100.0 | 0.0 | 42 | 5564.1 | 64 Glasshouse Grooming | 10 | 100.0 | 0.0 | 10 | 1455.2 | 93 Northstar Watches | 14 | 100.0 | 0.0 | 11 | 2655.08 | 85 Olive & Oak Pantry | 1 | 100.0 | 0.0 | 1 | 118.55 | 111 Ondine Botanicals | 27 | 100.0 | 0.0 | 25 | 1694.77 | 90 Papeterie Grossiste | 20 | 100.0 | 0.0 | 18 | 41189.21 | 18 (56 rows, first 50 printed) === Report 2: 2025 Monthly Stuck-Retry Rate for Sync Webhooks (ERP/WMS/Accounting) (written by Bea) === Engineer's note: Platform-wide monthly rates for webhooks named 'ERP sync', 'WMS sync', 'ACCOUNTING sync' across all tenants, using webhook_deliveries.created_at for 2025. "Pending or retrying" combines the pending and retrying status codes (the only non-success/non-failed states in this table). REPORT: 2025 Monthly Stuck-Retry Rate for Sync Webhooks (ERP/WMS/Accounting) QUESTION: Has the ~15% stuck-in-retry rate on sync webhooks been happening all year, or did it start/worsen recently? month | total_sync_deliveries | success_rate_pct | failed_rate_pct | pending_or_retrying_rate_pct --------------------+-----------------------+------------------+-----------------+----------------------------- 2025-01-01 00:00:00 | 4438 | 94.21 | 0.02 | 5.77 2025-02-01 00:00:00 | 3088 | 94.3 | 0.0 | 5.7 2025-03-01 00:00:00 | 3551 | 93.41 | 0.0 | 6.59 2025-04-01 00:00:00 | 3434 | 94.09 | 0.0 | 5.91 2025-05-01 00:00:00 | 3810 | 93.99 | 0.0 | 6.01 2025-06-01 00:00:00 | 3413 | 94.67 | 0.0 | 5.33 2025-07-01 00:00:00 | 3702 | 94.41 | 0.0 | 5.59 2025-08-01 00:00:00 | 3308 | 94.77 | 0.0 | 5.23 2025-09-01 00:00:00 | 3724 | 94.04 | 0.0 | 5.96 2025-10-01 00:00:00 | 4110 | 94.72 | 0.0 | 5.28 2025-11-01 00:00:00 | 5720 | 93.41 | 0.02 | 6.57 2025-12-01 00:00:00 | 7454 | 95.26 | 0.0 | 4.74 (12 rows) === Report 3: Carts.remind Failure Root Cause: Error Messages (written by Omar) === Engineer's note: One row per failed carts.remind job run in Nov 2025, with tenant, timing and error_message/attempt to spot shared vs. scattered root causes; expect 20 rows. REPORT: Carts.remind Failure Root Cause: Error Messages QUESTION: Did the 20 carts.remind failures in Nov 2025 share a common root cause (pointing to one shared bug/config issue) or look like scattered, tenant-specific problems (pointing to load/capacity)? tenant_name | queued_at | started_at | finished_at | error_message | attempt -----------------------+-------------------------+-------------------------+-------------------------+-----------------------------------------+-------- Gratlinie Outdoor | 2025-11-28 14:40:04.027 | 2025-11-28 14:40:05.004 | 2025-11-28 14:40:06.074 | 503 Service Unavailable: storefront API | 3 Marrowby & Vane | 2025-11-28 14:40:05.741 | 2025-11-28 14:40:10.093 | 2025-11-28 14:40:12.292 | 503 Service Unavailable: storefront API | 3 Cinder Peak Sports Co. | 2025-11-28 14:40:06.352 | 2025-11-28 14:40:06.776 | 2025-11-28 14:40:08.558 | 503 Service Unavailable: storefront API | 3 Petalcourt Beauty | 2025-11-28 14:40:06.878 | 2025-11-28 14:40:08.079 | 2025-11-28 14:40:09.171 | 503 Service Unavailable: storefront API | 3 Brennquell Kaffee | 2025-11-28 14:40:06.939 | 2025-11-28 14:40:07.282 | 2025-11-28 14:40:08.716 | 503 Service Unavailable: storefront API | 3 Kibblewick | 2025-11-28 14:40:07.143 | 2025-11-28 14:40:07.534 | 2025-11-28 14:40:08.362 | 503 Service Unavailable: storefront API | 3 Fennel & Flint | 2025-11-28 14:40:08.753 | 2025-11-28 14:40:12.312 | 2025-11-28 14:40:13.509 | 503 Service Unavailable: storefront API | 3 Studio Vrij | 2025-11-28 14:40:09.151 | 2025-11-28 14:40:12.35 | 2025-11-28 14:40:14.566 | 503 Service Unavailable: storefront API | 3 Ampfeld Elektronik | 2025-11-28 14:40:09.362 | 2025-11-28 14:40:11.517 | 2025-11-28 14:40:12.429 | 503 Service Unavailable: storefront API | 3 Cragsley Outfitters | 2025-11-28 14:40:09.758 | 2025-11-28 14:40:11.022 | 2025-11-28 14:40:13.348 | 503 Service Unavailable: storefront API | 3 Gratlinie Outdoor | 2025-11-28 15:40:03.673 | 2025-11-28 15:40:06.151 | 2025-11-28 15:40:07.4 | 503 Service Unavailable: storefront API | 3 Kibblewick | 2025-11-28 15:40:05.175 | 2025-11-28 15:40:07.89 | 2025-11-28 15:40:11.728 | 503 Service Unavailable: storefront API | 3 Cragsley Outfitters | 2025-11-28 15:40:05.504 | 2025-11-28 15:40:06.246 | 2025-11-28 15:40:09.483 | 503 Service Unavailable: storefront API | 3 Fennel & Flint | 2025-11-28 15:40:05.605 | 2025-11-28 15:40:10.525 | 2025-11-28 15:40:11.663 | 503 Service Unavailable: storefront API | 3 Studio Vrij | 2025-11-28 15:40:05.75 | 2025-11-28 15:40:06.179 | 2025-11-28 15:40:07.151 | 503 Service Unavailable: storefront API | 3 Ampfeld Elektronik | 2025-11-28 15:40:05.814 | 2025-11-28 15:40:06.925 | 2025-11-28 15:40:08.173 | 503 Service Unavailable: storefront API | 3 Marrowby & Vane | 2025-11-28 15:40:06.447 | 2025-11-28 15:40:10.262 | 2025-11-28 15:40:11.83 | 503 Service Unavailable: storefront API | 3 Cinder Peak Sports Co. | 2025-11-28 15:40:07.655 | 2025-11-28 15:40:08.369 | 2025-11-28 15:40:09.655 | 503 Service Unavailable: storefront API | 3 Petalcourt Beauty | 2025-11-28 15:40:09.473 | 2025-11-28 15:40:14.407 | 2025-11-28 15:40:15.307 | 503 Service Unavailable: storefront API | 3 Brennquell Kaffee | 2025-11-28 15:40:10.038 | 2025-11-28 15:40:11.938 | 2025-11-28 15:40:15.191 | 503 Service Unavailable: storefront API | 3 (20 rows)