We simulated an e-commerce platform, Tidewick Commerce, with AI agents role-playing
its people: the CEO, data analysts, engineers and more. This report shows how fast databases answer
the queries they asked.
Scale factor
Machines
Figure 1.Up and to the left is better.
Queries per second against what it costs to run
all 409 queries.
Each system on its most efficient configuration; in grey, its other configurations on the Pareto front (dashed).Each system on every configuration, joined by a line.Each system on its most efficient configuration; in grey, its other configurations on the Pareto front (dashed).Each system on every configuration, joined by a line.Each system on its most efficient configuration; in grey, its other configurations on the Pareto front (dashed).Each system on every configuration, joined by a line.Each system on its most efficient configuration; in grey, its other configurations on the Pareto front (dashed).Each system on every configuration, joined by a line.
Pricing based on AWS.
Timeouts and errors are penalized.
1Performance
System
Slowdown, relative to the fastest
QPS
Cost
Correct
Load
Table 1. Systems ranked by the geometric mean, over all 409 queries, of each query's time relative to the
fastest correct system (plus 10 ms), the bar on a log scale. A timed-out query counts as taking the full timeout
(5 s at SF1 and SF10, 10 s at SF100, 30 s at SF1000); a failed or wrong one as taking twice the slowest correct time of that query.
QPS (queries per second) and Cost as in Figure 1. Correct is out of 409 after the last load. Load is the
total of all five loads.
There are no reference results at this scale factor yet: every result that came back counts
as correct, without being checked.
Figure 1 and Table 1 can rank systems differently. Figure 1 rewards robustness:
being consistently fast and having no unusually slow queries. Table 1 rewards all-round speed: being close to the
fastest on every query. The difference comes from how each combines the queries: Figure 1 adds up their times,
Table 1 multiplies their factors. Take a system that runs two queries in
100 ms and 1000 ms. Making either one twice as fast, to 50 ms and 1000 ms or to
100 ms and 500 ms, improves Table 1's geometric mean about equally. Only the second saves real time in
Figure 1's sum: 500 ms instead of 50 ms.
2Queries
Order queries by
Figure 2. One row per system, one column per query, colored by the time relative to the fastest
correct system. Select a column to see the query.
3Methodology
The data
Tidewick Commerce in 251 tables, at scale factors 1, 10, 100 and 1000: from 6.1 million to 4.1 billion
rows.
The run
A fresh system loads everything through 2024, then 2025 one quarter at a time. After every load it answers
all 409 queries once, and it is never restarted. The figures show the run after the last load.
Correctness
After the last load, every result is compared by value with a reference result. Integers must match exactly;
other numbers may differ by at most 0.0001 + 0.0001 × |expected|. A wrong result counts as a
failed query.
Failures
A query still running after its timeout (5 s at SF1 and SF10, 10 s at SF100, 30 s at SF1000) is aborted. In every measure of speed, a timed-out query
counts as taking the full timeout, and a failed or wrong one as taking twice the slowest correct time of that
query.
Figure 1
Queries per second is 409 over the time all queries take together after the last load. The cost is
that time at the machine's price per hour (methodology, Machines).
The full methodology: machines, systems, isolation, how
results are checked, and how to run it yourself.