DB Olympics

Methodology

How a database system is run, what is measured, and how the numbers on the start page are computed.

Workload

The data is a deterministic simulation of Tidewick Commerce, a multi-tenant e-commerce platform: shops, orders, payments, shipping, support, marketing and more, in 251 tables. Shoppers, orders and events grow with the scale factor; the 120 merchants, their catalogs and the reference tables do not.

Scale factorRowsSize as Parquet
16.1 million0.2 GB
1045.5 million1.7 GB
100411 million15.5 GB
10004.1 billion151 GB

The data arrives over time. Every table is split into periods: everything through 2024 in one file, then one file per quarter of 2025.

The 409 queries were written by language-model agents role-playing Tidewick's managers, analysts and engineers over two simulated weeks. Each query can be traced to the conversation that led to it. They are standard SQL with a few placeholders where systems spell things differently (a year-month string, a difference of dates). Every query returns the same rows in the same order on every system; where needed, a tie-breaker was added.

Machines

Every system runs on every machine configuration of its scale factor below: pinned to that many hardware threads and limited to that much memory, everything included. The configurations are AWS c8a instances.

MachineCoresMemoryPer hourScale factors
c8a.medium12 GB$0.05391
c8a.large24 GB$0.10781
c8a.xlarge48 GB$0.21551, 10
c8a.2xlarge816 GB$0.43111, 10
c8a.4xlarge1632 GB$0.862210, 100
c8a.8xlarge3264 GB$1.724310, 100
c8a.12xlarge4896 GB$2.5865100, 1000
c8a.16xlarge64128 GB$3.4486100, 1000
c8a.24xlarge96192 GB$5.1730100, 1000

AWS instances are priced on demand, Linux, us-east-1.

All runs are on one server standing in for the instances: an AMD EPYC 9745 (Zen 5c, up to 3.7 GHz), 128 cores with 256 hardware threads, and 1007 GiB of DDR5 memory, under Linux 7.0.0-38-generic. A configuration gets the first hardware threads of the server and the memory of the instance. A vCPU of c8a is a physical core; its cores are Zen 5c, the cores of c8a’s generation made smaller and run at a lower clock, so we expect a core to be somewhat slower than a c8a vCPU. Every thread has a core of its own, as on c8a, and the smaller configurations likely get more of the cache and memory bandwidth than an instance of their size.

Systems

Every system gets the same schema and the same queries. A system may change type names and the spellings behind the placeholders, and use its own settings to follow the standard, nothing else; foreign keys are never declared, and tables have no keys. How to add a system.

SystemVersionHow it runs
DuckDB1.5.6embedded
Hyper0.0.26784Tableau's engine, its own server
GizmoSQLv1.42.0 (DuckDB 1.5.6)DuckDB behind an Arrow Flight SQL server
CedarDBv2026-10-08its own server
pg_duckdb1.1.0 (PostgreSQL 18.1)PostgreSQL tables, DuckDB runs the queries
PostgreSQL18.1its own server
DataFusion55.2.0embedded, queries the Parquet files in place
Firebolt5.0.0-pre.0.20261013125703its own server
ClickHouse26.9.22.3its own server
chDB26.9.2.1ClickHouse as a library, embedded
clickhouse_brewexperimental (ClickHouse 26.9.22.3)Umbra writes every query for ClickHouse, which runs it

Supporting different SQL dialects

The agents were told to write standard SQL wherever possible. Some queries still use constructs outside the standard, and many systems do not support every standard construct. Each query therefore exists in two forms: the benchmark query, one text for all systems, and each system's version of it, in which the placeholders of the benchmark query are spelled the way that system needs. The following criteria decide which change goes where. The standard is SQL:2023; what is common is judged by the four most used relational databases in the Stack Overflow Developer Survey 2026 (PostgreSQL, SQLite, MySQL and SQL Server), not by the systems in the benchmark.

  • A query uses something the standard does not have, or relies on what the standard leaves implementation-defined.
    • If enough of the four support another way to write it, the benchmark query uses that: two of the four for a standard equivalent, three for any other operation. Examples: CAST(x AS DATE) instead of DATE(x), ROW_NUMBER() instead of DISTINCT ON, JSON_VALUE instead of ->>. What the standard leaves implementation-defined is written out: where NULLs go in every sort, tie-breakers.
    • Otherwise the benchmark query has a placeholder (listed below), and each system's version spells it.
  • A system rejects standard SQL.
    • If a simple equivalent exists in the system's dialect, the benchmark query marks the construct as a placeholder, with the standard construct as its default, and that system's version uses the equivalent. Simple means a fixed pattern at that place in the query that gives the standard's result for every input and can be applied by a simple textual match, without semantic analysis. If the equivalent costs nothing on any system, such as renaming an alias, the benchmark query uses it instead.
    • Otherwise the error stays, even where a larger rewrite would work.
  • A system runs standard SQL and returns something else.
    • If two of the four return the standard's result, the system is wrong, and its result stays.
    • Otherwise the four mostly disagree with the standard as well, and the benchmark query says explicitly what it means, as for the types and scales of numbers.

Systems may apply their own settings at startup to follow the standard more closely, such as DuckDB's integer division mode.

Every change from the SQL the agents wrote to the benchmark query is recorded, query by query. Each query's page shows these changes, and every system's version.

date_diff_days(a, b)

the days from a to b; the difference of two dates is a number of days in PostgreSQL and an interval in the standard.

Default(CAST({b} AS DATE) - CAST({a} AS DATE))

  • DuckDB, GizmoSQLDATE_DIFF('day', {a}, {b})
  • ClickHouse, chDBdateDiff('day', {a}, {b})
year_month(x)

a timestamp as a 'YYYY-MM' string.

Defaultto_char({x}, 'YYYY-MM')

  • DuckDB, GizmoSQLstrftime({x}, '%Y-%m')
  • CedarDBto_char(CAST({x} AS timestamp), 'YYYY-MM')
  • ClickHouse, chDBformatDateTime({x}, '%Y-%m')
fetch_first(n)

the first n rows (the standard's FETCH FIRST n ROWS ONLY).

DefaultFETCH FIRST {n} ROWS ONLY

  • DataFusionLIMIT {n}
bool_or(x)

whether any value is true (PostgreSQL's aggregate).

DefaultBOOL_OR({x})

  • ClickHouse, chDBmax({x})
percentile_cont(p, x)

the p-th percentile of x, interpolated between the closest values.

DefaultPERCENTILE_CONT({p}) WITHIN GROUP (ORDER BY {x})

  • FireboltPERCENTILE_CONT({p}) WITHIN GROUP (ORDER BY CAST({x} AS DOUBLE PRECISION))
  • ClickHouse, chDBquantileExactInclusive({p})(toFloat64({x}))
percentile_cont_filter(p, x, condition)

the same, over the rows that meet a condition.

DefaultPERCENTILE_CONT({p}) WITHIN GROUP (ORDER BY {x}) FILTER (WHERE {condition})

  • HyperPERCENTILE_CONT({p}) WITHIN GROUP (ORDER BY CASE WHEN {condition} THEN {x} END)
  • FireboltPERCENTILE_CONT({p}) WITHIN GROUP (ORDER BY CAST({x} AS DOUBLE PRECISION)) FILTER (WHERE {condition})
  • ClickHouse, chDBquantileExactInclusiveIf({p})(toFloat64({x}), {condition})
string_agg_sorted(x, sep)

the values of x joined with a separator, in their order (PostgreSQL's STRING_AGG).

DefaultSTRING_AGG({x}, {sep} ORDER BY {x})

  • FireboltCASE WHEN COUNT({x}) = 0 THEN NULL ELSE ARRAY_TO_STRING(ARRAY_SORT(ARRAY_AGG({x})), {sep}) END
  • ClickHouse, chDBarrayStringConcat(arraySort(groupArray({x})), {sep})
string_agg_distinct_sorted(x, sep)

the same, each value once.

DefaultSTRING_AGG(DISTINCT {x}, {sep} ORDER BY {x})

  • FireboltCASE WHEN COUNT({x}) = 0 THEN NULL ELSE ARRAY_TO_STRING(ARRAY_SORT(ARRAY_DISTINCT(ARRAY_AGG({x}))), {sep}) END
  • ClickHouse, chDBarrayStringConcat(arraySort(groupUniqArray({x})), {sep})
count_distinct_pair(a, b)

the number of distinct pairs (a, b).

DefaultCOUNT(DISTINCT ({a}, {b}))

equals_any(x, arr)

whether an array column contains x; the queries said PostgreSQL's = ANY(array), the default is the standard's membership test.

Default{x} IN (SELECT dbo_unnest.dbo_element FROM UNNEST({arr}) AS dbo_unnest(dbo_element))

  • Hyper{x} = ANY(CAST('[' || SUBSTRING({arr} FROM 2 FOR CHAR_LENGTH({arr}) - 2) || ']' AS text[]))
  • ClickHouse, chDBCASE WHEN {arr} IS NULL THEN NULL WHEN empty({arr}) THEN false WHEN {x} IS NULL THEN NULL WHEN has({arr}, {x}) THEN true WHEN has({arr}, NULL) THEN NULL ELSE false END
add_minutes(ts, n)

a timestamp plus n minutes; the default is the standard's interval, which of the four most used databases only PostgreSQL supports.

Default{ts} + {n} * INTERVAL '1' MINUTE

  • ClickHouse, chDBaddMinutes({ts}, {n})
whole_days_between(a, b)

the whole days from a to b, two timestamps (PostgreSQL's day field of their difference).

DefaultEXTRACT(DAY FROM {b} - {a})

  • ClickHouse, chDBintDiv(dateDiff('microsecond', {a}, {b}), 86400000000)
timestamp_midpoint(a, b)

the timestamp halfway between a and b.

Default{a} + ({b} - {a}) * 0.5

seconds_between(a, b)

the seconds from a to b, exactly (PostgreSQL's EXTRACT(EPOCH ...)).

DefaultCAST(EXTRACT(EPOCH FROM ({b} - {a})) AS NUMERIC(18,6))

  • ClickHouse, chDBCAST(divideDecimal(toDecimal128(dateDiff('microsecond', {a}, {b}), 0), toDecimal128(1000000, 0), 6) AS Nullable(Decimal(18, 6)))
count_distinct_over(x, window)

the number of distinct values of x in a window.

DefaultCOUNT(DISTINCT {x}) OVER ({window})

  • Hyper, CedarDB, pg_duckdb, PostgreSQL, Firebolt, clickhouse_brew(DENSE_RANK() OVER ({window} ORDER BY {x} ASC NULLS FIRST) + DENSE_RANK() OVER ({window} ORDER BY {x} DESC NULLS LAST) - 1 - MAX(CASE WHEN {x} IS NULL THEN 1 ELSE 0 END) OVER ({window}))

Procedure

For every machine and every system, starting from nothing:

  1. Start the system and wait until it answers.
  2. Create the schema, all tables at once.
  3. Load everything through the end of 2024: every table's file, all tables at once.
  4. Run all 409 queries once, in order.
  5. Load the next quarter and run all queries again, until all of 2025 is loaded: five loads, five rounds of queries.
  6. Check every result of the last round against the reference results.
  7. Stop the system and remove everything it wrote.

The system is never restarted during a run, so its caches stay warm, as they would in production.

A query over its timeout (5 s at SF1 and SF10, 10 s at SF100, 30 s at SF1000) is aborted and recorded as a timeout; it is skipped for the rest of that round and tried again after the next load. A query that fails is treated the same.

A run fails as a whole when the system runs out of memory, crashes, stops answering, or cannot load a file. The figures then show the run up to that point.

Isolation

Measures

Query time is measured by the benchmark, not reported by the system: from sending the query to having its whole result, over a connection that stays open for the run, so no client starts per query. Every query runs once after a load, and that time counts.

Load time is measured the same way: from sending a load, every table's file of the period at once, to the system's answer that all of it is loaded. Creating the schema is timed apart and not counted.

Relative time (Table 1) is the geometric mean, over all 409 queries, of a query's time relative to the fastest correct system on that machine, both plus 10 ms so that the shortest queries do not dominate. A timed-out query counts as taking the full timeout (5 s at SF1 and SF10, 10 s at SF100, 30 s at SF1000); a failed one, one with a wrong result or one not run as taking twice the slowest correct time of that query on that machine (the timeout if no system answered it correctly).

Queries per second (Figure 1) is 409 over the time all queries take together after the last load (the median of each). A timed-out query counts as taking the full timeout, a failed or wrong one as taking twice the slowest correct time of that query.

Cost (Figure 1) is that same time at the machine's price per hour: what it costs to run all 409 queries once.

Efficiency (Figure 1) is queries per second (y-axis) divided by cost (x-axis). A system is efficient when it answers quickly without needing an expensive machine to do it: a bigger machine is worth it only if it makes the answers enough faster to justify its price. Looking at cost alone would favour a tiny machine that takes all day; looking at speed alone would always favour the biggest machine, whatever its price.

We measure efficiency by dividing queries per second by cost. Since both depend on how long the run takes, for ranking machines this comes down to one over price per hour × run time × run time. Chip designers use the same idea: the energy–delay product (Gonzalez and Horowitz, 1996). A chip run at a lower voltage uses less energy but computes more slowly. Judged by energy alone, the best chip would crawl at the lowest voltage; judged by speed alone, it would burn power at the highest. Multiplying energy by time finds the balance. For a database, the size of the machine plays the part of voltage, and the cost of the run the part of energy.

Another way to understand the metric is through Amdahl's law. No system scales perfectly with more resources, because some of its work cannot be split up. Say a system does 90 seconds of work that splits up easily across cores, and 10 seconds that does not, on machines priced by the core. On one core it takes 100 seconds, on 3 cores 40, on 9 cores 20. On 9 cores the split-up part takes 10 seconds, as long as the part that cannot be split, and that is where efficiency peaks. Before that point, more cores still shorten the run a lot; beyond it, the run mostly waits for the 10 seconds that no core can help with, and more cores add to the price far more than they take off the time.

Correctness

Results are compared by value with reference results, row by row in order. The reference results come from Umbra with exact arithmetic, and at every scale factor match DuckDB on every query DuckDB answers.

Only the last round is checked; in earlier rounds a result counts as correct when the query finishes.

Use of AI

The queries were asked by language-model agents. In the role play, Claude Sonnet 5 played every manager, analyst and engineer, and wrote the SQL. Every transcript is published with the queries.

The benchmark was built with an AI coding assistant. Claude Opus 5.5 designed the background story of Tidewick Commerce, the events that happen in it, and the data generation with its built-in skew.

Running the benchmark involves no AI. The data is generated deterministically, the 409 queries are fixed SQL, and results are checked against reference results. Nothing in it depends on a model or a vendor: any system that speaks SQL can run it.

Reproduce

Everything is in the repository: the generator, the queries, the reference results, the runner and the scripts for every system. On Linux with Podman:

cargo build --release --workspace
./target/release/dbo-bench run --benchmark bench/benchmark_sf1.toml --system duckdb --config sf1-c8a.medium

Results go to systems/<system>/<date>/<config>.csv, one row per load and query.