Add an entry for TDengine

[alexey-milovidov]

Adds an entry for TDengine TSDB OSS 3.4.2.5, an
open-source (AGPL-3.0) time-series DBMS written in C and aimed at industrial
IoT. It is installed from the vendor's Linux tarball (x64 and arm64 both
available, no registration) and queried over the REST endpoint taosadapter
exposes, which is the connection method its docs recommend over the native
protocol. Results are publishable — the licence has no DeWitt clause — but
there are none yet: those need runs on the benchmark's own machines.

tdengine/README.md covers all of the below in detail. Summarising here
because a reviewer will want to know why the entry does not look like its
neighbours.

The dataset is modelled as a supertable, one subtable per CounterID

This is the one judgement call in the entry and it is worth checking.

A basic (non-super) TDengine table lives in exactly one vgroup and is scanned
by exactly one thread — with hits as a flat table on a 96-core box, a
GROUP BY over a regex on Referer keeps one vnode-query thread busy and
leaves the other 95 idle. Benchmarking that would have measured one core on
every machine in the fleet.

So hits is a supertable with CounterID as its tag and one subtable per
counter, 6,506 of them, and ./load sets the database's vgroups to the core
count, which is how the vendor's
capacity planning guide
sizes it ("each CPU core can serve 1 to 2 vnodes"). This is the data model
TDengine documents for any dataset with a natural series key, and CounterID
is the natural one — it identifies the web counter that produced the hit, and
it is the leading column of ClickBench's primary key in the ClickHouse entry
too.

Two consequences, both in TDengine's favour and both called out in the README:
a tag is stored once per subtable rather than once per row, so CounterID is
effectively free in ./data-size; and a tag filter prunes to matching
subtables before reading anything, so Q37–Q43 (CounterID = 62) touch one
subtable, which is the same kind of advantage ClickHouse gets from its primary
key. SELECT * on a supertable returns tags last and TDengine requires the
primary timestamp first, so Q24 returns the same 105 values per row in a
different column order.

INSERT INTO <supertable> ... FILE needs a tbname column in the field list
and the list may not name a column twice, so csv-columns.txt maps
hits.csv's CounterID field onto tbname and ./load sets the tag
afterwards from information_schema.ins_tables — the subtable name is the
tag value, so nothing has to be inferred and the data is not read twice.

Five bugs, filed upstream

taosdata/TDengine#35449	taosd segfaults on COUNT(DISTINCT col) once the deduplicated set outgrows pqSortMemThreshold (16 MB) and the spill-to-disk merge sort takes over. Backtrace from a core dump is in the issue: initSpillSortHandle → tsortOpen → msortComparFn, reached with a column-info pointer that does not describe the column being sorted. All eight of ClickBench's COUNT(DISTINCT ...) queries are far above the threshold, so all eight took the server down; they now deduplicate in a subquery instead. Exact rewrites, no HYPERLOGLOG.
taosdata/TDengine#35450	GROUP BY ... LIMIT n returns every group when there is no ORDER BY. Q18 is exactly that shape and returned 830,079 rows for LIMIT 10 on a 1% slice; reproduces on a 1,000-row table. Q18 now wraps its aggregate in a subquery.
taosdata/TDengine#35451	AGG(DISTINCT x) beside an aggregate whose argument is an expression rather than a bare column fails with Out of range [0x80000112] on three rows. count(DISTINCT n), sum(n) is fine; count(DISTINCT n), sum(n+1) is not. No final query hits it, but it cost time while translating them.
taosdata/TDengine#35452	GROUP BY is capped at 10,000,000 groups (Too many groups/time window in query). Undocumented, no configuration parameter, on the merged result rather than per-vnode partials so sharding does not help, and enforced during execution — a doomed query still scans for two to ten minutes first.
taosdata/TDengine#35453	GROUP BY on a VARCHAR column is priced by the declared width rather than the data. Details below.

14 of the 43 queries produce no result at 100M rows

Measured with a full cold sweep — ./stop, drop the page cache, ./start,
./check, one run of each query, which is what lib/benchmark-common.sh does
before every cold measurement. Six are refused by the group ceiling, six run
out of query memory, two do not finish inside the timeout. The README has the
per-query table with each actual error and duration.

One of those failures needed explaining, because it is not a cardinality
problem: Q29 fails with only 3,007,986 groups. Grouping on a string column is
priced by the width the column is declared with, not by its values. Same
3,000,000 distinct 11-byte keys, changing nothing but the declaration: 10.4 s
at VARCHAR(32), 12.4 s at VARCHAR(256), 32.9 s at VARCHAR(1024), 111.4 s
at VARCHAR(2048), and at VARCHAR(4096) it stops completing. Declared widths
are not a free choice either, since TDengine rejects a value longer than its
declaration, so hits forces URL to VARCHAR(8192), Referer to
VARCHAR(3072) and SearchPhrase to VARCHAR(2048). Filed as
taosdata/TDengine#35453.

The queries are left in their natural form rather than approximated, so the
nulls say what the system does. A time-series engine built for windowed
rollups over a bounded set of series doing badly at grouping 100M rows by a
high-cardinality string is a fair result, and the entry says so rather than
working around it.

Loading

INSERT INTO hits (...) FILE is the straightforward SQL path and the one used,
but the taos client materialises an entire statement in memory before sending
anything, so the 81 GB hits.csv dies with Out of Memory [0x80000102] after
about 30 seconds of parsing. load-csv.py feeds it 128 MB at a time through a
single scratch file that each chunk overwrites, so the load needs one chunk of
spare disk rather than a second copy of the dataset.

Chunk boundaries have to respect CSV records rather than lines: 3,740 URL
values and 2,168 Referer values in hits contain literal newlines, and
splitting on \n corrupts rows silently, since the fragments still parse as
numbers and strings. A newline ends a record only when an even number of "
characters precede it. Chunks are byte-exact slices, so values reach the server
with their original bytes, including the non-UTF-8 ones.

Two other things the load has to do, both correctness rather than tuning:
KEEP is raised from its default 3650 days, because TDengine rejects any row
whose primary timestamp is older than now - KEEP and hits is from July
2013; and WatchID joins EventTime as a COMPOSITE KEY, because TDengine
overwrites on primary-key collision and EventTime has only 1,432,857 distinct
values across 99,997,497 rows — a naive load would keep 1.4% of the dataset.
(EventTime, WatchID) is unique across all of it, and after a full load
SELECT count(*) returns exactly 99,997,497.

Configuration

Left at the shipped defaults except for VGROUPS (above), KEEP (above), and:
monitor, telemetryReporting and crashReporting off, so no metrics are
written into a database inside the server being measured and nothing is
reported to the vendor mid-run; tempDir moved off /tmp, which on Ubuntu
25.04+ is a tmpfs, so sort spills do not either fail on a small instance or
land in RAM on a large one; queryWaitTimeout 1800 s, because the default 900
would cut off Q6 (about 870 s) while still being the only thing bounding the
queries that never finish. taoskeeper and taos-explorer, two of the four
units the installer enables, are disabled.

taosd snapshots its configuration into dataDir on first start and prefers
the snapshot over taos.cfg afterwards, so ./install deletes the snapshot
after editing — otherwise the edits read back as cfg_file in taosd -C while
the old values stay in force.

Verified

Full 100M-row load through these scripts: 81,136,059,858 bytes of CSV in
9,129 s, count(*) exactly 99,997,497, 6,506 subtables with zero NULL tag
values, ./data-size 76,202,317,209 bytes (about 40 GB of TSDB files plus
about 32 GB of WAL, which WAL_RETENTION_PERIOD drops an hour later).

And on a deterministic 1,000,765-row 1% sample, every one of the 43 queries
compared against clickhouse-local reading the same rows. The only
unexplained difference is Q4, where avg(UserID) overflows Int64: TDengine
accumulates in something wider and returns the true average, ClickHouse
returns the wrapped sum divided by the count. Q21–Q24 differ only because
TDengine's LIKE is case-insensitive — they match exactly once the reference
is lowered, and the queries keep LIKE, which is the same choice the
MySQL-family entries make. The eight queries whose ten rows are not uniquely
determined were re-compared with LIMIT/OFFSET removed so the whole grouped
result had to agree; all eight match as multisets. Q24's ten rows agree in all
1050 cells when compared by column name, which exercises every column's type
and the non-UTF-8 string values end to end.

shellcheck clean on all seven shell scripts, ruff clean on load-csv.py.

[github-actions]

The run of tdengine on t3a.small did not produce results.

Logs:

• tdengine on t3a.small: benchmark log, cloud-init log

[github-actions]

Results for tdengine are ready for: c6a.metal, c7a.metal-48xl, c8g.metal-48xl.
The result files are committed as 7ebf8ad.

Logs:

• tdengine on c7a.metal-48xl: benchmark log, cloud-init log
• tdengine on c8g.metal-48xl: benchmark log, cloud-init log
• tdengine on c6a.metal: benchmark log, cloud-init log

[github-actions]

The run of tdengine on c6a.4xlarge did not produce results.

Logs:

• tdengine on c6a.4xlarge: benchmark log, cloud-init log

[github-actions]

The run of tdengine on c6a.2xlarge did not produce results.
The run of tdengine on c6a.4xlarge did not produce results.
The run of tdengine on c6a.large did not produce results.
The run of tdengine on c6a.xlarge did not produce results.
The run of tdengine on c8g.4xlarge did not produce results.

Logs:

• tdengine on c6a.large: benchmark log, cloud-init log
• tdengine on c6a.xlarge: benchmark log, cloud-init log
• tdengine on c6a.2xlarge: benchmark log, cloud-init log
• tdengine on c6a.4xlarge: benchmark log, cloud-init log
• tdengine on c8g.4xlarge: benchmark log, cloud-init log