Proposal: ~2.5–3× faster parsing, conformance-preserving

[tfoutrein]

TL;DR

I have a set of parser-only optimizations that make tomlkit.parse() roughly
2.4–2.9× faster (content-dependent) with no behavioural change: the full
test suite passes, including 680/680 of the toml-test conformance corpus,
and ruff + mypy --strict are clean.

Before opening a large PR, I'd like to check your appetite and preferred shape —
the work is naturally splittable into independent, separately-reviewable pieces.

Why this matters / why I'm being careful

tomlkit is a hot dependency of Poetry (and poetry-core), so I treated
correctness as non-negotiable: every change is gated on the conformance suite,
not just round-trip of valid files. (In fact, while validating I caught a subtle
EOF-sentinel collision a naive char-interning optimization can introduce — the
fix is included and is itself covered by toml-test invalid/control/*-null.)

Reproducing the numbers

A dependency-free benchmark is attached (bench_standalone.py): run it on
master, then on the branch, and compare medians. Measured here (macOS,
CPython 3.11):

# master (8694e4d)
$ python bench_standalone.py
  median :  791.6 ms   (1583 us/parse)

# branch
$ python bench_standalone.py
  median :  316.8 ms   ( 634 us/parse)   -> ~2.5x faster

Cross-checked with a drift-immune A/B (both versions timed, interleaved, in one
process) on other workloads:

Workload	master	branch	factor
embedded sample (bench_standalone.py)	792 ms	317 ms	×2.50
number-heavy ~3 KB doc	~2970 ms	~1045 ms	×2.85
262 distinct toml-test/valid files	1747 ms	717 ms	×2.44

What the changes do (grouped, so you can pick)

The core idea: the parser advanced one character per call (inc() ~2.2 M
times on the benchmark). The big wins replace those with bulk scans over the
underlying string. Grouped by theme, independently reviewable:

#	Theme	Idea	Risk
A	Bulk char scanning	index-based Source; advance_while/advance_until replace per-char inc() loops in bare-keys, whitespace, numbers, comments, single-line strings	medium — hot path, but pure scan-equivalence
B	Dispatch micro-wins	frozenset membership, precomputed enum values, is vs set membership, hoisting loop invariants, binding Source delegates	low — mechanical
C	Object interning	module-level TOMLChar cache (+ the NUL/EOF fix)	medium — includes the correctness fix above
D	__deepcopy__	hand-rolled Container/Trivia deepcopy to skip the reflective machinery (super-table merge path)	higher — touches copy/alias semantics, wants careful review

Themes A + B alone deliver most of the gain at the lowest risk; C and D can be
deferred or dropped.

What I've verified — and what I haven't (yet)

• ✅ Full pytest tests green incl. toml-test (680/680), matching master.
• ✅ ruff check, ruff format, mypy --strict clean.
• ✅ Drift-immune A/B benchmark (numbers above).
• ⏳ Only CPython 3.11 / macOS so far — not the full CI matrix (3.9–3.14 ×
  3 OS) nor the Poetry/poetry-core integration jobs. I'll run those before any PR
  if you're interested.

Question for the maintainers

1. Is a parser speedup of this size something you'd consider merging?
2. If so, do you prefer one PR per theme (A, then B, …) or a single curated
   PR of A+B with C/D as follow-ups?
3. Any themes you'd rather not take (e.g. the __deepcopy__ rewrite) on
   maintainability grounds?

Happy to adapt to whatever keeps the hot path readable for you.

---

bench_standalone.py (zero-dependency, copy-paste & run)

#!/usr/bin/env python3
"""Standalone, dependency-free parse benchmark for tomlkit.

Run it on `main`, then on the optimization branch, and compare the medians:

    python bench_standalone.py            # default: 500 iters x 15 runs
    python bench_standalone.py --iters 1000 --runs 25

Only stdlib + the locally-installed `tomlkit` are used, so the numbers are
trivially reproducible. The sample document below is embedded (no external
files) and exercises the common value kinds: strings, ints/floats, bools,
datetimes, arrays, inline tables, nested tables, AoT and comments.

Methodology: warmup, then `runs` timed batches of `iters` parses each;
we report the median batch time (robust to OS scheduling noise) and stdev.
"""
from __future__ import annotations

import argparse
import statistics
import time

import tomlkit

# Representative ~3 KB TOML covering the value kinds the parser dispatches on.
SAMPLE = """\
# A representative TOML document
title = "tomlkit benchmark"
version = 2
ratio = 3.14159
enabled = true
created = 2024-01-15T08:30:00Z
tags = ["alpha", "beta", "gamma", "delta"]
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
inline = { x = 1, y = 2, z = 3 }

[server]
host = "localhost"
port = 8080
timeout = 30.5
backends = ["10.0.0.1", "10.0.0.2", "10.0.0.3"]

[server.tls]
enabled = true
ciphers = ["TLS_AES_256", "TLS_CHACHA20"]

[database]
url = "postgres://localhost:5432/app"  # inline comment
pool_size = 20
read_only = false

[[products]]
name = "Widget"
sku = 738594937
price = 19.99

[[products]]
name = "Gadget"
sku = 284758393
price = 49.95
in_stock = false

[logging]
level = "info"
format = "%(asctime)s %(message)s"
rotate_mb = 100
"""


def measure(content: str, iters: int) -> float:
    t0 = time.perf_counter()
    for _ in range(iters):
        tomlkit.parse(content)
    return (time.perf_counter() - t0) * 1000.0  # ms


def main() -> None:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--iters", type=int, default=500)
    ap.add_argument("--runs", type=int, default=15)
    ap.add_argument("--warmup", type=int, default=3)
    args = ap.parse_args()

    for _ in range(args.warmup):
        measure(SAMPLE, args.iters)

    samples = sorted(measure(SAMPLE, args.iters) for _ in range(args.runs))
    median = samples[len(samples) // 2]
    stdev = statistics.stdev(samples) if len(samples) > 1 else 0.0

    print(f"tomlkit {tomlkit.__version__ if hasattr(tomlkit, '__version__') else '?'}")
    print(f"{args.iters} parses x {args.runs} runs (warmup x{args.warmup})")
    print(f"  median : {median:8.1f} ms   ({median / args.iters * 1000:.1f} us/parse)")
    print(f"  stdev  : {stdev:8.1f} ms")
    print(f"  min/max: {samples[0]:.1f} / {samples[-1]:.1f} ms")


if __name__ == "__main__":
    main()

[dimbleby]

cf #467.

(Your "Representative ~3 KB TOML" is less than 750 bytes)

[tfoutrein]

@dimbleby that's a good point 😅

[tfoutrein]

Thanks @dimbleby, and fair point on the bench — my "representative" sample was under 750 bytes. I've fixed bench_standalone.py so it self-reports the real byte size and accepts a file path, and it now runs on pyproject.toml directly.

Quick context on why I'm being so thorough: this is part of a meetup talk I'm giving on self-improvement harnesses, where the thesis is that you can point one at a real, widely-used library and get genuine, conformance-preserving speedups. tomlkit (a hot Poetry dependency) is the case study — and I'd genuinely like the improvements to land upstream, not just live on a slide.

#467 is the right foundation here. I ported several of its ideas (credited) onto current master and added a few levers on top: bulk char-scanning (advance_while/advance_until instead of per-char inc()), a one-slot header cache that drops the peek→restore→reparse round-trip, iterative dotted-key construction, and fast-paths on SingleKey.__eq__/__hash__ and Container.__contains__. This builds on #467 rather than competing with it.

Same master baseline (6057016), identical methodology, best-of-10, single machine (macOS / CPython 3.11):

workload	master	#467	this branch
pyproject.toml (1426 B)	×1.00	×4.52	×6.34
data.toml (~4 KB config)	×1.00	×2.60	×3.86
nested super-tables (150 synthetic)	×1.00	×2.37	×3.49

The extra headroom on pyproject.toml is mostly the bulk-scanning + header cache that #467 doesn't have. Everything is iso-functional and gated per-commit: full pytest including toml-test conformance (680/680), a differential guard on as_string()/unwrap()/Python types, an ABC-contract guard, plus a drift-immune A/B bench; ruff + mypy --strict clean.

Happy to package this however suits you — themed PRs against master, a curated set, or just hand the commits over. How would you like it sliced?

FYI if you ever split #467 into independent pieces

#467 is correct as-is. But if you wanted the in-place super-table merge (dropping copy.deepcopy(current)) to stand alone on master, it needs the rest of #467's parser refactor to stay correct — applied by itself it trips test_update_nested_out_of_order_table and test_out_of_order_table_can_add_multiple_tables. A master-safe variant: keep the in-place merge but deep-copy in OutOfOrderTableProxy.validate() (the only consumer that needs the defensive copy), which passes independently of the parser work.

[tfoutrein]

@frostming, dimbleby pointed here from #467, so I'll pick up the thread. Thanks @dimbleby for the hand-off, and good luck with tomlrt.

With dimbleby focusing on tomlrt, I'm happy to carry the performance work forward against current master. Up front, since this is harness-assisted: every change is individually benchmarked and gated before it can land. Full pytest including toml-test conformance (680/680), a differential guard comparing as_string() / unwrap() / parsed Python types against master, an ABC-contract check, and a drift-immune A/B bench. Anything that isn't iso-functional, or isn't actually faster, gets rejected by the gate rather than merged. I review and stand behind each PR personally; the harness does the legwork, I'm accountable for what's proposed.

Rather than drop one large PR on you, I'd suggest starting with a single small, self-contained change with its own before/after numbers, for example the SingleKey.__eq__/__hash__ and Container.__contains__ fast-paths, which don't depend on the parser refactor. That lets you judge the code quality and review cost on something low-risk before deciding whether the rest is worth your time. If the first one looks good, I can follow up with the bigger levers (bulk char-scanning, the one-slot header cache, iterative dotted-key construction) as separate themed PRs, each independently benchmarked and gated.

Your call entirely on scope and ordering. And if you'd rather not take on parser-level perf changes against master right now, that's a fine answer too. Just let me know how you'd like it sliced.

[frostming]

@tfoutrein it SGTM.

If performance improvements can be divided into many small improvements, that would be optimal