Lexer: return a struct-of-arrays TokenList (BC break, ~26% faster lexing)

[JanTvrdik]

Follow-up to #313, now merged and released as 2.3.5. Rebased onto 2.3.x, so
this is one commit against current master. This one is a BC break and needs a
major, which #313 deliberately was not.

The BC break

Lexer::tokenize(string $s): array               →  : TokenList
TokenIterator::__construct(array $tokens, …)    →  (TokenList $tokens, …)
TokenIterator::getTokens(): array               →  : TokenList

Lexer::VALUE_OFFSET                             →  removed
Lexer::TYPE_OFFSET                              →  removed
Lexer::LINE_OFFSET                              →  removed

Those six, and nothing else — the BC checker on this PR reports exactly them.

TokenList holds three parallel packed arrays plus a count:

final class TokenList implements Countable
{
	/** @var list<string> */  public array $values;
	/** @var list<int> */     public array $types;
	/** @var list<int> */     public array $lines;
	/** @var int<0, max> */   public int $count;

	public function count(): int;
}

TokenList implements Countable, so count($tokens) written against the
array keeps working. The $count property is the fast path for hot loops; the
method exists for the migration.

The three offset constants are dropped rather than left behind. They index into
a tuple that no longer exists, so keeping them would only let code compile that
cannot work — a fatal at the point of use is worse than a missing constant the
migration can grep for.

Who has to change:

• Anything that indexes the token list itself — $tokens[$i][Lexer::TYPE_OFFSET]
  becomes $tokens->types[$i]. This is the whole migration.
• Anything that passes tokens straight into a TokenIterator — nothing to do.
  new TokenIterator($lexer->tokenize($s)) keeps compiling, because both ends of
  the break move together.

This repository has consumers of both kinds, which makes a good worked
example. The five new TokenIterator($lexer->tokenize(...)) call sites needed
nothing. tools/phplrt/Fuzzer/TokenStream indexes tokens and is migrated here:
mostly $tokens[$i][Lexer::TYPE_OFFSET] → $tokens->types[$i]. Splitting the
tuple also let three of its helpers narrow from the whole token list to just the
types array they actually read, which is the sort of tidy-up the new shape
invites. That file is not covered by make check, so the grammar CI jobs caught
the migration rather than my local run — worth knowing if you go looking for
other consumers.

For calibration I also grepped phpstan-src: all five phpdoc-parser call sites
there (PhpDocStringResolver, TypeStringResolver, InvalidPHPStanDocTagRule,
InvalidPhpDocTagValueRule, PhpDocEditor) are the second kind and need no
edit, and nothing in it indexes a phpdoc-parser token. I could only survey the
consumers on my disk, though — you will have a far better sense of what else
reaches into the token array.

Why

Since 2.3.5 tokenize() extracts the matches with PREG_PATTERN_ORDER, but it
still copies the values out of $matches[0] one at a time to build a tuple per
token. With three parallel arrays that last per-token allocation disappears:
$matches[0] becomes the values array of the TokenList, and only the types
and the lines are filled in a loop.

Numbers

Same corpus as #313 — 37 150 unique docblocks, 8.38 MiB, 2 355 859 tokens — one
core, php -n, minimum of 15 rounds with the variants interleaved.

	lexing	parse	bytes/token
2.3.5	235 ms	777 ms	269
+ TokenList	175 ms (−26%)	671 ms (−14%)	124 (−54%)

With opcache and the tracing JIT: −28% lexing and −18% parse, so the gain holds
up there too.

For scale against the release before #313 landed: in a single interleaved run of
all three, 2.3.4 lexed the same corpus in 416 ms and parsed it in 1002 ms
against 2.3.5's 249 ms / 822 ms and this PR's 184 ms / 692 ms — so #313 and this
PR together are −56% lexing and −31% parse.

The memory figure is the tokens of the whole corpus held at once: 605 MiB → 277
MiB. Three packed arrays cost 16 bytes per token per array; a packed array of
three zvals costs an array header plus a rounded-up slot allocation.

Behaviour

Tokens and the ASTs printed from them are byte-identical to 2.3.5 over all
37 150 docblocks (xxh128 over both). On PHP 8.5, with the make grammars-install
toolchain in place so nothing is skipped: 7747 tests pass, including the grammar
sync and fuzzer suites, plus phpcs, PHPStan and the no-capturing-groups guard
test added in #313, which applies unchanged.

Alternative I measured and discarded

Token objects instead of tuples, the other obvious way to shrink a token:
+21% lexing, +8% parse, 164 bytes/token. new Token(...) is a userland call
where the array literal is one opcode, and the ~2% the parser gains from reading
properties instead of indexing arrays does not pay for it. Same migration cost
as this PR, less than half the payoff — so if a token-representation break is
worth taking at all, this is the one to take.

[Copilot]

Pull request overview

This PR introduces a new TokenList “struct-of-arrays” token representation and updates core parser/printer code to consume it, enabling significantly faster lexing and reduced memory usage at the cost of a major BC break (changing Lexer::tokenize() and TokenIterator token APIs).

Changes:

• Change Lexer::tokenize() to return TokenList (parallel arrays + count) instead of list<array{string,int,int}>.
• Update TokenIterator and Printer to use TokenList fields (values/types/lines/count) rather than tuple offsets.
• Adjust tests to access token values from TokenList.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/PHPStan/Parser/TypeParserTest.php	Updates test logic to read token text from TokenList->values.
src/Printer/Printer.php	Switches format-preserving printing token reads to TokenList fields and uses count.
src/Parser/TokenIterator.php	Updates iterator internals and public API to accept/return TokenList.
src/Lexer/TokenList.php	Adds the new TokenList data structure backing the new token representation.
src/Lexer/Lexer.php	Produces TokenList from regex matches (values reuse + types/lines arrays).

Suppressed comments (1)

src/Lexer/TokenList.php:38

• If TokenList implements \Countable, it should provide the required count() method (ideally returning the cached count).

		$this->count = count($values);
	}

}

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JanTvrdik]

CI is as expected: the two red checks are Backward Compatibility, which is the point of the PR, and PHPStan (7.4), which is unrelated — phpstan 2.2.11 released today fatals on PHP 7.4 during its own phar bootstrap (Call to undefined function str_contains() in Turbo/TurboExtensionSelector.php:113), before it reads a single project file. The same base commit passed that job yesterday on 2.2.10, and #313 hits it too.

The BC checker lists exactly the three signatures the description names, and nothing else:

[BC] CHANGED: The return type of Lexer#tokenize() changed from array to TokenList
[BC] CHANGED: The parameter $tokens of TokenIterator#__construct() changed from array to TokenList
[BC] CHANGED: The return type of TokenIterator#getTokens() changed from array to TokenList

Everything else is green, including Grammars (8.4) and Grammars (8.5) — those caught the one consumer in this repo that indexes tokens (tools/phplrt/Fuzzer/TokenStream), which is migrated in the same commit.