DDIR: the optimizer as a DDIR program (an e-graph over each scope) - #856
Open
frankmcsherry wants to merge 12 commits into
Open
DDIR: the optimizer as a DDIR program (an e-graph over each scope)#856frankmcsherry wants to merge 12 commits into
frankmcsherry wants to merge 12 commits into
Conversation
Both the row and the corgi `leave_dynamic` read `cap.time()`, which now
panics ("expected a singleton stamp") when a message carries timestamps
from two epochs — late iterations of one alongside early ones of the next,
which any steady-state run of an iterative program produces. Hold a
capability per stamp element, each truncated exactly as the records are,
and open the session on the set.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
The round loop churned synthetic rows from `gen_row`, which with a file-fed input meant retracting rows the dataflow never saw and inserting random ones (duplicate ids, cycles). Each update now retracts one of the file's rows and re-inserts it with its last field rewired below its first, so a round is an edit to data the dataflow holds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
compare_idx and gather read sum discriminants in place (WIP master 3c4b0d6f), which is what took the e-graph's 200k-node saturation from 43s to 23s on the corgi backend. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
…a program An iteration quiesces when its feedback carries no records, so a var whose step's additions and retractions cancel only logically — a `negate` under a `concat` — never quiesced: the vec backend ran forever on a program whose fixpoint was reached in one round. Every var is now set to its consolidated step (`Backend::consolidate`, through the arrangement by default). The suite's timings are unchanged; the programs there all fed their vars through a reduce, which consolidated by nature. tests/identifiers.rs writes DD's `identifiers` algorithm in DDIR: each record proposes hash(round, record); among the proposers of an id the least (round, record) wins and the losers try the next round; with hash, min, negate and a var, over 64 slots for 32 records, every record gets its own id (25 settle at round 0, the rest by round 4). It is the primitive an e-graph rule needs to mint a node for a new term inside the program. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
…messages come from Instrumenting the batch-shipping operators on the run that panicked (a DDIR program at 20k nodes, four workers, epochs fed without waiting) shows the multi-element stamps being minted: an arrange retiring (1, [18]) and (2, []) — epoch 1 at round 18 alongside epoch 2 just entered — in one batch, then a join forwarding its input batch's set, then a reduce retiring the same pair, and finally an arrange whose batch, forwarded by as_collection under the retiring set and delayed element-wise by the feedback, reached the scope's exit as (1, [19]), (2, [1]). Which retirements coincide depends on scheduling, so the test makes the message directly: an operator that ships its input under a capability for epoch e round 3 and one for epoch e + 1 round 0, into leave_dynamic. It panics with "expected a singleton stamp" before this branch's fix and passes with it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
frankmcsherry
force-pushed
the
ddir-egraph
branch
from
September 2, 2026 12:02
3a962de to
411b3b5
Compare
…ount, threshold route by time A batch is shipped under the set of capabilities it retires, so a message of batches carries several timestamps when two epochs retire together. The operators that turn batches back into records forwarded the records under that same set, which is how a multi-element stamp reached `leave_dynamic` and anything else that reads a message's time as one. `StampRouter` (collection.rs) gives one capability per element of a message's stamp and routes each record to the first element at or before its time (a batch's times are at or beyond one of the capabilities it retired under, so every record has one). It is applied at every seam that makes records from batches: the vec `as_collection` (`flat_map_batches`), `count`, `threshold`, the columnar `as_recorded_updates`, and DDIR's corgi `as_collection` (a chunk under one capability goes whole, as before; under several it is split by row). With that, a collection's messages carry one timestamp each; only streams of batches carry sets. Not routed: `join_with_tactic`'s output, whose tactic builds containers the operator cannot split by time (the fix belongs in the tactics, a builder per element of the fresh batch's set), and `as_container`, which forwards opaque containers. `leave_dynamic` keeps holding a capability per element, so it is safe either way. tests/dynamic.rs: a hand-built batch under capabilities for epoch 0 round 3 and epoch 1 round 0, through `as_collection` into an operator that reads each message's time as one, lands its records in their two epochs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
tests/programs/egraph_math.ddp: equality saturation over a term DAG with declared terms (Num/Var/Add/Mul/Neg), the congruence-closure core of arrange_idem, and node-CREATING rules: commutativity mints the mirror term, constant folding mints the value, plus x+0, x*1, x*0, and double negation. Identity is exact — signatures and terms are keys — and a minted node's name is the hash of its canonical form, with a `collisions` export that must stay empty. Extraction is a cost fixpoint (cheapest term per class) and `best` names the node achieving it. tests/egraph.rs runs the program against a small Rust equality-saturation engine with the same rules on six random DAGs: the partition of the input nodes, every class cost, and the absence of collisions all agree. The program also joins the backend gate (vec vs corgi at 1-4 workers and over serializing channels). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
interactive/src/egraph: reify a scope into a term table (one node per linear step; projections, predicates, reducers, labels, and child-scope bodies interned to opaque ids; child scopes optimized first and entered as opaque leaves), run `optimize.ddp` — congruence closure with unordered Concat children plus arrange idempotence, and a cost-fixpoint extraction — on the vec backend, and rebuild the scope from the chosen node per class, folding single-consumer chains of steps back into Linear operators. tests/egraph_opt.rs: on every corpus program the result evaluates like the original and matches Scope::optimize's operator count exactly (reach 11, scc 32, tour 31, ...); on the explanation rewrites it does better: reach 168 -> 143 by hand, 140 by e-graph; scc 647 -> 519 by hand, 516 by e-graph, with identical outputs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
… facts
The optimizer prices each node by an op weight times one plus a row width:
the width of the rows an Arrange or a Reduce holds (its input's), the width
of the rows anything else produces. Widths come from a forward pass over the
scope given the root inputs' widths (`optimize(program, &source_widths)`),
through projections, reducers, joins, feedback vars, and child scopes.
Rules now come in two kinds. `optimize.ddp` states arrange idempotence over
the term table alone. The host instantiates the rules that need a fact
about a scalar operator, and hands them to the program as asserted
equalities (input 2):
* demand pushdown — a reducer that reads no values (count, distinct)
equals itself over a key-only projection of its input; a join that
reads only some fields of an input's values equals itself over that
input narrowed to those fields (below the input's arrange), reading
them at their new positions;
* join commutativity — a join equals the join of its inputs swapped,
reading them swapped; congruence then merges the two spellings.
`egraph::scalar` is the interface those rules need of the scalar language,
answered for DDIR's terms: widths after an operator, which fields of a row
a term reads (`Demand`), rewriting a term to read a narrowed row or renamed
rows, and the narrowing step itself. It is meant to grow by the same kind
of question — fuse two operators, extract a common subexpression, report
the equalities an operator implies — rather than by any syntax.
tests/egraph_opt.rs now compares costs under the model instead of op
counts (a narrowing is one more op and cheaper): corpus and explain
programs still evaluate identically; the e-graph matches or beats the
hand-written optimizer everywhere (adt 30 -> 28, tour 229 -> 223 from the
count/distinct narrowings; explain(reach) 553 -> 548, explain(scc) 1543 ->
1538); and three focused tests show the narrowing before a count, the
narrowing of a join input below its arrange, and two joins written both
ways around merging into one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
…h it Tree extraction (the cost fixpoint in optimize.ddp) counts a shared subterm once per use, so it will narrow a join input that another join needs whole and pay for a second arrangement of it. Optimal DAG extraction is NP-hard; the host now hill-climbs from the tree optimum on the weight of what the choices materialize from the roots (each reached class once): a move changes one reached class's node and is taken when the whole DAG gets strictly lighter, which also rules out cyclic choices. A test shows the shared arrangement kept. `Program::optimize` now runs the e-graph, with the root inputs' widths unknown (rows priced by operator alone, so nothing narrows); the new `optimize_with_widths` takes them, and the ddir driver passes its inputs' arities. `Scope::optimize` remains as the hand-written optimizer, which the e-graph program itself is optimized by (it cannot optimize itself), and which the tests compare against. The whole interactive suite runs through the e-graph unchanged. On the explanation rewrites it takes 20-35 ms in release (170-370 ms in debug) against under a millisecond hand-written. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
…to a fixpoint Concatenation is associative: a Concat's signature now lists the classes of its leaves through nested Concats (a static closure in optimize.ddp), so (a + b) + c and a + (b + c) are one class, and the rebuild splices a single-consumer nested Concat into its parent, the way it folds a chain of steps into one Linear. Filter pushdown, host-instantiated from scalar facts: a filter equals the projection below it over the filter composed through that projection (`scalar::compose`: every $0[i]/$1[j] replaced by the field's term, through spreads when the widths are known); the same filter below a negation or an entry; the concatenation of the filter over each part; and, over a join, the join with the filter composed through its projection on whichever input alone the composed predicate reads, below that input's arrange. For any of that to be chosen the price has to see rows, not operators: each node is now priced by the host as weight x (1 + width) x a relative volume, where inputs carry a base volume and a filter halves what it sees (an estimate, there to order the alternatives a rule proposes). The program takes the price as given. Rules now apply to the nodes they mint too, to a fixpoint (a minted node is structurally below what it matched), with nodes interned by signature so re-minting an existing node yields it; a filter therefore sinks as far as its predicate can be expressed. Tests: nested concatenations however associated become one three-part Concat; a filter after a join sinks to the input it reads, below the arrange and through the projection that built the input; a filter after a concatenation sinks into each part. The corpus and explain programs still evaluate identically and cost no more than the hand-written optimizer's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
…extraction's alone `Extraction::Random(seed)` walks from the tree optimum through a few hundred random moves, each kept if the choices stay acyclic, whatever it costs. The test runs it over every corpus program and both explanation rewrites, four seeds each: every program evaluates identically, through alternatives the cost never picks (swapped joins, pushed filters, narrowed inputs). Saturation and the rules are the same whatever is chosen; the price is a preference, not a premise. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK
frankmcsherry
force-pushed
the
ddir-egraph
branch
from
September 2, 2026 12:16
411b3b5 to
a8ccf74
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #855 (the first four commits here are that PR; merge it first and this diff shrinks to the e-graph).
What.
Program::optimizenow runs an equality-saturation optimizer written in DDIR (interactive/src/egraph/optimize.ddp) over each scope of a program, innermost first, and rebuilds the scope from the extraction.Scope::optimize, the hand-written pass, remains: the e-graph program is optimized by it (it cannot optimize itself), and the tests compare against it.The e-graph. Each scope is reified at the grain of one node per linear step, with everything an operator closes over (a projection, a predicate, a reducer, a child scope's body) interned to an opaque id, and each node priced by the host. The program computes congruence closure (a concatenation's children are the unordered multiset of its leaves through nested concatenations, so commutativity and associativity are free), arrange idempotence, and a tree-cost extraction. The host instantiates the rules that need a fact about a scalar operator and hands them to the program as asserted equalities: demand pushdown (a reducer or join that reads only some of an input's fields equals itself over that input narrowed to them, below the input's arrange), join commutativity, and filter pushdown (a filter equals itself composed through the projection, negation, concatenation, or join below it). Rules apply to the nodes they mint too, to a fixpoint.
egraph/scalar.rsis the whole interface to the scalar language: widths after an operator, which fields a term reads, rewriting a term to a narrowed or renamed row, composing a predicate through a projection.Extraction and cost. The cost model lives entirely in extraction; saturation never sees a price. A node costs weight × (1 + row width) × a relative row volume (inputs carry a base volume, a filter halves it: an estimate there to order alternatives, nothing more). The program's tree-cost optimum is refined by the host against the DAG cost, what the choices materialize from the roots with sharing counted once, by hill-climbing (optimal DAG extraction is NP-hard).
Extraction::Randomwalks random acyclic choices instead, and the test shows every corpus and explanation program evaluates identically under it.Results. On the eight corpus programs the e-graph matches the hand-written optimizer's operator count exactly; where the new rules fire it costs less (adt, tour). On the explanation rewrites it finds three more merges than the hand-written pass on each (reach 168 -> 143 by hand, 140 here; scc 647 -> 519 vs 516), purely from phase ordering. Outputs are identical before and after everywhere. The whole interactive suite runs through the e-graph. The pass takes 20-35 ms in release on the explanation programs (about 40x the hand-written pass; roughly a third is Debug-string interning in the host and half is one DD evaluation per scope, both addressable).
Also here: the arithmetic e-graph program (
egraph_math.ddp) with its Rust reference oracle, the exploration this grew from, and the ddir driver passing its inputs' widths so the narrowing rules can fire.🤖 Generated with Claude Code
https://claude.ai/code/session_012k2GSwxmvD2LvckkoXi6GK