Skip to content

perf: Remove an aggregate whose GROUP BY covers a unique key - #24789

Draft
Dandandan wants to merge 2 commits into
apache:mainfrom
Dandandan:feat/eliminate-aggregate-superkey
Draft

perf: Remove an aggregate whose GROUP BY covers a unique key#24789
Dandandan wants to merge 2 commits into
apache:mainfrom
Dandandan:feat/eliminate-aggregate-superkey

Conversation

@Dandandan

@Dandandan Dandandan commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • Closes #.

Built on #24787, which this needs to be correct. The first commit here is
that fix; review it there. Without it, an aggregate above a list unnest would
be removed on the strength of a key the unnest has already invalidated.

Rationale for this change

When the GROUP BY expressions include a unique key of the input, every group
holds exactly one row. The aggregate then produces one output row per input
row, like a projection, and every aggregate function returns the value of that
single row.

DataFusion did not use this. With lineitem keyed by (l_orderkey, l_linenumber):

SELECT l_orderkey, l_linenumber, sum(l_quantity)
FROM lineitem GROUP BY l_orderkey, l_linenumber

still built a hash table over all 60M rows to put one row in each group.

What changes are included in this PR?

EliminateGroupByConstant already simplifies an aggregate based on what its
GROUP BY contains, matches on Aggregate, and runs at the right point, so the
check goes there rather than in a new rule and costs no extra plan traversal.
Its name is now a little narrow for what it does; happy to rename it if
reviewers prefer.

The aggregate becomes a projection:

Projection: l_orderkey, l_linenumber, CAST(l_quantity AS Decimal128(25, 2))
--TableScan: lineitem

The single-row value is known for min, max, sum, avg, first_value and
last_value (the argument, cast to the aggregate's return type where that
differs) and for count (1, or 0 when the argument is NULL). Any other
aggregate keeps the aggregate node.

DISTINCT, ORDER BY and IGNORE NULLS make no difference to a group of one
row. A FILTER does: it can exclude the row and leave the aggregate with no
input at all, which is a different value for every function, so those keep the
aggregate.

Two cases are not eligible:

  • an empty GROUP BY, which returns one row for an empty input where a
    projection returns none;
  • grouping sets, which do not group by every expression at once, so a key among
    them proves nothing.

The rule also removes a SELECT DISTINCT over a unique key, which the planner
turns into an aggregate with no aggregate expressions.

Benchmarks

TPC-H SF10 with the keys declared, median of 5, three interleaved rounds:

without the key with the key
sum(q) over GROUP BY l_orderkey, l_linenumber 751 ms 31 ms 24x
SELECT DISTINCT c_custkey, c_name, c_address 51 ms 19 ms 2.7x

Both return identical results.

Of the benchmark suites, only TPC-DS q54 changes. dfbench declares primary
keys for both TPC-H and TPC-DS, and with those declared q54 drops a DISTINCT
over customer's primary key. Its runtime is unchanged at SF1 (50.4 ms before
and after, median of 7 over three rounds), since the aggregate removed is over
a dimension table. Every other TPC-H, TPC-DS and ClickBench plan is byte for
byte identical, checked by diffing all 184 optimized plans against main.

Are these changes tested?

Yes. functional_dependencies.slt gains a section covering the rewrite for
each supported aggregate, NULL handling for all of them, extra grouping
expressions beyond the key, and the four cases that keep the aggregate (FILTER,
grouping sets, an unsupported aggregate, and grouping by a non-key).

The rule fires in existing tests in aggregate.slt, distinct_on.slt,
explain.slt, functional_dependencies.slt and group_by.slt; those expected
plans are updated and no expected result changes. The full sqllogictest suite
(504 files) passes.

Are there any user-facing changes?

Queries that group by a unique key, or take a DISTINCT over one, no longer run
an aggregate. Results are unchanged.

`Unnest::try_new` copied the input's functional dependencies unchanged, with
the comment "We can use the existing functional dependencies". That is not
true for a list unnest, which turns one input row into several. A determinant
that occurred once in the input can occur many times in the output.

Optimizer rules that read those dependencies then produce wrong results. With
a declared key:

  CREATE TABLE t_list (k INT, vals INT[], PRIMARY KEY (k))
    AS VALUES (1, [10, 20, 30]), (2, [40]);
  CREATE TABLE t_join (k INT) AS VALUES (1), (2);

  SELECT u.k, u.v FROM (SELECT k, unnest(vals) AS v FROM t_list) u
  JOIN t_join j ON u.k = j.k;

returns 2 rows instead of 4, because `eliminate_join` sees `u` as unique on
`k` and rewrites the inner join into a semi join, which drops the repeated
rows. Without the PRIMARY KEY the same query returns 4.

The dependency still holds in the weaker sense: all rows produced from one
input row share the determinant, so it determines the same columns. Downgrade
it to `Dependency::Multi` rather than dropping it, which keeps it useful and
stops it being read as a uniqueness guarantee.

Unnesting a struct produces one row per input row, so those dependencies are
left as they are.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgqwvctZdvR1ZCz2hbkEJC
@github-actions github-actions Bot added logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Aug 30, 2026
When the GROUP BY expressions include a unique key of the input, every group
holds exactly one row. The aggregate then produces one output row per input
row, like a projection, and each aggregate function returns the value of that
single row.

DataFusion did not use this. With lineitem keyed by (l_orderkey, l_linenumber):

  SELECT l_orderkey, l_linenumber, sum(l_quantity)
  FROM lineitem GROUP BY l_orderkey, l_linenumber

still built a hash table over all 60M rows, one row per group.

This extends `EliminateGroupByConstant`, which already simplifies an aggregate
based on what its GROUP BY contains and runs at the right point, so it costs no
extra plan traversal. The aggregate becomes a projection:

  Projection: l_orderkey, l_linenumber, CAST(l_quantity AS Decimal128(25, 2))
    TableScan: lineitem

On TPC-H SF10 with the key declared, summing that result goes from 751 ms to
31 ms. A DISTINCT over a unique key, which the planner turns into an aggregate
with no aggregate expressions, goes from 51 ms to 19 ms.

The single-row value of each aggregate is known for min, max, sum, avg,
first_value and last_value (the argument, cast to the aggregate's return type
where that differs) and count (1, or 0 when the argument is NULL). Anything
else keeps the aggregate. DISTINCT, ORDER BY and IGNORE NULLS make no
difference to one row, but a FILTER can exclude it and leave the aggregate with
no input at all, so those are left alone.

An empty GROUP BY is not eligible: it returns one row for an empty input, which
a projection would not. Grouping sets are not eligible either, since they do
not group by every expression at once.

Of the benchmark suites, only TPC-DS q54 changes: it drops a DISTINCT over
customer's primary key. Its runtime is unchanged at SF1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SgqwvctZdvR1ZCz2hbkEJC
@Dandandan
Dandandan force-pushed the feat/eliminate-aggregate-superkey branch from 3943ad7 to 9988cb9 Compare August 30, 2026 07:40
@2010YOUY01
2010YOUY01 self-requested a review August 30, 2026 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant