refactor: separate aggregate group completion from input ordering - #24697
refactor: separate aggregate group completion from input ordering#24697xavlee wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24697 +/- ##
==========================================
+ Coverage 81.42% 81.62% +0.20%
==========================================
Files 1121 1123 +2
Lines 402142 409619 +7477
Branches 402142 409619 +7477
==========================================
+ Hits 327460 334371 +6911
- Misses 55484 55600 +116
- Partials 19198 19648 +450 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
f58e128 to
372429f
Compare
372429f to
d1d1d2f
Compare
d1d1d2f to
0267ad1
Compare
gene-bordegaray
left a comment
There was a problem hiding this comment.
approved with some non blocking suggestoins. Thank you @xavlee
fd7a377 to
e7f8b42
Compare
e7f8b42 to
0108b7c
Compare
|
Rereviewed after changes and everything looks good 👍 |
NGA-TRAN
left a comment
There was a problem hiding this comment.
Looks great. It is a pleasure to review. Nice explanation.
I wonder if we should add an attribute in the explain to show this property? Maybe in a follow-up PR? If it is not that invasive or very easy to review, maybe adding that in this PR?
| /// | ||
| /// For example, with `GROUP BY (a, b)`, `Partial(vec![0])` means all rows | ||
| /// for each value of `a` are contiguous, while an `(a, b)` tuple may recur | ||
| /// within that range. |
There was a problem hiding this comment.
Does this mean I have 2 keys (a, b) and data is sorted on (a) only?
| assert_eq!( | ||
| aggregate.group_completion_mode, | ||
| GroupCompletionMode::Partial(vec![0]) | ||
| ); |
| // This captures the behavior before #24438. When the source can declare | ||
| // `(key, time_bin)` group-contiguous, the corresponding case can use | ||
| // `EmissionType::Incremental`. | ||
| assert_eq!(aggregate.cache().emission_type, EmissionType::Final); |
Which issue does this PR relate to?
Rationale for this change
Aggregate execution currently derives both input-order metadata and the runtime mechanism for recognizing completed groups from
InputOrderMode.Completing a group incrementally is contingent on knowing that "after processing a group, we will never see this group again in this stream". As such, input ordering covers only a subset of valid group completion cases. A sorted input guarantees that after processing a group, we will never see it again; however, the general case shows that so long as all rows in a groups are contiguous, the group completion property is satisfied regardless of the input ordering.
Consider the following:
This PR introduces a new GroupCompletionMode that decouples the input ordering from group completion semantics. Currently, we directly map existing InputOrderMode to this property. In a follow up PR, we will read group contiguity expressions to evaluate this property.
What changes are included in this PR?
GroupCompletionMode::{None, Partial, Full}variants.GroupCompletionModefromInputOrderModeduringAggregateExecconstruction.GroupCompletionModethrough ordered aggregate tables, streams, and spill replay paths.InputOrderModefor required ordering, output ordering, andmaintains_input_ordermetadata.GroupOrdering::try_new(&InputOrderMode)as the public compatibility entry point.The ordering-derived conversion is:
Stack
Are these changes tested?
The characterization from #24737 remains
Linear,None,Final, and single-hash after the refactor.Are there any user-facing changes?
No.
GroupCompletionModeis crate-private, and the publicGroupOrderingconstructor is unchanged.Review this layer
View only this PR layer