GH-47393: [C++][Acero] Support for multi threaded input: SortedMergeNode Version 2 - #51141
Open
gitmodimo wants to merge 4 commits into
Open
GH-47393: [C++][Acero] Support for multi threaded input: SortedMergeNode Version 2#51141gitmodimo wants to merge 4 commits into
gitmodimo wants to merge 4 commits into
Conversation
added 4 commits
September 2, 2026 14:14
Contributor
Author
|
@ursabot please benchmark |
Member
|
Benchmark runs are scheduled for commit 8470111. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete. |
Contributor
Author
|
Since codebench doesn't seem to work here are my local benchmark run on i9-14900K:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
SortedMerge uses a dedicated worker thread and assumes input batches arrive in logical order. This prevents safe threaded execution and leaves backpressure, completion, stopping, and error propagation split across separate execution paths. The earlier implementation in #47394 added per-input sequencing and backpressure while retaining the existing worker thread; this PR is a new version that addresses #47393 with a single executor-managed implementation.
What changes are included in this PR?
SerialSequencingQueues restore logicalExecBatch::indexorder for threaded, serial, and compile-time no-thread builds.ExecBatchpayloads until direct materialization with Arrow builders. This supports scalar payloads and bypasses Boolean slices with non-zero offsets reported in [C++][Acero] materializeColumn for boolean type overflow #48072.Are these changes tested?
Yes. All tests pass in threaded and
ARROW_ENABLE_THREADING=OFFbuilds. Coverage includes signed keys crossing zero, scalar payloads, deterministic sequencing under jittered delivery, and downstream backpressure, resume, and stop.Are there any user-facing changes?
SortedMerge now supports concurrent input delivery, upstream and downstream backpressure, scalar payloads. Inputs that violate their declared ascending order return an
Invalidstatus.No API changes.