fix(scan): preserve snapshot id for empty data evolution index plans - #258
fix(scan): preserve snapshot id for empty data evolution index plans#258wangyong9999 wants to merge 3 commits into
Conversation
| std::optional<int64_t> snapshot_id = core_options_.GetScanSnapshotId(); | ||
| if (snapshot_id) { | ||
| return snapshot_id; | ||
| } |
There was a problem hiding this comment.
Please validate the snapshot before returning the empty plan. With a pre-supplied empty GlobalIndexResult, a nonexistent scan.snapshot-id (for example, 999) is currently copied directly into a successful plan. Please resolve/load the snapshot here and add a regression test for nonexistent or expired snapshot IDs.
There was a problem hiding this comment.
This behavior seems to predate the PR: the global-index scan already appears to ignore tag- and timestamp-based time travel and fall back to the latest snapshot. What changes here is that the new resolver makes this more visible by attaching that latest snapshot ID to an empty plan. Since global-index time travel does not seem to be supported at the moment, would it make sense to reject these modes explicitly instead of silently scanning and reporting the latest snapshot?
There was a problem hiding this comment.
Thanks, good catches — addressed in aeeb00a.
The pre-supplied empty-result fast path now loads the resolved snapshot through SnapshotManager before constructing the empty plan. Nonexistent, expired, unreadable, or malformed snapshots therefore propagate an error, while a table with no snapshot still returns the null-snapshot empty plan. The regression coverage includes both snapshot 999 and a previously valid snapshot whose file was removed.
Tag- and timestamp-based time travel now returns a single NotImplemented error (Global index scan does not support time travel) once planning actually obtains a Global Index result. This covers internally evaluated results and pre-supplied nonempty results, while preserving the ordinary batch-scan fallback when no usable index exists. The tests cover both options and the unindexed fallback boundary.
2 participants
Purpose
Linked issue: none.
A global-index miss on a non-empty DataEvolution table returned a correct empty row set but lost the snapshot used by the index scan. A null snapshot is otherwise reserved for a table with no snapshot.
Snapshot and data-flow changes: the explicit or latest snapshot ID is resolved before the internal Global Index scan, and a zero-split miss plan carries that same ID. For a pre-supplied empty
GlobalIndexResult, the fast path now loads the resolved snapshot throughSnapshotManagerbefore constructing the plan. Missing, expired, unreadable, or malformed snapshots therefore propagate an error instead of producing a successful plan with an unvalidated ID. A table with no snapshot still returns the null-snapshot empty plan.Time-travel boundary: Global Index planning does not yet support tag- or timestamp-based time travel. Once evaluation or a pre-supplied result actually yields a range-based Global Index result, planning now returns
NotImplemented: Global index scan does not support time travelinstead of silently binding that result to the latest snapshot. If no usable index exists and the evaluator returns no result, the ordinary batch scan still handles tag/timestamp time travel normally.The change adds no shared state or concurrency mechanism. Existing snapshot-manager I/O and error propagation are reused. The only compatibility change is that previously successful but semantically invalid Global Index plans now fail explicitly. Expected benefits are accurate empty-plan snapshot metadata, rejection of stale snapshot IDs, and prevention of Global Index/data-manifest snapshot mismatches.
Tests
paimon-global-index-testandpaimon-core-testsuccessfully with GCC 8.3 in Debug mode.paimon-global-index-test: 138/138 passed.paimon-core-test: 1863/1863 passed.FileFormat/GlobalIndexTest.TestDataEvolution*: 54/54 parameterized cases passed.DataEvolutionBatchScanTest.*andTableScanTest.TestNoSnapshot: 4/4 cases passed.git diff --checkpassed.API and Format
No public API, ABI, storage-format, or protocol change. The modified scan helpers are private implementation details and no class data member is added.
Documentation
No new feature or user-facing configuration; no documentation change is required.
Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)