Skip to content

feat(file_index): add indexer contracts and outer-format composition - #764

Merged
JingsongLi merged 2 commits into
apache:mainfrom
QuakeWang:file-index-contracts
Sep 1, 2026
Merged

feat(file_index): add indexer contracts and outer-format composition#764
JingsongLi merged 2 commits into
apache:mainfrom
QuakeWang:file-index-contracts

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

Closes: #34

Bitmap and Bloom implementations already existed, but callers still had to match identifiers and construct concrete readers and writers directly. This leaked implementation details into future scan and writer integrations.

This change adds crate-private contracts and built-in dispatch so outer-format index payloads can be composed without depending on concrete index implementations.

Brief change log

  • Add a common FileIndexWriter contract and built-in Bitmap/Bloom factory.
  • Build per-column readers from outer-format metadata and schema.
  • Handle nested map columns, empty indexes, type/options validation, and unknown identifiers.
  • Add component E2E coverage for multiple columns and indexes.

Tests

  • cargo fmt --all -- --check
  • cargo test --locked -p paimon --all-targets --features fulltext,vortex

API and Format

Documentation

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Comment thread crates/paimon/src/file_index/file_index_format.rs Outdated

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 0aba451 into apache:main Sep 1, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement FileIndex

2 participants