feat(realtime): support shredding-aware realtime reads & refactor rt read

[lxy-9602]

Purpose

Linked issue: #158

This change moves realtime offset handling from RealtimeStore plugins into the Paimon read framework and adds shredding-aware realtime reads for both append-only and primary-key tables.

The main changes are:

• Materialize _REALTIME_OFFSET before append batches are passed to RealtimeStore.
• Remove offset_begin from RealtimeStore::CreateQueryReaders.
• Remove RealtimeQueryContext::enable_predicate_pushdown; callers now decide whether to pass a predicate.
• Add RealtimeOffsetBatchReader to:
  • filter rows using the actual _REALTIME_OFFSET values and a left-closed, right-open visible range;
  • intersect the result with the input bitmap;
  • remove _REALTIME_OFFSET before passing data downstream.
• Add RealtimeStoreReadPipeline, shared by append-only and primary-key query paths:
  • derive the physical schema requested from RealtimeStore;
  • apply offset filtering first;
  • convert physical Store data into the requested logical schema.
• Support shared-shredding MAP reads from realtime memory:
  • selected-key MAP output;
  • selected-key STRUCT access;
  • recursive selected-key filtering for nested fields.
• Support VARIANT access from realtime memory:
  • read shredded physical fields;
  • assemble logical VARIANT access results;
  • use the binary fallback for fields absent from the configured shredding schema.
• Reuse KeyValueDataFileRecordReader for primary-key realtime data.
• Separate primary-key reader creation into CreateForQuery and CreateForCommit.
• Add explicit primary-key schema helpers:
  • CreateWriteSchema includes _VALUE_KIND, _SEQUENCE_NUMBER, and _REALTIME_OFFSET;
  • CreateLogicalSchema excludes _REALTIME_OFFSET.
• Share MAP selected-key utilities between file reads and realtime reads.

The resulting query pipeline is:

RealtimeStore reader
-> RealtimeOffsetBatchReader
-> physical-to-logical conversion
-> table-specific reader processing

Disk and realtime-memory data now produce the same logical output for shared-shredding MAP and VARIANT projections.

Tests

Added or updated coverage for:

• RealtimeOffsetBatchReader
  • filtering with an input bitmap;
  • filtering without an input bitmap;
  • removing _REALTIME_OFFSET;
  • rejecting nullable offset batches;
  • rejecting unsupported outer-reader access.
• RealtimeStoreReadPipeline
  • selected-key MAP output;
  • selected-key STRUCT access;
  • VARIANT shredded-field access and binary fallback.
• Primary-key realtime readers
  • write and logical schema layouts;
  • query offset filtering;
  • query and commit reader creation;
  • reordered transport fields;
  • nested values and invalid input.
• Append-only and primary-key integration tests
  • union reads with shared-shredding selected MAP keys;
  • union reads with MAP access STRUCT output;
  • union reads with VARIANT access across disk and realtime memory.

API and Format

This changes the RealtimeStore plugin contract:

• CreateQueryReaders no longer receives offset_begin.
• RealtimeQueryContext::enable_predicate_pushdown is removed.
• Append-mode Store write schemas and batches include _REALTIME_OFFSET.
• Query readers return the requested physical fields; Paimon applies offset filtering and physical-to-logical conversion.

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)