Skip index write interface

[joacoc]

Rationale for this change

This PR contains the implementation for the skip index write interface. Based on the previous PR #8933, which already had a working example on a Bloom filter prototype. So, this PR removes the Bloom filter bits (except for the file/skip-index integration test) and consolidates the skip-index interface.

• Related to: Tracking Issue: Skipping Indexes on ZonedLayout #8901

What changes are included in this PR?

This PR carries over the following from #8933, with some changes:

• The SkipIndex trait, which provides an aggregate_fn and registers its required components with a session.
• WriteStrategyBuilder::with_field_zoned_options.
• The Bloom skipping-index integration test. The Bloom implementation itself is tracked in #9398.
• The relocation of the min/max statistics into a new aggregates dir, where other skipping-index aggregate functions can live.

This PR also adds the following notable changes:

• SkipIndexRef and SkipIndexSessionExt for simpler registration and handling.
• Updates to ZonedLayoutOptions and ZonedStrategy so that they store configured skipping indexes and resolve their aggregates while writing, rather than during configuration.
• Updates to StatsRewriteCtx and zoned pruning so rewrite rules can access the aggregate functions, and allow them to use the serialized aggregate options.

What APIs are changed? Are there any user-facing changes?

Yes. This PR contains the following user-facing changes:

1. WriteStrategyBuilder adds with_field_zoned_options, which configures zoned-layout options for a field.
2. SkipIndexSessionExt adds skipping-index registration methods to VortexSession.
3. ZonedLayoutOptions adds with_skip_index, which configures a skipping index for the zoned layout.

---

[codspeed-hq]

Merging this PR will regress 2 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 2 regressed benchmarks
✅ 2193 untouched benchmarks
⏩ 206 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	allocate_drop_arrow[0]	402.7 ns	456.9 ns	-11.86%
❌	WallTime	mul_u32_nonnull_avx512	5.6 µs	6.3 µs	-10.23%
⚡	WallTime	arrow_checked_add_u32_neon[16384]	20.3 µs	13.3 µs	+52.74%
⚡	Simulation	decompress[u64, (4000, 1024)]	85.6 µs	70.4 µs	+21.59%
⚡	WallTime	arrow_checked_add_u32_avx512[16384]	21.3 µs	17.6 µs	+21.11%
⚡	WallTime	add_shapes_neon[(128, PerRowPerRow)]	2 µs	1.8 µs	+10.64%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing joacoc:skip-index-writer-integration (97fa19c) with develop (4afdbd4)}

Footnotes

1. 206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[joacoc]

One part I'd highlight for review are the changes to rewrite.rs and pruning.rs.

I needed a way to pass the serialized aggregate fn options from the file to StatsRewriteRule during pruning. Passing the aggregate functions through StatsRewriteCtx does the work, but I’m not sure if there’s a better or more appropriate way, since I’m not too familiar with this.

The interface ended up like this:

let session = VortexSession::default();
let index = BloomSkipIndex::new(BloomOptions::default());

// Registration (enough for reads)
session.register_skip_index(&index);

// Configuration for writes
let strategy = WriteStrategyBuilder::default()
    .with_field_aggregates(field_path!(id), [index.aggregate_fn()])
    .build();

session
    .write_options()
    // Required because Bloom filters are not currently part of any edition
    .disable_editions()
    .with_strategy(strategy);

[joseph-isaacs]

What is the intended difference between this and ZonedLayout (

vortex/vortex-layout/src/layouts/zoned/mod.rs

Line 86 in 3451462

pub type ZonedLayout = Layout<Zoned>;

).

That is also a skip index.

[connortsui20]

@joseph-isaacs this hooks into zoned layout (it explains this in the description). I think this is generally the right direction, we want a nicer interface for aggregations that we know are going to be treated as indexes specifically and not just statistics