Skip to content

docs: refresh the arrangements concept page - #38599

Draft
ggevay wants to merge 1 commit into
MaterializeInc:mainfrom
ggevay:gabor/arrangements-concept-refresh
Draft

docs: refresh the arrangements concept page#38599
ggevay wants to merge 1 commit into
MaterializeInc:mainfrom
ggevay:gabor/arrangements-concept-refresh

Conversation

@ggevay

@ggevay ggevay commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Motivation

get-started/arrangements.md is linked from 13 pages (the EXPLAIN PLAN operator table among them) as the explainer for arrangements, and its core claims no longer match the system: it presents arrangements as "the mechanisms that maintain materialized views" and cross-query arrangement sharing as automatic ("builds the index once ... sharing the required resources across all queries"). Only an index's arrangement is shared; a materialized view's output goes to storage and a view is compiled into every dataflow that uses it. Its index-key advice dates from source primary keys pulled from a Confluent schema, and its links point at retired /ops/ paths.

Description

Same title, menu entry, alias and ## Arrangements anchor; the content is rewritten around what the code does (adapter/src/optimize/dataflows.rs import_into_dataflow, transform/src/join_implementation.rs, compute_state.rs TraceManager):

  • Intro and dataflows: a dataflow exists for an index, a materialized view, a SUBSCRIBE, and a SELECT not served from an index; it reads inputs from storage or from an index's arrangement.
  • Where arrangements come from: indexes, and inside dataflows (join inputs by the join columns, aggregation input and output by the group key, the TopK / MIN/MAX hierarchies), with a pointer to the operator reference.
  • What is shared: only an index's arrangement; a materialized view is read from storage; a view used by five dataflows is computed and arranged five times unless indexed (once within a single dataflow).
  • Arrangement size: one update per (data, time) seen, folded by compaction to one record per current row, size ≈ records × bytes per record, transient old-plus-new batches right after a large change; the taxi histogram example stays. Hydration named as the memory peak.
  • Analyzing: EXPLAIN PHYSICAL PLAN, EXPLAIN ANALYZE, mz_dataflow_arrangement_sizes, the dataflow troubleshooting page.
  • Choosing index keys: an index is a full copy arranged by its key; reuse requires the exact key expression; default key = a known unique key, else all columns (as CREATE INDEX documents).
  • Type casting, now with the mechanism: an implicit cast keys the arrangement on the cast expression, so an index on the plain column is read as a full scan and a second arrangement is built; index the cast expression or align the types. Verified on v26.39.0: cast_t1.a int4 = cast_t2.b int8 with an index on cast_t1(a) gives *** full scan *** plus an ArrangeBy keyed on integer_to_bigint(#0{a}); with CREATE INDEX ON cast_t1 (a::int8) both indexes are used directly.
  • Links: /ops/troubleshooting/ (now an alias of the ingest troubleshooting page) and /ops/optimization/ replaced by their current targets, the "Deployment" entry that pointed at optimization dropped, the blog URL updated to where it redirects.

Verification

Prose only. hugo builds cleanly; every link target and anchor on the page exists. Behavior claims checked against the code paths named above and on the v26.39.0 emulator.

@ggevay ggevay added the A-docs Area: documentation label Sep 1, 2026
The page said arrangements are "the mechanisms that maintain materialized
views" and that Materialize "builds the index once and maintains it in
memory, sharing the required resources across all queries". Only an
index's arrangement is shared: a materialized view writes its output to
storage, a view is compiled into each dataflow that uses it, and every
other arrangement is private to its dataflow. The page now says where
arrangements come from, what is shared, and how size and compaction
relate, and its index-key and type-casting advice matches how the
optimizer reuses arrangements (exact key expression match; an implicit
cast keys the arrangement on the cast expression). Stale /ops/ links,
the wrong "Deployment" link and the redirected blog URL are fixed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ggevay
ggevay force-pushed the gabor/arrangements-concept-refresh branch from a99e1d0 to 8e17c8b Compare September 1, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-docs Area: documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant