[DNM] spike: host storage objects on the compute timely cluster

[antiguru]

Motivation

Explores unifying the storage and compute timely clusters by migrating the objects instead of the runtime, the reverse of the protocol-first approach in #37091. The wire protocols and controllers stay byte-identical, so the change is confined to clusterd and the cluster-side crates. The goal of this draft is CI signal on the unified topology, not review.

Description

clusterd builds a single timely cluster and serves the storage protocol from a guest StorageState embedded in the compute worker loop. Timely requires all workers to construct dataflows in the same order, and storage already funnels all dataflow construction through its internal command sequencer while compute funnels it through the command channel. The change merges the two into one sequencing lane carrying both command kinds, so one definitive cross-worker order covers compute and storage construction alike. Storage-local behavior such as suspend-and-restart and replica-side resume-upper computation is unchanged.

Default on in this draft to exercise CI. MZ_UNIFIED_CLUSTER=0 falls back to separate clusters. The mz-compute -> mz-storage dependency edge is newly allowed in ci/test/lint-deps.toml.

Validated locally: source, materialized view, and index co-hosted on one cluster; concurrent creation storms on multi-process replicas; restarting an existing catalog into the unified topology; environmentd kill and recovery mid-storm. Untested so far: Kafka sinks, oneshot ingestion, and controller reconnects without a process restart, which is part of what this CI run is for.

---

Posted by Claude Code.

[antiguru]

Path to productizing

The spike validates the mechanism: storage objects render, run, and recover on the compute timely cluster with the wire protocols and controllers byte-identical. Smoke, concurrent creation storms on multi-process replicas, a topology swap of an existing catalog, environmentd kill and recovery, and source suspend-and-restart all pass locally, and the PR test pipeline is green up to the items below. What remains falls into decisions, validation, and productionizing.

Decisions

• Isolation semantics. The one substantive CI failure was cluster-controller.td: a blocking compute dataflow (an mz_sleep gate) on a shared worker starves co-hosted storage objects, including their command processing, and storage's single-replica scheduler has no liveness signal and can pick the wedged replica. Options are liveness-aware ingestion scheduling, class-based dataflow placement onto a second in-process runtime, or accepting and documenting the coupling. This decision shapes stage 1 and should come first.
• The mz-compute -> mz-storage dependency edge, previously denied in ci/test/lint-deps.toml. Either bless it or move the hosting glue behind a trait implemented in clusterd.
• Retirement schedule for the two-cluster topology. The deletion payoff (storage server loop, log bridge, second Timely cluster) only lands when the legacy path dies.

Validation

• A controller reconnect without a process restart, exercising guest reconciliation over live dataflows, and a compute reconciliation while storage dataflows are installed. Both need a dedicated test.
• The full nightly suite, in particular the upgrade, restart, and kill-clusterd checks matrices, zippy, parallel workload, and feature benchmarks for the shared-pool performance delta. Nightly runs so far were consumed by CI checkout outages.

Productionizing

• Replace the environment variable with an orchestrator-passed flag wired from a system parameter, with a staged rollout and revert path. Mixed fleets are safe since the protocol is unchanged.
• Derive the worker park deadline from actual storage maintenance and statistics intervals instead of the fixed 100ms cap, which costs idle wakeups.
• Audit dashboards and alerts keyed on the storage Timely metric series and thread names, which disappear with the second cluster.
• Clean up the spike scaffolding: the guest state handoff in the worker loop, the internal-command constructor shims, and the duplicated blocking reconcile wrapper.

Process

• A design doc covering the single-lane ordering argument, the isolation decision, and the deletion plan.
• Land as a stack of focused PRs: cluster guest client, storage refactors, command-lane generalization, compute-side hosting, clusterd wiring plus flag and test updates.

---

Posted by Claude Code.