Skip to content

Adopt backend-issued claim write tokens and opt-in fenced session streams - #4792

Draft
adityavkk wants to merge 1 commit into
electric-sql:mainfrom
adityavkk:feat/agents-backend-write-tokens
Draft

Adopt backend-issued claim write tokens and opt-in fenced session streams#4792
adityavkk wants to merge 1 commit into
electric-sql:mainfrom
adityavkk:feat/agents-backend-write-tokens

Conversation

@adityavkk

Copy link
Copy Markdown

Lets the Durable Streams backend, rather than the Agents Server's in-memory store, be the mint for claim-scoped write tokens, and adds an opt-in mode in which entity session streams are write-fenced by the backend itself. Addresses the distribution gaps called out in #4286 and gives the pull-runner path from #4299 the same write-authority story as webhooks.

Problem

#4286 notes that claim write tokens are currently held in memory on one server: validity is lost on restart, not shared between instances in a distributed deployment, and not coupled to the claim lease or heartbeat. It also notes the enforced invariant is only "while an active claim is present, only that claim token may write" — not yet "only the current claimed worker may write", because validation happens solely in the forwarding server while the Durable Streams backend accepts whatever the server forwards under its own identity.

Approach

The Durable Streams Write Fencing extension (spec) has the backend mint a write token per subscription claim and deliver it as additive write_token fields on the existing §7 surfaces: the webhook wake notification, the pull-claim response, and every non-done ack. This PR makes the Agents Server adopt those tokens wherever they appear, keeping ClaimWriteTokenStore the single validation authority while the backend becomes the mint:

  • Webhook wakes (internal-router): a write_token on the subscription webhook body is held per wake and adopted when the runtime's claim callback mints — the runtime receives the backend's token as its writeToken.
  • Pull-runner claims (runners-router, RFC: Run Agents Anywhere — Local Runners, Worker Pools, and Sandboxes #4299): a write_token on the claim response is held the same way, so the runner's claim callback adopts it identically.
  • Heartbeats (internal-router + runtime process-wake): the backend re-mints the token on each ack (its TTL tracks the claim lease); the server adopts the refresh and surfaces it as writeToken on the heartbeat response, and the runtime picks it up so appends from long-running activations keep a live token. The store keeps the immediately-previous token valid across a same-consumer refresh so an append already in flight is not rejected; a different consumer's claim still evicts everything.
  • Opt-in fencing (fencedSessionStreams server option, env ELECTRIC_AGENTS_FENCED_SESSION_STREAMS): entity session streams are created (and forked) with Write-Fence: true, and runtime appends forward the presented token as Write-Token plus a Write-Fence: true class assertion when proxied to the backend. The backend then rejects deposed or lapsed writers itself — durable across server restarts, shared across instances, and lease-coupled, which is the mechanism Agents auth follow-up: claim-scoped write tokens, shared-state auth, and distributed validity #4286 asks for without building a second distributed store inside the Agents Server. The assertion also means a lost token is a loud 401 downstream rather than a silent write under the forwarding server's identity.

Key invariants

  • When the backend supplies no token, behaviour is byte-for-byte today's: the store mints its own token exactly as before (stated in a comment at the adoption site).
  • With fencedSessionStreams off (the default), no new headers are sent anywhere.
  • Command and shared-state appends never carry the fenced-class assertion; only validated entity-stream appends do.
  • The write token is never part of the runner notification body; it reaches the runtime only through the claim/heartbeat callback, as today.

Compatibility

  • Default off; no configuration change is required anywhere.
  • The new write_token fields are optional on every surface; a base Durable Streams server ignores Write-Fence/Write-Token entirely (pinned by a test that creates, forks, and appends fenced against DurableStreamTestServer).
  • Chronicle is a Durable Streams server implementing the extension, usable as a backend for the fenced mode.
  • One version-skew caveat (documented as a compatibility matrix at the adoption site in internal-router.ts): token adoption is data-driven while fencing is opt-in, so a pre-adoption runtime pointed through this server at a token-minting backend loses write authority once heartbeat refreshes rotate the token past the store's one-refresh grace — upgrade runtimes before the backend starts minting; every other skew combination degrades to today's behaviour.

Verification

pnpm --filter @electric-ax/agents-server exec vitest run test/claim-write-token-store.test.ts test/stream-append.test.ts test/subscription-webhooks-routing.test.ts test/runners-router.test.ts test/stream-client.test.ts   # 65 passed
pnpm --filter @electric-ax/agents-server exec vitest run test/server-claim-write-token.test.ts    # 13 passed (docker backend)
pnpm --filter @electric-ax/agents-server exec vitest run test/stream-client-fork.test.ts          # 4 passed
pnpm --filter @electric-ax/agents-runtime exec vitest run test/process-wake.test.ts               # 55 passed
pnpm --filter @electric-ax/agents-server test     # full suite: 513 passed, 41 skipped (docker backend + built server UI)
pnpm --filter @electric-ax/agents-runtime test    # full suite: 982 passed, 64 skipped
pnpm --filter @electric-ax/agents-server typecheck && pnpm --filter @electric-ax/agents-runtime typecheck
pnpm --filter @electric-ax/agents-server stylecheck && pnpm --filter @electric-ax/agents-runtime stylecheck
GITHUB_BASE_REF=main node scripts/check-changeset.mjs   # ✅ covers @electric-ax/agents-runtime, @electric-ax/agents-server

New coverage: backend-token adoption vs minted fallback in the store and through both wake-delivery routes; no delivered-token entry left behind by wakes auto-acked or rejected before any claim; previous-token grace across a heartbeat refresh and eviction on takeover; header forwarding on the append path with the flag on, off, and for shared-state; an end-to-end heartbeat refresh through a real server; and the runtime adopting a refreshed token for subsequent producer appends.

Files changed

  • packages/agents-server/src/claim-write-token-store.ts: mint accepts a backend-issued token; delivered-token holding (recordDelivered/takeDelivered); previous-token grace on same-consumer refresh.
  • packages/agents-server/src/stream-client.ts: Write Fencing header constants (spec-linked); write_token on the claim response; writeFence opt-in on create/fork.
  • packages/agents-server/src/routing/internal-router.ts: accept write_token on webhook bodies and heartbeat ack responses; adopt at the mint site (a delivered token is held only once its wake is actually forwarded, so auto-acked or rejected wakes leave no entry).
  • packages/agents-server/src/routing/runners-router.ts: hold the claim response's write_token for the runner's claim callback.
  • packages/agents-server/src/routing/stream-append.ts: forward Write-Token + Write-Fence on validated entity appends when fencing is enabled.
  • packages/agents-server/src/{entity-manager,runtime,standalone-runtime,server}.ts: the fencedSessionStreams option, threaded to the manager with an env fallback; fenced create/fork of session streams.
  • packages/agents-runtime/src/process-wake.ts: refresh writeToken from heartbeat responses.
  • Tests as described above; .changeset/agents-backend-issued-write-tokens.md (patch, both packages).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant