Adopt backend-issued claim write tokens and opt-in fenced session streams - #4792
Draft
adityavkk wants to merge 1 commit into
Draft
Adopt backend-issued claim write tokens and opt-in fenced session streams#4792adityavkk wants to merge 1 commit into
adityavkk wants to merge 1 commit into
Conversation
…ed session streams
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets the Durable Streams backend, rather than the Agents Server's in-memory store, be the mint for claim-scoped write tokens, and adds an opt-in mode in which entity session streams are write-fenced by the backend itself. Addresses the distribution gaps called out in #4286 and gives the pull-runner path from #4299 the same write-authority story as webhooks.
Problem
#4286 notes that claim write tokens are currently held in memory on one server: validity is lost on restart, not shared between instances in a distributed deployment, and not coupled to the claim lease or heartbeat. It also notes the enforced invariant is only "while an active claim is present, only that claim token may write" — not yet "only the current claimed worker may write", because validation happens solely in the forwarding server while the Durable Streams backend accepts whatever the server forwards under its own identity.
Approach
The Durable Streams Write Fencing extension (spec) has the backend mint a write token per subscription claim and deliver it as additive
write_tokenfields on the existing §7 surfaces: the webhook wake notification, the pull-claim response, and every non-done ack. This PR makes the Agents Server adopt those tokens wherever they appear, keepingClaimWriteTokenStorethe single validation authority while the backend becomes the mint:internal-router): awrite_tokenon the subscription webhook body is held per wake and adopted when the runtime's claim callback mints — the runtime receives the backend's token as itswriteToken.runners-router, RFC: Run Agents Anywhere — Local Runners, Worker Pools, and Sandboxes #4299): awrite_tokenon the claim response is held the same way, so the runner's claim callback adopts it identically.internal-router+ runtimeprocess-wake): the backend re-mints the token on each ack (its TTL tracks the claim lease); the server adopts the refresh and surfaces it aswriteTokenon the heartbeat response, and the runtime picks it up so appends from long-running activations keep a live token. The store keeps the immediately-previous token valid across a same-consumer refresh so an append already in flight is not rejected; a different consumer's claim still evicts everything.fencedSessionStreamsserver option, envELECTRIC_AGENTS_FENCED_SESSION_STREAMS): entity session streams are created (and forked) withWrite-Fence: true, and runtime appends forward the presented token asWrite-Tokenplus aWrite-Fence: trueclass assertion when proxied to the backend. The backend then rejects deposed or lapsed writers itself — durable across server restarts, shared across instances, and lease-coupled, which is the mechanism Agents auth follow-up: claim-scoped write tokens, shared-state auth, and distributed validity #4286 asks for without building a second distributed store inside the Agents Server. The assertion also means a lost token is a loud 401 downstream rather than a silent write under the forwarding server's identity.Key invariants
fencedSessionStreamsoff (the default), no new headers are sent anywhere.Compatibility
write_tokenfields are optional on every surface; a base Durable Streams server ignoresWrite-Fence/Write-Tokenentirely (pinned by a test that creates, forks, and appends fenced againstDurableStreamTestServer).internal-router.ts): token adoption is data-driven while fencing is opt-in, so a pre-adoption runtime pointed through this server at a token-minting backend loses write authority once heartbeat refreshes rotate the token past the store's one-refresh grace — upgrade runtimes before the backend starts minting; every other skew combination degrades to today's behaviour.Verification
New coverage: backend-token adoption vs minted fallback in the store and through both wake-delivery routes; no delivered-token entry left behind by wakes auto-acked or rejected before any claim; previous-token grace across a heartbeat refresh and eviction on takeover; header forwarding on the append path with the flag on, off, and for shared-state; an end-to-end heartbeat refresh through a real server; and the runtime adopting a refreshed token for subsequent producer appends.
Files changed
packages/agents-server/src/claim-write-token-store.ts:mintaccepts a backend-issued token; delivered-token holding (recordDelivered/takeDelivered); previous-token grace on same-consumer refresh.packages/agents-server/src/stream-client.ts: Write Fencing header constants (spec-linked);write_tokenon the claim response;writeFenceopt-in oncreate/fork.packages/agents-server/src/routing/internal-router.ts: acceptwrite_tokenon webhook bodies and heartbeat ack responses; adopt at the mint site (a delivered token is held only once its wake is actually forwarded, so auto-acked or rejected wakes leave no entry).packages/agents-server/src/routing/runners-router.ts: hold the claim response'swrite_tokenfor the runner's claim callback.packages/agents-server/src/routing/stream-append.ts: forwardWrite-Token+Write-Fenceon validated entity appends when fencing is enabled.packages/agents-server/src/{entity-manager,runtime,standalone-runtime,server}.ts: thefencedSessionStreamsoption, threaded to the manager with an env fallback; fenced create/fork of session streams.packages/agents-runtime/src/process-wake.ts: refreshwriteTokenfrom heartbeat responses..changeset/agents-backend-issued-write-tokens.md(patch, both packages).