feat(knowledge): permission-aware knowledge bases behind a feature flag - #7385
feat(knowledge): permission-aware knowledge bases behind a feature flag#7385waleedlatif1 wants to merge 96 commits into
Conversation
- Build/Search mode switcher in the chat composer; Search mode swaps the suggested actions for Connect Sim Search rows that open the OAuth modal - Search tab beside Integrations and Skills: a personally scoped catalog of knowledge-base connectors with connect-in-place rows and a connected-credential page without sharing or members - <source> special tag contract: inline citation chips at the citation point plus a scrolling, right-faded sources strip under the reply - chip `shape` variant so a fully round chip is chip chrome, not a class override - keep the connector catalog out of the workflow editor's module graph and record the new routes in the boundary baseline
- validate <source> urls by parsing them and requiring a host, so a malformed value never renders a dead citation link - gate Search-mode suggestions on deployment OAuth availability, sharing the predicate with the Search catalog - match connected credentials across a service's additional provider ids (Salesforce sandbox) and count them in connector telemetry - collect footer sources from the rendered text segments, covering a block-less message's fallback text and excluding subagent lanes - use a distinctive citation-link sentinel and an absolute import for SuggestedActions - teach the email tokens transcription test the composed chip geometry
Adds document.acl (sorted access tokens, fast default '{ws}') with a partial
GIN index and a token-shape CHECK, source_modified_at, the members-mode
columns and lease on knowledge_connector, a real FK on credential_id (left
NOT VALID until script migration 0011 has remapped legacy account ids), and
three tables: knowledge_connector_member, knowledge_document_observation,
knowledge_connector_member_sync_log.
The document contract stops passing unknown keys through so the new column
and storageKey never reach clients.
Pure building blocks for document-level access: the token vocabulary and its single sorting/validation site, the subject token derived from a managed credential row, the read-side predicate (scalar binds, literal ARRAY), the per-principal scope resolver with a lazy memoising provider, the branded system scope, and the members-mode sync limits. Nothing consumes them yet.
Every document loader, chunk list, tag usage, search leg, metadata lookup,
and knowledge-base file serve now takes a required access scope and applies
the shared ACL predicate. Context resolvers attach a lazy, memoised scope
provider derived from the acting principal and load documents through it,
so a document the caller may not read is absent from the first read. Raw v1
routes and legacy helpers resolve the same scope; the outbox handler is the
one branded system-scope caller. Vector legs run inside a transaction that
enables pgvector's relaxed iterative scan (with a 42704 fallback) so a
narrow ACL still fills topK. Forked workspaces copy documents as '{ws}'.
Behaviour is identical today: every row carries the default '{ws}' token.
…engine Splits the content sync engine into four modules without changing behaviour: sync-lock (lease predicates, heartbeat, and a SyncRunLease the stages run under), sync-persistence (storage and document-row writes), sync-primitives (the pure decision functions plus the six orchestration stages: listing pass, owned-corpus load, classification, hydrate/persist/dispatch, deletion reconciliation, and the stuck-document sweep), and sync-engine (lock acquisition, terminal state, and executeSync composed from the stages with its original closures). The stages take the token, hydration, and lease as inputs so a second engine can run the same guards under a different lease. Tests keep every assertion; only import paths moved with the symbols.
Registers Google Drive as a Credential Group provider (provider list, adapter registry, and the Google managed-OAuth policy), marks the Drive and Confluence connectors as permission-scoped listings with the config fields that would cap them, and extends the Credential Group policy with a knowledge_connector principal and a credential_group:OptionId condition so a group can grant one connector the credentials collected under one option. The policy codec gains one KnowledgeConnectorCredentialAccess statement per option, decoded by SID; workflow-access edits now carry connector grants through untouched. Knowledge-side helpers grant and revoke the binding under the policy's revision CAS, mint a member token only when the policy names the connector for the credential's option (audited per access), enumerate an option's credentials in every state for membership reconciliation, and validate a members-mode binding before anything is touched. Nothing calls them yet.
Adds the engine that crawls a knowledge connector once per Credential Group member with that member's own token and derives each document's ACL from who observed it. A run takes the member lease (mutually exclusive with the content engine's), finishes any pending access rewrite, mirrors the option's credentials onto member rows (active or suspended, tokens rewritten, departed members dropped), then claims due members one at a time until its budget ends: each member's listing is admitted into a first-writer-wins union, every listed document is hydrated once through its observers' tokens, observations are recorded, removals follow only a full, complete, non-suspect listing, the affected ACLs are rematerialised from the observation graph, and the document lifecycle tombstones the unobserved, resurrects the re-observed, and purges the long-unobserved under the lease. The run re-dispatches itself while members remain due. Around the engine: a member queue mirroring the content dispatch (token, lease, lost-hand-off release), a Trigger.dev task on its own queue, a cron scheduler that reclaims stale leases and lost dispatches, closes orphaned run logs, sweeps members whose crawls stopped, and dispatches due connectors, and the Docker and Helm cron entries. The content engine and its dispatch refuse members-mode connectors, workspace-mode syncs restore the workspace ACL on completion, document writes carry the ACL each mode needs, the hard-delete guard can verify the member lease, and the Drive and Confluence connectors report a scope the member cannot reach as an empty listing rather than an error. Nothing enters members mode yet.
Wires the members-mode engine to the connector surface. A connector can be created in members mode (admin only) bound to a Credential Group option, or switched between modes through a new admin-only access operation and route: the switch takes the connector's content lease, rewrites the documents' ACLs to the new mode's default in bounded batches (finishing under the member run when the request budget runs out), grants or revokes the option's credentials in the group's policy, flips the mode, and queues the first run of the new engine. Members-mode source edits refuse listing caps and queue a member run, a manual sync routes to the member queue, keep-documents deletion is refused, and deletion revokes the grant. Contracts carry the access mode, the binding, the member-sync state, the member summary, and member run logs; the member scheduler and dispatch honour a paused connector.
Add an Access section to the add and edit connector modals so a workspace admin can sync a permission-scoped connector once per credential-group member. The connector card shows the per-member badge, member sync status, and a member run history in place of the content sync history.
Knowledge bases report whether a live connector syncs per member, and the knowledge block's selector explains that such documents depend on who triggers the run.
Connectors may open a change feed over a member's view of the source. Google Drive reads changes.list, where a lost share arrives as a removal, so a member's access is withdrawn without waiting for a full listing; Confluence filters its CQL listing by lastModified for incremental content refreshes. Each member stores where their feed resumes, and a member with an open feed is relisted fully only weekly as a check.
Search fuses the full-text and semantic legs unless a caller asks for vector-only, at every surface: internal, v1, v2, the knowledge block, and Sim's own knowledge tools. Connector documents now record when the source last changed them, and a recently modified document edges past a stale one of similar relevance.
…rden the stack Add the knowledge-member-access feature flag (workspace allowlist and platform admins via AppConfig; KNOWLEDGE_MEMBER_ACCESS off-AppConfig). It gates members mode at creation and switching, the member engine, and the hybrid-by-default retrieval with its recency boost; the connector modals read it from the workspace host context. The knowledge block's retrieval mode gains an Automatic option so the server's default applies. An admin with no credential group for the connector's provider can create one from the Access field; the group is named after the connector and members are invited from Settings. Review fixes: manual-only connectors no longer re-dispatch member runs forever; ACL materialisation is idempotent so a run that died mid-way is repaired; incomplete listings retry at once only when they can resume; suspended members keep their observations; a scope the member cannot reach closes their change feed; the workspace ACL restore runs inside the completion lock and only for workspace-mode connectors; the mode switch grants before rewriting and flips in one transaction, clearing listing caps; minting checks group, option, and enrollment status; connector document lists, bulk select-all operations, and tag usage stats apply the access scope; the connector card reads the member engine's status; the edit modal applies an access switch on its own instead of racing the settings save.
…osed when the flag is off Members mode is gated by the workspace at creation, switching, in the member engine, and in the workspace host context, so no gate can open a mode another will refuse; the flag's admin clause now affects only the retrieval default, and search passes the signed-in person rather than a billing owner or key creator. Member tokens count only where the feature is on, so turning it off hides member-scoped documents on the next read. New and reactivated members are due at once so a drain re-dispatches until everyone has been listed. Leaving members mode flips first under the lease with the rewrite marked pending, then rewrites; a failed move between options of one group restores the previous option. The edit modal closes after a switch and cannot re-choose per-member where the flag is off; connector document exclusion honours the caller's scope.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Choosing per-member access no longer asks the admin for anything: the server reuses the workspace's one credential group for the connector's provider or creates one, invites every workspace member to connect, and each member run invites people who joined since. A member's only step is to connect: the knowledge base page shows a banner for every per-member connector they have not connected, and a new enroll route mints their enrollment link on demand so the invitation email is never needed. The connector list reports the viewer's membership for that banner. The permission-scoped listing CI test now compares OAuth provider ids, which is what the runtime checks. The knowledge page module baseline is re-recorded for the provisioning graph.
Choosing per-member access provisions a group named after the connector and the first member run invites everyone in the workspace, under the run's lease; the request itself sends nothing, so nobody gets two links. A member connecting queues a member run for every connector on that option, and the knowledge base page shows what each viewer must do — connect, reconnect, verify their email, or nothing — opening enrollment in a new tab and refreshing until they are connected. Every gate reads one availability check (flag and Credential Groups). When it is off, readers get no member token, the engine waits instead of suspending anyone, and the field cannot re-choose per-member access. Member tokens go only to current workspace members with a live group and option. A connector that just entered members mode never tombstones its documents before a member has listed, the stale sweep leaves paused and manual connectors alone, a disabled member sync is re-enabled by re-applying its binding, leaving members mode forces a full content sync, and by-id bulk operations honour the caller's scope. Deleting a per-member connector always takes its documents.
One read returns every per-member connector in the workspace with where the viewer stands on each, so a surface outside the knowledge base can ask them to connect. The member sync status enum now has one home in lib/knowledge/types.
Collapses the stack's two migrations into one, 0318_permission_aware_knowledge, generated from the schema on top of staging's 0317 and written in the expand/contract form the migration gate requires. The knowledge_connector credential_id foreign key is deferred until script migration 0011 has remapped legacy rows in production. Also makes the on-connect member sync dispatch reachable: the grant persistence returned before it ran.
The Search tab lists the knowledge bases whose connectors sync per member under "Shared with you", with where the viewer stands on each and the same one-click connect the knowledge base page offers. One hook opens enrollment in a new tab and refreshes the surface until the viewer is connected; the knowledge base banner uses it too.
…connector Google Slides, Docs, Forms, Calendar, Gmail, Notion, Jira, Jira Service Management, Box, Dropbox, monday, ClickUp, Asana, Salesforce, Linear, DocuSign, and Zoom list only what the caller's own account can read, so each now declares its listing caps and can sync per member. A shared listing-scope error lets a connector report a folder, space, board, or calendar the member cannot reach as a complete listing of nothing; Notion and Asana report it from their typed errors, Dropbox from its 409, and the Atlassian connectors also from an unreachable site. Google Docs and Forms join the credential-group providers so their accounts can be collected per person.
- The credential-group policy canonicaliser carries knowledge-connector statements through instead of rejecting a third statement - Google Docs and Box report an unreachable scope so a member's access is withdrawn rather than retried forever; Notion leaves members mode because its page access is granted to the shared integration bot - Provisioning reuses a Credential Group only when other members-mode connectors already sync through it, never a group curated for something else; the fast dispatch on connect accepts the same statuses as the sweep - Enrollment opens its tab inside the click so popup blockers cannot swallow it, and the membership poll stops after ten minutes - Session-only routes use session auth; members mode stores the cap-stripped config; one liveness rule for members and managed bindings - Drop dead code: the access barrel, system.ts and rank.ts folded into their siblings, unused ACL helpers, the unread change_cursor_at column, stale documentation, and the engine/provisioning/queue import cycle
- Bump the chart version for the member-sync cron and secret - Pass an access scope to getDocuments in the list-convention test and resolve a knowledge scope only for knowledge-base file reads - Drop memoisation nothing observes, read the enrollment error from the mutation, hoist the empty connector list, use the default Cancel variant, let Badge own its gap, and keep the sidebar lit on the Search tab
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 350 files
Not reviewed (too large): apps/sim/lib/knowledge/connectors/sync-engine.ts (~2,879 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
Note: This PR contains a large number of files. cubic selects up to 200 of the highest-priority eligible files for this review, so some files may not have been reviewed.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Fix all with cubic | Re-trigger cubic
…ace and carry only chat params on the handoff
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
…s disabled on every mint, workflow runs included
There was a problem hiding this comment.
No issues found across 351 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Not reviewed (too large): apps/sim/lib/knowledge/connectors/sync-engine.ts (~2,879 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
Note: This PR contains a large number of files. cubic selects up to 200 of the highest-priority eligible files for this review, so some files may not have been reviewed.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Re-trigger cubic
…t, follow the search query in a chat, restore a queued message's mode, and seed the base list
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 352 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Not reviewed (too large): apps/sim/lib/knowledge/connectors/sync-engine.ts (~2,879 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
Note: This PR contains a large number of files. cubic selects up to 200 of the highest-priority eligible files for this review, so some files may not have been reviewed.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Re-trigger cubic
Summary
Permission-aware knowledge bases: a workspace admin can set a connector to sync per member, so each person's searches, document lists, and downloads return only what their own account can open. Nothing is asked of IT and nothing is asked of the admin beyond flipping the switch: a Credential Group is provisioned in the background, every workspace member is invited to it, and each person connects their own account from the knowledge base page or the Search tab. The connector then crawls the source once per member with that member's token, and a document's access list is the set of members whose crawl returned it.
Everything ships behind the
knowledge-member-accessfeature flag (KNOWLEDGE_MEMBER_ACCESSoff-AppConfig), and only where Credential Groups are available. Members mode is judged by the workspace allowlist alone, everywhere (creation, switching, the engine, the UI, the read scope); theadminEnabledclause additionally opens the hybrid retrieval default to a platform admin in any workspace. With the flag off every surface behaves exactly as today, and turning it off hides member-scoped documents on the next read without touching any data.This PR also carries #7376 (Sim Search) merged in, with the Search tab gaining a "Shared with you" section for per-member connectors.
What changes
Access control (always on, no behaviour change until a connector enters members mode)
document.acl text[]with a partial GIN index; every read path (search legs, document/chunk loaders, file serving, v1/v2 routes, connector document lists, bulk select-all, tag usage) appliesacl && <caller tokens>with scalar binds. Existing rows backfill to{ws}with a fast default.Members mode (flag)
0319_permission_aware_knowledge(expand-only, after staging's0318managed-MCP migration):knowledge_connector_member,knowledge_document_observation,knowledge_connector_member_sync_log, connector columns for mode, binding, member-sync state, and pending rewrites. Theknowledge_connector.credential_idFK is deferred (contract-pendingmarker) because legacy rows still holdaccount.idvalues; script migration0011remaps them first.background/knowledge-connector-member-sync, cron*/5): lock CAS, membership reconciliation from the option's credentials, invitations for anyone who joined the workspace since the last run, a time-boxed drain loop claiming members withFOR UPDATE SKIP LOCKED, first-writer-wins union, hydration through observers, observations, idempotent ACL materialisation, tombstone/resurrect/purge lifecycle, stuck sweep, immediate re-dispatch while members are due.knowledge_connectorpolicy principal +credential_group:OptionIdcondition; per-optionKnowledgeConnectorCredentialAccessstatements, which the db-package canonicaliser now carries through; minting audited asCREDENTIAL_ACCESSED.changes.list(a lost share arrives as a removal, so access is withdrawn without a full listing); Confluence CQLlastModifiedfor incremental content refreshes. Feed members relist fully weekly as a check.Connectors that can sync per member (their listing is scoped by the caller's own permissions, and a scope the member cannot reach is treated as a complete empty listing so their access is withdrawn): Airtable, Asana, Bitbucket, Box, ClickUp, Confluence, DocuSign, Dropbox, Gmail, Google Calendar, Google Chat, Google Docs, Google Drive, Google Forms, Google Meet, Google Sheets, Google Slides, Jira, Jira Service Management, Linear, Microsoft Excel, Microsoft Teams, monday.com, OneDrive, Outlook, Salesforce, SharePoint, Zoom. Their listing caps (
maxDocs,maxFiles, ...) are hidden and cleared in members mode, since a cap per member would suppress removals. Credential Groups gain managed OAuth policies for the Google services above, for Microsoft (subjectoid, tenanttid, id token verified against the common discovery keys, email accepted only when the token asserts it verified), and for Bitbucket (subjectaccount_id, primary confirmed email). Deployment notes: the Bitbucket OAuth consumer must have the Email permission or enrollment fails fast; Microsoft work accounts enroll only when the app registration emitsxms_edov, and personal accounts are refused.Retrieval (flag)
source_modified_atand a bounded rank-based recency boost lets a fresh document edge past a stale one. The knowledge block gains an "Automatic" retrieval mode so the server's default applies.UI
Access section in the add/edit connector modals (Workspace / Per member); the edit modal applies an access switch on its own control, with re-enable for a disabled binding. Connector cards show the per-member badge, member sync status, "Updating access" while a rewrite is pending, and a member run history.
Chat, Search mode (Sim Search): under the composer a Sources strip lists every source a person can connect themselves, as chips. One click opens OAuth; a source that needs a site or space (Confluence, Jira, Jira Service Management, Asana, ClickUp, Salesforce) asks for it once in a small modal on the connect that creates it, and everyone after clicks straight through. The first connect of a source in a workspace provisions the "Sim Search" knowledge base and a per-member connector for it and requires a workspace admin (an admin turns a source on once; a member who is first is told to ask an admin); connecting queues the person's member run, so indexing starts on its own, and the chip shows Indexing, then how many documents that person can read. Sources whose listing is not permission-scoped do not appear. Searching lists what the person may read as result rows (source icon, title linking back to the document, source app, author, updated date, and the passage around the first query term with the terms in bold, skipping an email's header block), one per document, with a header that says how many and that the search ran as them, an indexing note while a source is still indexing, source and date filters once the list is long and mixed, hover actions (copy link, Summarize), arrow-key navigation, and an Answer with Sim button that hands the query to the Assistant. The query stays in the box after a search and lives in the URL as
q, so a search is a shareable link; emptying the box returns to the sources. Existing chats never open in Search. The agent's knowledge tool returns each result's title, link, connector, and modified time and is told to cite with<source>tags carrying a snippet, so a prose answer ends with the same cards.Search tab: the full catalog with each source's state, the same inline setup, and a "Shared with you" section for per-member connectors in other knowledge bases. Connecting opens the enrollment link in a tab opened inside the click (popup-safe) and polls the membership for up to ten minutes.
Chat, Assistant mode: the composer's third mode beside Build and Search, the same vocabulary as Glean. The mode lives in the URL as
mode(Build is the clean URL), so a refresh, back, forward, or shared link lands in the same mode as Glean's separate routes do; the new-chat handoff carries the query string across the path swap. The previous mode store is gone. The Home route seeds the knowledge-base list, so a first Assistant question typed before the list arrives is still grounded; a question handed to the Assistant from a result empties the composer as a send does; a chat's composer follows the live search query; and a queued message re-enters the composer in the mode it was written in. Search lists the matching documents; Assistant makes the query a normal turn of the agent in the same chat UI, with the searched knowledge bases attached the way an@mention attaches one, so the answer is grounded in exactly the documents Search shows and ends with the same<source>cards. Summarize and Answer with Sim on a result switch to Assistant, and a follow-up in the chat stays an Assistant turn. An Assistant turn sendsmode: ask, which the chat handler forwards instead of hard-codingagent, and the turn carries a built-in skill: search the attached bases with the knowledge tool first, cite each claim, say so when nothing is found, reach for a connected integration only when the indexed sources cannot answer (live data, or an action the person asked for), and keep suggested follow-ups to questions. The reply's cited documents sit behind a counted button in the action row, beside copy and fork, opening a popover of one row per document; the prose keeps its inline citation chips.Chat, Credential Group credentials: the agent can now act as the signed-in person through a credential they collected under their own Credential Group enrollment. Those credentials appear in the agent's credentials file typed
managed_oauth, and a Chat tool call proves itself with a copilot delegation naming the user and scoped to the one credential. Authorization evaluates the group policy's existing actor statement (the enrollee may use the credential under their own enrollment) with no workflow, so nothing the model passes can reach another enrollee's credential; the tool-scope checks and audit are unchanged, and the Go agent needs no change.Deliberately not in this PR
listing_modecolumn.ts_headlinesnippets; copilot query expansion; title-prefixed embeddings (needs a re-embed).drive.readonly; a read-only member scope is a follow-up.searchModegets hybrid + recency where the flag is on (was always vector). Intended, and called out here because the versioned API's default becomes workspace-dependent.Demo setup
knowledge-member-access: { workspaceIds: ["<demo workspace id>"] }to the hostedfeature-flagsAppConfig document and start a fast deployment; the demo workspace also needscredential-groupson and an Enterprise plan.0319(expand-only) and script migration0011, and deploy the Trigger.dev tasks from this branch (the member run is the newknowledge-connector-member-synctask; without it every dispatch fails and the cron retries).q, so the search can be shared.Verification
bun run type-check(apps/sim, packages/db, packages/testing),bun run check:audits,bun run check:migrations, biome — clean. The tool-registry boundary baseline was re-recorded for the knowledge module's new imports./cleanuppasses (effects, memo, callback, state, React Query, emcn design, url-state, comments) plus seven line-by-line audits (schema/enforcement, member engine, credential-group plumbing, contracts/routes/hooks, UI/UX, and two full LOC sweeps). Defects found and fixed there: mutation responses failed contract validation (viewerMembership); bulk enable/disable wrote to documents outside the caller's scope; the staleness sweep tombstoned deferred connectors; Gmail and Calendar kept a 500-item cap after the switch; Jira/ClickUp/Asana/Linear/Dropbox classified "scope unavailable" on the wrong status; a Credential Group serving a members-mode connector could be deleted under it; the Search tab showed "No connectors found" beside member rows; a connector could not be rebound when exactly one other group existed; the iterative vector scan ran for every workspace; the member-scoped hint ignored the flag.apps/simsuite green after the swarm (2900 files / 39,900 tests), plus type-check on apps/sim, packages/db, packages/testing,check:audits,check:migrations,drizzle-kit check, and biome./cleanuppasses and the four/simplifyangles over the Sim Search surface, applied in pass order: the connect flow lives in the enrollment hook and is shared by the strip and the Search page, one availability gate serves every members-mode refusal, setup fields are precomputed on the catalog, the chip gained a trailing adornment slot, a search is a shareable link (q) that is dropped when the box empties or the mode leaves Search, and the processor takes its source access as one object./cleanupover the results and sources: result rows use the chat surface's row rhythm with hairlines,OverflowTexttitles and meta lines, an icon button for Copy link, a ghost Summarize, keyboard-revealed actions, and a linkless document rendered in the same row; the source and date filters live in the URL besideq; the member-connector query is gated withenabledand cancelled before the optimistic queue write.