feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes - #3090
Open
feloy wants to merge 3 commits into
Open
feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes#3090feloy wants to merge 3 commits into
feloy wants to merge 3 commits into
Conversation
feloy
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
September 1, 2026 13:54
Collaborator
BlockedGator is blocked because PR #3090 currently has merge conflicts with Next action: @feloy, please update the branch from Gator metadata
|
johntmyers
added
the
gator:blocked
johntmyers
reviewed
Sep 2, 2026
johntmyers
left a comment
johntmyers
left a comment
Collaborator
There was a problem hiding this comment.
gator-agent
PR Review Status
The merge-conflict blocker from the previous head is resolved. The initial code review found two concrete blockers in the new VM proxy path: CA bundle staging can consume an unbounded special file and accepts certificates the guest will reject, and the published loopback recipe is incorrect for the supported QEMU/TAP backend.
Action required: @feloy, please address GATOR-dc1218b5-01 and GATOR-dc1218b5-02, then push an update.
Blocking findings:
GATOR-dc1218b5-01: bound and fully validate the host CA bundle before staging it.GATOR-dc1218b5-02: make the host-loopback guidance accurate for both libkrun/gvproxy and QEMU/TAP.
Carried findings:
- None
Gator metadata
- Validation: Project-valid implementation of linked issue #3088 with a clear VM corporate-proxy user path, threat-boundary analysis, docs, and E2E coverage.
- Docs: Updated, but the host-loopback contract is materially incorrect for QEMU/TAP.
- Checks: Current-head Branch Checks and Helm Lint gates are pending; required branch workflows have not been dispatched.
- E2E:
test:e2eis required for VM networking and credential flow; dispatch is deferred until review blockers are resolved. - Head SHA:
dc1218b51ac14f1315c1a633cae316ff45963549 - Base SHA:
7b64c5c88ece0878592160cb86da5e0fd1f64158 - Merge base SHA:
7b64c5c88ece0878592160cb86da5e0fd1f64158 - Patch ID:
e940265736870bac2a568c5f4c0be0ba145c2786 - Gator payload:
8 - Review mode:
initial - Previous reviewed SHA: none
- Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
johntmyers
added
gator:in-review
…dboxes The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and already merged: openshell-supervisor-network implements CONNECT chaining, NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as six argv-only flags. Podman gained the driver half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so VM sandboxes on proxy-only networks could not reach any destination requiring the proxy even when policy allowed it. The blocking piece was not proxy logic but delivery: the VM guest init script runs as PID 1 and execs a fixed supervisor command line, and libkrun's krun_set_exec receives an empty argv, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no environment fallback, and build_guest_environment merges user-supplied environment, so the guest env is not a safe transport either. Add a driver-authored argument file, mirroring the existing init.d manifest: the driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch and the guest reads it verbatim, one argument per line, appending it to every supervisor exec. It is written even when empty, which is what makes the channel unforgeable -- the upperdir always shadows the read-only image layer, so an image can neither supply its own arguments nor disable the operator's by omitting the file. Because both launch backends exec the same init script, this covers libkrun and QEMU without touching either. A microVM has no bind mounts or container secrets, so the credential and CA bundle are staged into the per-sandbox overlay the way the gateway JWT already is: credential root-only at 0600, CA at 0644, both rewritten every launch so a removed setting clears prior material, and both deleted with the sandbox state directory. This places the credential at rest in the overlay image on the gateway host, which differs from the Podman secret model and is documented as an explicit security consideration. Validation is fail-closed and shared: a new openshell_core::driver_utils::validate_upstream_proxy_settings holds the pairing rules the Podman driver established, and both the gateway and the driver call it so an invalid table names the offending key instead of surfacing as an opaque driver-readiness timeout. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only through host.openshell.internal; the guest to gateway callback is unaffected. Closes NVIDIA#3088 Signed-off-by: Philippe Martin <phmartin@redhat.com>
Two review findings on the corporate forward proxy support for microVM sandboxes. The driver read the operator's proxy_ca_bundle with an unbounded fs::read and accepted it on a substring match for the PEM BEGIN CERTIFICATE marker. A special file such as /dev/zero therefore grew driver memory without bound on every authorized sandbox create, and a PEM block holding invalid DER passed the host check but contributes no trust anchor in the guest, so every supervisor would fail after boot with an error attributed to the sandbox rather than to the setting. Move the read into openshell-core as read_upstream_proxy_ca_bundle_file: it reuses the credential reader's bounded-read path (non-regular files rejected on fstat, size capped, read bounded even if the file grows), then requires at least one anchor that RootCertStore::add_parsable_certificates accepts. The supervisor's own reader now delegates to it, so host acceptance and guest acceptance are the same function and cannot drift. The published host-loopback recipe was written for libkrun only. gvproxy NATs host.openshell.internal to the gateway host's 127.0.0.1, but GPU sandboxes run on the QEMU/TAP backend where that name resolves to the TAP host address and the driver's own nftables input chain accepts only the gateway port from the guest — no proxy on the gateway host is reachable there at any bind address, so an operator following the generic recipe lost all proxy-required egress while configuration validation succeeded. Scope the recipe to libkrun in every reference and reject a gateway-host proxy URL when a launch plan resolves to QEMU, naming the reason, instead of booting a sandbox whose policy-approved CONNECTs all time out. Signed-off-by: Philippe Martin <phmartin@redhat.com>
johntmyers
reviewed
Sep 2, 2026
johntmyers
left a comment
johntmyers
left a comment
Collaborator
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @feloy. I checked your replies against current head 38e83fad785099c25a56901d73dd08a5957f26a2: the shared bounded regular-file reader plus rustls trust-anchor validation resolves GATOR-dc1218b5-01, and the backend-scoped documentation and QEMU preflight resolve the original GATOR-dc1218b5-02 obligation. The follow-up review found one new blocker in that preflight: it does not identify the actual per-sandbox TAP host address and rejects gvproxy's unrelated private address instead.
Action required: @feloy, please address GATOR-38e83fad-01 and push an update.
Blocking findings:
GATOR-38e83fad-01: classify the selected QEMU TAP host address without rejecting an unrelated routable proxy address.
Carried findings:
- None; both prior findings are resolved.
Gator metadata
- Validation: Project-valid implementation of linked issue #3088.
- Docs: Updated for the libkrun/QEMU reachability split; implementation parity remains blocked by the new finding.
- Checks: Current-head Branch Checks and Helm Lint are pending; test dispatch remains deferred while review feedback is open.
- E2E:
test:e2eis required for VM networking and credential flow and will be dispatched after blockers are resolved. - Head SHA:
38e83fad785099c25a56901d73dd08a5957f26a2 - Base SHA:
74960ebfaeec4673885089ed995fad902459749f - Merge base SHA:
74960ebfaeec4673885089ed995fad902459749f - Patch ID:
a2aceafceaca70aa799bc9a7503cadbae7203036 - Gator payload:
8 - Review mode:
follow_up - Previous reviewed SHA:
dc1218b51ac14f1315c1a633cae316ff45963549 - Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
The gateway-host proxy guard added for the QEMU/TAP backend classified the wrong set of addresses in both directions. It ran at the top of configure_qemu_launch_plan, before the subnet allocation that settles plan.host_ip, so it could not compare against the address the guest actually reaches the host on. An operator pointing https_proxy at the sandbox's own TAP host address, such as 10.0.128.1, passed the check, and the driver's nftables input chain — which accepts only the gateway port from the guest — then dropped every policy-approved CONNECT, which is exactly the silent timeout the guard exists to prevent. In the other direction it rejected 192.168.127.254 unconditionally. That address is special only to libkrun/gvproxy; on QEMU/TAP it is an ordinary address that may be routable through the guest's masqueraded egress, so the guard refused a working configuration. Run the check after the launch plan's network allocation, on both the freshly-allocated and already-complete paths, and compare IP literals with that sandbox's selected TAP host. Loopback literals, localhost, and the documented host aliases that write_host_gateway_aliases seeds to the TAP host still classify as the gateway host, and the failure names the address. The gvproxy host-loopback constant returns to being a documentation anchor. Signed-off-by: Philippe Martin <phmartin@redhat.com>
|
Label |
Collaborator
|
/ok to test eb4f236 |
johntmyers
reviewed
Sep 2, 2026
johntmyers
left a comment
johntmyers
left a comment
Collaborator
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @feloy. I checked your detailed response against current head eb4f23630e44e1e8336412ea9fad3885bbb5ea3a: the proxy preflight now runs after the QEMU TAP host address is finalized, compares parsed addresses with that selected host, no longer rejects gvproxy's unrelated 192.168.127.254 address, and covers both launch-plan directions. This resolves GATOR-38e83fad-01; the follow-up review found no new blockers.
Blocking findings:
- No blocking findings remain
Carried findings:
- None;
GATOR-dc1218b5-01,GATOR-dc1218b5-02, andGATOR-38e83fad-01are resolved.
Gator metadata
- Validation: Project-valid implementation of linked issue #3088 with an explicit maintainer-requested phase.
- Docs: Updated for the VM corporate-proxy UX and backend reachability contract.
- Checks: Current-head Branch Checks, Helm Lint, and Branch E2E workflows are running.
- E2E:
test:e2eapplied; the copy-PR mirror was authorized for this head and Branch E2E run 33622651241 is in progress. - Head SHA:
eb4f23630e44e1e8336412ea9fad3885bbb5ea3a - Base SHA:
74960ebfaeec4673885089ed995fad902459749f - Merge base SHA:
74960ebfaeec4673885089ed995fad902459749f - Patch ID:
fc66764b47b57c726c06c8e9029441978d2bae30 - Gator payload:
8 - Review mode:
follow_up - Previous reviewed SHA:
38e83fad785099c25a56901d73dd08a5957f26a2 - Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:watch-pipeline
johntmyers
added
gator:watch-pipeline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gives the MicroVM compute driver the same operator-owned corporate forward-proxy surface the Podman and Kubernetes drivers already have, so VM sandboxes on proxy-only corporate networks can reach the destinations policy allows. The supervisor-side machinery from #1792 is reused unchanged; what this adds is the per-driver half — configuration, validation, a driver→guest argument channel, and credential/CA delivery into the guest.
Related Issue
Closes #3088
Changes
The argv channel (the one genuinely new mechanism). Podman and Kubernetes build the supervisor's command line in Rust. The VM guest init script runs as PID 1 and execs a fixed argv, and libkrun's
krun_set_execreceives an empty argument array, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have noenv =fallback, andbuild_guest_environmentmerges user-supplied environment, so the guest environment is not a safe transport either./opt/openshell/supervisor-argsinto the overlay upperdir on every launch; the guest init script reads it verbatim, one argument per line (no word splitting, globbing, or expansion), and appends it to every supervisorexec.init.dmanifest, which solves the same trust problem for guest init drop-ins.runtime.rsis untouched.Configuration.
[openshell.drivers.vm]acceptshttps_proxy,no_proxy,proxy_auth_file,proxy_auth_allow_insecure,proxy_connect_by_hostname, andproxy_ca_bundle, with matching--https-proxy…--proxy-ca-bundleflags andOPENSHELL_VM_*environment variables on the driver binary (needed independently, since the driver also runs as an external gRPC driver). The booleans travel as explicit values rather than presence flags so an operator's explicitfalsesurvives the gateway→driver hop and still trips the pairing checks. The keys are not inheritable from[openshell.gateway]and are rejected bytemplate.driver_config.vm.Shared validation, fail-closed. Added
openshell_core::driver_utils::validate_upstream_proxy_settingsholding the pairing rules the Podman driver established, called by both the gateway and the driver — without the gateway-side check an invalid table surfaces as an opaque driver-readiness timeout instead of an error naming the key. Podman and Kubernetes are deliberately left on their own copies to keep this PR scoped.Credential and CA delivery. A microVM has no bind mounts or container secrets, so both are staged into the per-sandbox overlay the way the gateway JWT already is — credential root-only at
0600, CA at0644, both at fixed/opt/openshellpaths, both rewritten every launch (so removing a setting clears material a previous launch staged into a preserved overlay), and both removed with the sandbox state directory.Docs.
docs/reference/gateway-config.mdx(MicroVM section),docs/reference/sandbox-compute-drivers.mdx,crates/openshell-driver-vm/README.md,architecture/sandbox.md, and a VM corporate-proxy troubleshooting section in thedebug-openshell-clusterskill.Security considerations for review
overlay.ext4on the gateway host, which differs from the Podman secret model. It is taken because the per-sandbox gateway JWT — a bearer token for the gateway — already travels this exact path, so the exposure is not new; it is documented inarchitecture/sandbox.mdand the config reference rather than left implicit. Alternatives considered and rejected as disproportionate: copying to tmpfs via an init drop-in and deleting from the overlay, or a new driver→guest vsock secret channel.host.openshell.internal(gvproxy NATs192.168.127.254to the host's127.0.0.1). Documented, and exercised by the e2e. The guest→gateway callback is unaffected and never traverses the proxy.Testing
mise run pre-commitpassesUnit — 29 new tests:
openshell-core: the shared validator (unsupported scheme, inline credentials, present-but-empty values, each auxiliary key without a proxy URL, the cleartext acknowledgement including thehttps://exemption, CA bundle valid with a plain-HTTP intercepting proxy) and the three new guest paths in the control-root test.openshell-driver-vm: argv builder emits guest paths and never the operator's host paths;Some(false)is not passed as a presence flag; the argument file renders empty when unconfigured and one-argument-per-line otherwise, preserving values containing spaces; newline/NUL values rejected; credential and CA validated with the supervisor's own validators and errors never echo file contents;build_guest_environmentemits no proxy setting and no user-supplied environment can forge an--upstream-*argument;template.driver_config.vmrejects all six keys;Debugredacts the proxy URL and credential path.openshell-server:spawnforwards the settings as driver argv and omits unset ones; invalid config is rejected before the driver starts, naming the key.E2E — new
e2e/rust/tests/vm_corporate_proxy.rsin thee2e:vmlane, the VM counterpart ofpodman_corporate_proxy.rs. Fixtures run as host processes (no container-runtime dependency for this lane) and are reached from the guest throughhost.openshell.internal, which is also what proves the documented host-loopback reachability rule. Three cases:CONNECT 192.168.127.254:<port> auth=ok, proving both the validated-IP CONNECT target and that the overlay-staged credential reached the supervisor (the proxy answers 407 without it). The policy-denied destination never appears in the proxy log at all, and a port-qualifiedno_proxydestination is reached while never touching the proxy.https://proxy trusted viaproxy_ca_bundle, with the CA minted by the fixture and recovered from its output.no_proxywithouthttps_proxystops the gateway from serving, with the offending keys named in its log.Verification performed on this branch:
mise run pre-commitcargo clippy --all-targets -- -D warnings(core, driver-vm, server, e2e)mise run test:rust(full workspace incl.openshell-server)mise run e2e:vmsmoke1,host_gateway_alias4,vm_gateway_start1,vm_corporate_proxy3smokepassing is a deliberate regression signal: every sandbox now runs the argument reader before the supervisor exec, so a mistake in the reader or in the empty-array expansion underset -uwould break ordinary VM boot, not just the proxy path.Two notes for anyone running the VM lane locally on macOS, both discovered here and neither a product bug:
brew install e2fsprogsis required — the driver shells out tomke2fson the host to build the guest rootfs. It is keg-only; the driver probes/opt/homebrew/opt/e2fsprogs/sbindirectly.TooLargepanic. E2E workloads of any size should useSandboxGuard::create(delivered viaExecSandbox) rather thancreate_keep_with_args.Checklist