Skip to content

test(e2e): pin the second README-walkthrough batch (EAI-8046..8049) - #292

Open
fredespi wants to merge 8 commits into
mainfrom
e2e-pin-the-contracts-for-the-second
Open

test(e2e): pin the second README-walkthrough batch (EAI-8046..8049)#292
fredespi wants to merge 8 commits into
mainfrom
e2e-pin-the-contracts-for-the-second

Conversation

@fredespi

Copy link
Copy Markdown
Collaborator

Summary

Test-only. Pins the correct behaviour for the second README-walkthrough batch of defects (EAI-8046–8049, tracked upstream as EAI-8050), following the same shape as #216: one behavioural scenario per defect, each asserting the CORRECT behaviour and registered as an xfail in expectations.toml citing its EAI id. The mock lane stays green now, and each row goes stale (XPASS) the day its bug is fixed. No product behaviour changed.

All four defects were reproduced by hand on an MI300X app-dev pod, then re-confirmed in a plain Linux container against this branch's binary before the scenarios were written. Each scenario was verified to fail on its assertion step (setup steps green) for the stated reason — not a miswiring.

What each ticket now pins

  • EAI-8046rocm chat ignores a piped prompt. chat.feature scenario 8 pipes a prompt with no --prompt and asserts the model actually received it. The load-bearing assertion is the mock's received request (the bug's signature is that no request is made), not stdout. Needed a new run_rocm_with_stdin helper — Command::output gives the child a null stdin, so no existing step could reach the piped path.
  • EAI-8047 — automation checks listed by rocm automations list cannot be enabled from what the listing shows. New automations.feature + steps: the scenario derives each check's identifier from the listing (an explicitly-exposed id if present, else the display-name slug as today's fallback) and asserts automations enable <id> succeeds. Deriving from the listing — rather than hard-coding the real ids — is the contract.
  • EAI-8048rocm dash --replay <missing> opens the interactive dashboard instead of refusing. dash.feature scenario 10 drives it under a PTY, imposes its own deadline (the current behaviour hangs), and asserts both a prompt non-zero exit and that the alternate-screen interactive view never opened.
  • EAI-8049 — the activation hint printed by rocm engines shell names a path that does not exist (the recorded env_path already includes the interpreter directory, so the hint doubles that segment). engine_shell.feature scenario 2 runs it under a PTY and asserts the hint names a file that exists — phrased as path-exists so the row goes stale on any correct fix.

Notes

  • Harness additions live entirely in the e2e-cucumber test crate: run_rocm_with_stdin, a new automations_steps module, and PTY-driver support (wait_for_any_exit, screen_logical_lines, and a race-free latch that detects the alternate-screen enter sequence in the raw byte stream with a bounded post-exit reader drain).
  • The four expectations.toml rows are all when = {} (platform-independent), each with a comment stating the scope and the removal condition.

Test plan

  • Blocking e2e mock job is green: pass-or-xfail, no XPASS. (Locally on a native-Linux container: clippy -D warnings clean, e2e-cucumber lib tests pass, mock lane reconciles to 6 xfail / 0 XPASS / 0 unexpected failures.)
  • Each new scenario xfails for its stated reason (verified: each fails on its assertion step with setup steps green).
  • No GPU dispatch required — all four run on the fast mock lane.

@fredespi
fredespi requested a review from a team as a code owner August 20, 2026 12:22
@fredespi
fredespi requested a review from rominf August 20, 2026 12:22
Add one behavioural scenario per defect, each asserting the CORRECT
behaviour and registered as an xfail in expectations.toml citing its EAI
id, so the mock lane stays green now and the row goes stale (XPASS) the
day the bug is fixed. No product behaviour changed.

- EAI-8046: `rocm chat` ignores a piped prompt (reads no stdin, sends no
  request, exits 0 with a status summary). New run_rocm_with_stdin helper
  since Command::output gives the child a null stdin; the load-bearing
  assertion is that the mock received the prompt, not stdout.
- EAI-8047: automations list publishes no usable identifier, so the
  checks it lists cannot be enabled from what it shows. The scenario
  derives each id from the listing rather than hard-coding the real ids.
- EAI-8048: `dash --replay <missing>` opens the interactive view instead
  of refusing; the step imposes its own deadline so the known hang xfails
  promptly rather than burning the CI budget.
- EAI-8049: the engine shell's activation hint names a path that does not
  exist, because the recorded env_path already includes the interpreter
  directory. Phrased as path-exists so the row goes stale on any fix.

All four re-confirmed in a plain Linux container against this branch's
binary before writing the scenarios.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/expectations.toml
#	tests/e2e-cucumber/tests/e2e.rs
Address pre-PR review: each of the three step assertions pinned the bug's
present symptom rather than the behaviour that must hold, so a valid fix
could stay xfailed and the row would never go stale.

- EAI-8047: parse the identifier the listing explicitly exposes (inline
  [id]/(id) on the header, or an `id:`/`identifier:` detail line), keeping
  the display-name slug only as the current-broken-output fallback. A fix
  that publishes the id now flips the row to XPASS.
- EAI-8048: also assert the interactive view never opened, not just a
  prompt non-zero exit. Adds a latched entered_alternate_screen() to the
  PTY driver so an "open the TUI, then exit non-zero" impl no longer
  satisfies the contract.
- EAI-8046: assert the user-role message content rather than messages[0],
  so a fix that prepends a system message still XPASSes.

Mock lane unchanged: still 6 xfail / 0 XPASS; each scenario still fails on
its assertion step alone.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Round-2 review: the latch checked the parser's alternate_screen() state
only after processing a whole PTY read, so a chunk carrying both the enter
(?1049h) and leave (?1049l) sequences left it false; and wait_for_any_exit
returned before the reader drained, so a still-buffered enter could be
missed. Either produced a false negative that would let a fixed dashboard
open the TUI and still pass the "must not open the interactive view" half.

- Detect the ?1049h enter sequence in the RAW byte stream, with a
  one-short-of-marker carry-over so a boundary-split sequence is still
  found. Independent of what follows it in the same read.
- Drain the reader (bounded by DRAIN_TIMEOUT) after the child exits, before
  the caller reads the latch — same post-exit drain wait_for_screen uses.

Mock lane unchanged: 6 xfail / 0 XPASS.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi force-pushed the e2e-pin-the-contracts-for-the-second branch from 450d4c8 to dd733fe Compare August 20, 2026 12:58
The CI Clippy (e2e harness + steps) job runs clippy -p e2e-cucumber --test e2e -D warnings, which the local container gate did not (its clippy step excludes e2e-cucumber). Four lints surfaced:

- future-not-send: drain_reader took &self, and a shared &TuiSession held across .await is not Send (the struct holds !Sync PTY handles), poisoning every step awaiting it. Take &mut self like the sibling waits.
- collapsible-if, or-fun-call (automations_steps) and map-unwrap-or (engines_steps): mechanical rewrites, no behaviour change.

Verified with the exact CI commands in a Linux container: both clippy invocations clean, e2e mock lane unchanged (6 xfail / 0 XPASS).

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/features/automations.feature
#	tests/e2e-cucumber/tests/e2e/automations_steps.rs
Upstream (EAI-8072) independently added a run_rocm_with_stdin helper that also
takes an env slice, and the merge left both definitions in e2e.rs (E0428), with
my chat step calling the 3-arg form (E0061).

Keep upstream's more general helper and delete mine; the chat stdin step now
passes an empty env slice. Behaviour is identical - both pipe stdin and close
the handle before waiting, so a command reading to EOF still terminates.

Verified with the exact CI commands in a Linux container: both clippy
invocations clean, e2e mock lane 6 xfail / 0 XPASS.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

@michaelroy-amd michaelroy-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 1452ee174a87cf05693284b32a8b16fce0cc094a. The four scenarios pin the intended user-visible contracts rather than implementation details: piped chat reaches the model, listed automation identifiers are usable, a missing replay file fails without opening the TUI, and the engine-shell activation path exists. Each expectation is scoped to its ticket and removal condition; helper changes have bounded PTY drains and preserve command environment isolation. All 17 required checks pass, including the mock E2E gate and DCO/signoff. No blocking finding. Approving.

@fredespi
fredespi added this pull request to the merge queue Aug 27, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Aug 27, 2026
@rominf

rominf commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

I read through this PR's diff (scenarios, step definitions, expectations.toml, and the tui_driver additions) and traced each of the four claimed defects against the actual source:

  • EAI-8046: confirmed. Command::Chat in apps/rocm/src/main.rs only reads stdin via the interactive dash path when interactive_terminal() is true; with a piped, non-interactive run and no --prompt, it falls straight to render_chat_text, which never touches stdin or sends a request. That matches the --prompt help text's claim ("reads from standard input... when omitted") and the EXAMPLES block, which this scenario correctly proves false today.
  • EAI-8047: the derivation logic in the new automations_steps.rs (parsing the automations list output for an id inline in brackets/parens or on a detail line, falling back to slugifying the display name) looks correct against render_automations_text/watcher_plain_name in main.rs, and won't silently pass once a fix publishes a real id.
  • EAI-8048: confirmed dash::run in apps/rocm/src/dash.rs doesn't validate the --replay path before entering the interactive event loop. The new wait_for_any_exit/entered_alternate_screen latch in tui_driver.rs is a sound way to assert "refused before opening the TUI" rather than just "exited non-zero," and the byte-level alternate-screen-enter detection (with a carry buffer for reads split across the escape sequence) is correctly bounded.
  • EAI-8049: confirmed by reading resolve_engine_env plus runtime_python_activation_hint/runtime_python_env_bin_dir in rocm-core, and the vLLM install's env_path = command.parent() in engines/vllm/src/lib.rs. The doubled bin/bin/activate path is real, and phrasing the assertion as "the hint names a file that exists" (rather than "not doubled") is the right way to keep the scenario meaningful after any valid fix.

I also checked the expectations.toml additions against the file's own grammar and existing rows, and the four new @id: tags line up 1:1 between the feature files and the xfail table.

One thing worth flagging, though it's not a code issue: GitHub currently reports this PR as having a merge conflict. That's because dash.feature's scenario 10 slot was independently filled by a different, later-merged PR (the "launcher shows live serving instance" scenario) after this branch's base commit — diffed against the actual merge-base with main, this PR's own changes apply cleanly and are a pure addition. It'll need a rebase to reconcile scenario numbering with main, but that's a mechanical rebase, not a defect in the tests or steps themselves.

Nothing else stood out. This looks safe to approve after a rebase and once CI (mock e2e lane, clippy) is green — this is a diff-only read, not a substitute for CI or a maintainer's own pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants