Fix profiler and TAU harness checks by dannybaths · Pull Request #181 · amd/HPCTrainingExamples

dannybaths · 2026-08-21T10:22:31Z

Summary

This PR fixes several profiler and TAU test harness issues found while validating the AAC7 OpenMPI row.

Profiler tests now assert produced artifacts instead of matching fragile tool output, and the
corresponding CTest pass expressions are updated to match.
TAU checks now fail on build failure and assert real profile/trace artifacts.
TAU and Jacobi MPI detection now query the active MPI wrapper instead of checking whether ompi_info
exists somewhere on PATH.
Jacobi-based TAU launches avoid placing ranks on nodes that cannot see the node-local build
directory.
The PyTorch rocprof-compute run checks the dependency pins that analyze enforces before profiling,
and the rocprof-compute tests fail immediately, with the reason, when Python is older than 3.10.
These are reported as failures rather than skips deliberately, so the failure count is not quietly
reduced.

Why

Several tests could pass or fail for the wrong reason:

Some pass regexes matched diagnostic text printed on failure. profile.0 is a substring of
Could not open profile.0.0.0, and proto appears in output the tool prints when it produces
nothing.
One TAU script continued after make failed, so the final verdict depended only on regex output.
Because PASS_REGULAR_EXPRESSION overrides the exit code, the script and the CMake criteria had to
be fixed together.
MPI detection confused mixed environments where OpenMPI's ompi_info was visible but mpic++ had
been replaced by MPICH wrappers.
The PyTorch profiling run spent about 25 minutes profiling before reaching an analyze step that
could not run, turning a dependency problem into a timeout that named no cause.

Validation

Validated on AAC7 with the OpenMPI PrgEnv row: MI300A, ROCm 7.14.0, OpenMPI 5.0.10 over OFI/CXI.

The six tests whose pass criteria this PR rewrites were run against these exact changes and all
pass: Rocprof-sys_ROCm_Stream_Overlap, Rocprof-compute_ROCm_Roofline_Check, TAU_Trace_Check,
TAU_Profile_Check, TAU_Check_HIP_Profile and TAU_Check_MPI_Profile. Each now matches on a
string only the success path can produce.
The rocprof-compute Python precondition was exercised in both directions. Under Python 3.9.21 the
test fails in about three seconds, reporting the version it found and the version it needs. Under
Python 3.12.12 it runs to completion and the roofline assertion finds a real roofline.csv. The
failing case is deliberately a failure rather than a skip, so an environment that cannot run
rocprof-compute is not quietly removed from the failure count.
A single-node GPU probe showed that the detection and pass-criteria fixes make the Jacobi-based TAU
tests build and produce real profile.* / tautrace.* artifacts and pprof output containing MPI
and HIP routines.
The full AAC7 two-node suite then exposed a separate launch-directory issue: the tests built under
node-local /tmp, but rank 1 was launched on the second node where that directory did not exist. This
PR fixes that launch assumption too, by keeping the ranks on the node that owns the build directory.
Verified on two nodes with the checkout in node-local /tmp — the failure reproduces without the fix
and both invocations pass with it — and then in the full two-node suite, where all four Jacobi TAU
tests pass on real artifacts (319 pass / 27 fail / 15 skip / 1 timeout of 362, suspicious-pass 0).
Three TAU RCCL tests still fail for an environment/package reason: MPICH-built TAU cannot instrument
the OpenMPI-built rccl-tests benchmark. This PR does not attempt to solve that packaging mismatch.
The rocprof-sys PyTorch thread explosion is tracked separately as a rocprof-sys issue, not as a test
harness bug.

Make profiler and TAU tests assert real artifacts instead of matching diagnostic text, fail TAU builds explicitly, detect MPI by querying the wrapper that will actually be used, and check the rocprof-compute dependency pins before a long profiling run rather than after it.

gcapodagAMD · 2026-08-24T15:20:59Z

 endif

-IS_OMPI=$(shell which ompi_info 2>/dev/null)
+# Ask the wrapper we are about to invoke. `which ompi_info` reports that some


please revise these comments, if we remove the line that has ompi_info then it does not make sense to mention it in a comment. please revies all comments in general to make sure they add something useful, if not remove thanks

Removed the ompi_info mention from the Makefile comments and trimmed the narrating comments in the other files. Also fixed one older comment in tau_exec_check.sh that still said detection was via ompi_info. This is commit 445fae8.

gcapodagAMD · 2026-08-24T15:22:38Z

 source ${PROFILER_TOP_DIR}/setup.sh

+# Dependency precondition. `analyze` enforces the exact pins in its own
+# requirements.txt and this run ends in `analyze`, so verify them before


make sure to test that the changes you are proposing here do not break anything on aac6: run this branch there to make sure these python stuff works there too. rocprof-compute + python can be tricky. thanks

I ran the four rocprof-compute scripts on AAC6 (PPAC_MI300A_SPX, job 19332, node ppac-pl1-s24-26) with module load rocm/7.14.0 and the PR checkout at commit 1cec3a7. Profile and roofline pass. Analyze and single_process fail because rocprof-compute's pinned Python packages are missing or the wrong version — the same pin wall as AAC7, not the >=3.10 guard. System python3 is already 3.12.3, and rocprof-compute is 3.7.0 here, the same generation as AAC7, not the older 3.2.3 build. The follow-up commit only changes comments, so that run still applies. Note and log on AAC6: /shared/prerelease/home/amd_int/danieleb/HPC_OpemMPI_on_AAC7/docs/tasks/pr181/pr181-aac6-2026-08-31.md and evidence/pr181-aac6-2026-08-31-rerun.log beside it.

Follow-up on AAC6 rocm/7.2.4 as you asked (job 19345, same checkout 1cec3a7). All four scripts passed, including analyze and single_process (~37 min). That install is a frozen ELF under rocm-patches-7.2.4 (rocprof-compute.bin), not the Python wrapper, and there is no requirements.txt.

On AAC6 rocm/7.14.0 (job 19332) the tool is still the 3.7.0 wrapper, so analyze fails on its pins. Same PR, Python 3.12 both times. The harness is fine; 7.14.0 still needs the frozen overlay. No further commit on this PR.

gcapodagAMD · 2026-08-24T15:25:33Z

 set_property(TEST Rocprof-sys_ROCm_Run_Check PROPERTY SKIP_REGULAR_EXPRESSION "module spider")

 add_test(NAME Rocprof-sys_ROCm_Stream_Overlap COMMAND ../rocprof-sys_stream_overlap.sh )
-set_property(TEST Rocprof-sys_ROCm_Stream_Overlap PROPERTY PASS_REGULAR_EXPRESSION "proto")


as you see in the other tests, we want the tests to have a twofold purpose: 1 be used to check that the installation is working, that is why they are part of the test suite. 2 make sure that when a user runs them standalone they can still get the information they need so we don't want the tests to just output a "PASSED" message, we want the test to also print out the configuration and parameters someone would be interested in seeing. please make sure your changes still allow this, thanks

Left CMakeLists.txt as sent. Each script still prints the tool's own config, paths and parameters, and only adds the artifact-assertion line. The six emit strings still match the PASS_REGULAR_EXPRESSIONs.

gcapodagAMD · 2026-08-24T15:26:00Z

+# unions. Fail here, naming the reason, rather than later with a SyntaxError
+# from inside the tool that names no cause.
+if ! python3 -c 'import sys; sys.exit(0 if sys.version_info >= (3,10) else 1)' 2>/dev/null; then
+  echo "ERROR: rocprof-compute needs Python >= 3.10, but python3 is $(python3 -V 2>&1)."


double check this claim please

Double-checked on AAC7. ROCm 7.14.0's rocprof-compute (3.7.0) native_tool_finder.py uses Path | None with no from future import annotations. System python3 3.9.21 raises TypeError at import (line 38); Cray Python 3.12.12 imports fine. So it is a TypeError, not a SyntaxError — the comment is fixed in 445fae8. The site 3.2.3 tool still runs on 3.9. On AAC6, python3 is already 3.12.3 and the tool is also 3.7.0, so the >= 3.10 guard does not fire there and should stay as written.

push that commit directly so we see the files arleady change thanks

nevermind I see it

gcapodagAMD · 2026-08-24T15:27:12Z

+#   directory: going to /tmp instead
+#   [proxy:1@node2] launch_procs: unable to change wdir to /tmp/.../build_ewLCPv
+#
+# The test wants two ranks, not two nodes -- Jacobi's "-g 2 1" topology on two


please rephrase these AI generated comments in human lingo and maybe trim a bit if it applies

Rephrased the /tmp launch comment in plain language and cut it from about 19 lines to 6. In 445fae8.

gcapodagAMD

thanks @dannybaths please take a look at the comments I left in the files.

Address review: drop leftover ompi_info wording, shorten the TAU launch comment, and record that rocprof-compute 3.7.0 fails on Python 3.9 with TypeError rather than SyntaxError.

gcapodagAMD · 2026-08-31T18:50:36Z

thanks @dannybaths

gcapodagAMD reviewed Aug 24, 2026

View reviewed changes

gcapodagAMD requested changes Aug 24, 2026

View reviewed changes

Trim PR 181 comments and correct the Python TypeError note.

445fae8

Address review: drop leftover ompi_info wording, shorten the TAU launch comment, and record that rocprof-compute 3.7.0 fails on Python 3.9 with TypeError rather than SyntaxError.

dannybaths requested a review from gcapodagAMD August 31, 2026 09:05

gcapodagAMD merged commit ae1c76f into amd:main Aug 31, 2026

Uh oh!

Conversation

dannybaths commented Aug 21, 2026

Summary

Why

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gcapodagAMD left a comment

Choose a reason for hiding this comment

Uh oh!

gcapodagAMD commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants