Fix test-suite sys.executable leak that skipped free-threaded memory tests (#1066)

Root cause: test_coverup_30::test_scalene_cpu_count constructs Scalene(args)
in-process to check __availableCPUs. Scalene.__init__ runs the full profiler
setup, including scalene.redirect_python.redirect_python -- production
behavior that rewrites sys.executable / sys.path / PATH to a
/tmp/scalene*/python bash wrapper so child processes re-enter through
Scalene. That is correct for a real one-shot 'scalene run' process, but in
the shared, long-lived pytest interpreter it leaks: the test mocks
ScaleneMapFile but not the redirect and never restores the globals.

Every later test that spawns [sys.executable, '-m', 'scalene', ...] then
launched the wrapper instead of real Python, so memory profiling failed to
initialize ([Errno 2] .../tmp/scalene-malloc-signal<pid>) and the memory_*
tests skipped with 'no usable profile' (or, for a few CPU tests, timed out).
Ordering-dependent; it surfaced on free-threaded CI but is not
free-threaded-specific. A full-suite scan found test_coverup_30 to be the
sole polluter.

Fix (minimal, at the source): test_coverup_30's cleanup fixture now
snapshots and restores sys.executable / sys.path / PATH. Verified on Linux
3.13t: the test no longer leaves sys.executable pointing at the wrapper, and
the memory tests pass in suite order even with bare sys.executable in the
spawn helpers.

Also:
- tests/test_memory_stacks_bigmem.py: once the test stopped skipping it
  actually ran and exposed a too-strict assertion. The driver's own
  list.append() sites legitimately allocate (CPython list-backing-store
  growth), so a memory_stacks leaf occasionally lands on the append line,
  not make_vec. Require the allocator line to dominate (>=90% of leaf bytes)
  instead of being every leaf; the make_big/make_small split asserts are
  unchanged.
- tests/conftest.py + tests.yml: a SCALENE_TEST_DIAG-gated hook that prints
  each skip's reason live/flushed (off by default; on for t-builds), so skip
  reasons survive a job-timeout cancellation for future debugging.

Author: emeryberger

.github/workflows/tests.yml

@@ -113,6 +113,12 @@ jobs:
     - name: run tests
       # Free-threaded Python support is experimental; don't block CI on failures
       continue-on-error: ${{ endsWith(matrix.python, 't') }}
+      # SCALENE_TEST_DIAG=1 makes tests/conftest.py print each skip's reason
+      # live and flushed. On free-threaded builds the 45-min job timeout can
+      # cancel pytest before its -rs summary, so this keeps skip reasons in
+      # the log for future debugging.
+      env:
+        SCALENE_TEST_DIAG: ${{ endsWith(matrix.python, 't') && '1' || '' }}
       run: |
         python3 -m pytest
 

tests/conftest.py

@@ -0,0 +1,36 @@
+"""Pytest hooks for the Scalene test suite.
+
+The free-threaded CI jobs (3.13t/3.14t) can run past the 45-minute job
+timeout and get cancelled mid-suite. When that happens pytest never reaches
+its end-of-session ``-rs`` summary, so the *reasons* any memory_* tests
+skipped are lost. ``pytest_runtest_logreport`` fires as each test finishes,
+so printing the skip reason here (flushed) guarantees it lands in the CI log
+even if the job is killed before the summary. Gated on SCALENE_TEST_DIAG=1
+so normal local runs stay quiet.
+"""
+
+import os
+import sys
+
+
+def pytest_runtest_logreport(report):
+    if os.environ.get("SCALENE_TEST_DIAG") != "1":
+        return
+    if not report.skipped:
+        return
+    if report.when not in ("setup", "call"):
+        return
+    # report.longrepr for a skip is a (path, lineno, reason) tuple.
+    reason = report.longrepr
+    if isinstance(reason, tuple) and len(reason) == 3:
+        reason = reason[2]
+    print(
+        f"\n[SCALENE_TEST_DIAG] SKIP {report.nodeid}\n"
+        f"  reason: {reason}\n"
+        f"  LD_PRELOAD={os.environ.get('LD_PRELOAD')!r} "
+        f"DYLD_INSERT_LIBRARIES={os.environ.get('DYLD_INSERT_LIBRARIES')!r}\n"
+        f"  PYTHONMALLOC={os.environ.get('PYTHONMALLOC')!r} "
+        f"PYTHON_GIL={os.environ.get('PYTHON_GIL')!r}\n"
+        f"  sys.executable={sys.executable!r}",
+        flush=True,
+    )

tests/test_coverup_30.py

@@ -12,9 +12,22 @@
 
 @pytest.fixture
 def cleanup():
-    # Fixture to clean up any state after the test
+    # Constructing Scalene(args) runs the full profiler __init__, which calls
+    # scalene.redirect_python.redirect_python -- production behavior that
+    # rewrites sys.executable / sys.path / PATH to a /tmp/scalene*/python
+    # wrapper so child processes re-enter through Scalene. That is correct for
+    # a real one-shot `scalene run` process, but here it leaks into the shared
+    # pytest interpreter: later tests that spawn [sys.executable, "-m",
+    # "scalene", ...] would launch the wrapper instead of real Python and fail
+    # to profile. Snapshot and restore the mutated global state.
+    orig_executable = sys.executable
+    orig_path = list(sys.path)
+    orig_environ_path = os.environ.get("PATH")
     yield
-    # No specific cleanup required for this test
+    sys.executable = orig_executable
+    sys.path[:] = orig_path
+    if orig_environ_path is not None:
+        os.environ["PATH"] = orig_environ_path
 
 
 @patch("scalene.scalene_profiler.ScaleneMapFile")

tests/test_memory_stacks_bigmem.py

@@ -111,12 +111,22 @@ def test_memory_stacks_split_across_two_callers(tmp_path: Path) -> None:
     profile = _run_scalene(tmp_path)
     memory_stacks = profile["memory_stacks"]
 
-    # Every captured stack must have the allocator line 20 as its leaf.
+    # The allocator line 20 must dominate the captured leaves. We don't
+    # require *every* leaf to be line 20: the driver's own
+    # ``small_list.append(...)`` / ``big_list.append(...)`` sites (lines 35/37)
+    # grow CPython's list backing store, a genuine allocation whose
+    # synchronously-captured leaf is that append line, not make_vec. That is
+    # correct attribution, just not the allocator helper, and it shows up as
+    # an occasional stray leaf under sampling noise (seen on macOS). Assert
+    # that make_vec dominates rather than that nothing else ever allocates.
     leaves = [_leaf_line(frames) for frames, _mb in memory_stacks]
-    assert all(leaf == ALLOCATOR_LINE for leaf in leaves), (
-        f"Expected all memory_stacks leaves to land on allocator line "
-        f"{ALLOCATOR_LINE}, got {leaves}. Intermediate frames being "
-        f"skipped or the allocator being misattributed."
+    alloc_leaf_mb = sum(mb for frames, mb in memory_stacks if _leaf_line(frames) == ALLOCATOR_LINE)
+    total_leaf_mb = sum(mb for _frames, mb in memory_stacks)
+    assert total_leaf_mb > 0 and alloc_leaf_mb >= 0.9 * total_leaf_mb, (
+        f"Expected allocator line {ALLOCATOR_LINE} to dominate memory_stacks "
+        f"leaves (>=90% of bytes), got {alloc_leaf_mb:.1f} of {total_leaf_mb:.1f} MB; "
+        f"leaves={leaves}. Intermediate frames being skipped or the allocator "
+        f"being misattributed."
     )
 
     # The caller frame (directly above the allocator leaf) must pick up