Drop C-side smear from per-line memory attribution and the flame view

[emeryberger]

Summary

This change should have been part of #1050 but didn't make it into the squash merge. Restoring it now as a follow-up.

A pure-arithmetic hot loop like z = z * z on floats was consistently showing hundreds of MB of malloc traffic, gigabyte-scale peak / sparkline values, a sizable wedge in the memory-activity pie, and a fat path in the --stacks memory-flame chart — even though the line's bytecode (LOAD_FAST / BINARY_OP / STORE_FAST) had no way to invoke a C-side allocator.

The mechanism: the C++ heap interposer stamps every malloc with the leaf user frame (and the captured Python stack) from whereInPython, but CPython internals (arena resizes, GC, scalene's own bookkeeping under the GIL) emit raw malloc calls whose stamped leaf is whichever line the eval loop happens to be on. The bytes are real but the attribution is wrong — they smear onto whatever's hot, both per-line and along the captured call chain.

Fix

New helper line_has_alloc_opcode(filename, lineno) in scalene_utility.py compiles each user source once via linecache.getlines + compile + dis.get_instructions, recording the set of source lines whose bytecode contains at least one allocation-capable opcode. Set membership: startswith(\"CALL\" / \"BUILD_\" / \"LIST_\" / \"SET_\" / \"MAP_\" / \"DICT_\" / \"IMPORT_\" / \"MAKE_\" / \"FORMAT_\") plus exact opnames covering GET_ITER, FOR_ITER, SEND, YIELD_VALUE, BINARY_SUBSCR, STORE_SUBSCR, LOAD_ATTR, STORE_ATTR, LOAD_BUILD_CLASS, RAISE_VARARGS, BEFORE_WITH, WITH_EXCEPT_START, etc. Handles Python 3.13's Instruction.line_number change and the older starts_line semantics by propagating last-seen line forward through the instruction stream. Result is cached per file; subsequent lookups are O(1).

process_malloc_free_samples consults the helper for every sample. When the leaf line is not allocation-capable, the sample is dropped from every per-line accumulator — memory_malloc_samples, memory_python_samples, memory_max_footprint, memory_current_footprint, memory_aggregate_footprint, per_line_footprint_samples, memory_free_samples, memory_free_count, bytei_map — and from stats.memory_stacks. The bytes still flow into the global running footprint so the global peak and sparkline stay accurate, but no line, function, or flame-chart path takes the blame.

The reasoning for dropping memory_stacks too: the captured stack records which user frames were active at the moment of the malloc, but the actual cause is heap pressure set up by a prior frame that's already returned (e.g., doit1 ran, allocated, returned; later GC fires during doit2's arithmetic loop and the captured stack shows doit2, not doit1). Crediting the captured chain in the flame view smears the same way the per-line table did, just along a different axis. Two earlier drafts tried lighter touches — redirect-up-the-stack, and keep-flame-view-intact — and both failed for the same reason: there's no honest way to attribute these bytes from the synchronously-captured stack alone.

Frees use the same check. The C++ process_free path doesn't capture a stack so they couldn't be redirected anyway, but skipping them on smear-prone leaves keeps the per-line current-footprint and sparkline columns consistent with the dropped mallocs.

Verification

On test/testme.py (the canonical reproducer):

Pre-fix:

• doit2's body lines showed ~1.2 GB of phantom malloc traffic with a ~810 MB peak attributed to the inner `z = z * z` line.
• The flame chart showed a fat `→ stuff → doit2 → z = z * z` path.

Post-fix:

• Every column inside doit2 and stuff()'s call sites is zero.
• Function-rollup view: doit2 is zero, doit1 carries its full ~1.9 GB.
• memory_stacks contains only the three legitimate paths through doit1's list comprehensions (`stuff:58 → doit1:47 → lines 13/14/15`).

Tests

tests/test_memory_arithmetic_smear.py (new, four tests) with fixture test/line_attribution_tests/arithmetic_smear.py:

• test_arithmetic_lines_have_minimal_malloc_traffic — per-line `n_malloc_mb` on arithmetic lines stays bounded.
• test_allocator_lines_still_dominate — guards against over-suppression: the legitimate list-comprehension lines keep their bytes.
• test_function_rollup_does_not_smear_onto_arith_function — function-level rollup row for the arithmetic function is zero across every memory column.
• test_memory_stacks_preserve_full_paths — memory_stacks contains legitimate alloc-leaf paths and no entries whose leaf is a pure-arithmetic line.

All existing memory-attribution / memory-stacks tests still pass (test_line_attribution_nested.py × 3, test_memory_stacks_bigmem.py × 3).

Test plan

• `python3 -m pytest tests/test_memory_arithmetic_smear.py tests/test_line_attribution_nested.py tests/test_memory_stacks_bigmem.py` — 10/10 pass
• `ruff check scalene/` clean
• `mypy scalene/` clean across all 65 source files
• `testme.py` end-to-end: per-line + per-function memory view shows `doit2` and its `z = z * z` body at zero across every column; `memory_stacks` contains only `doit1`'s legitimate paths

🤖 Generated with Claude Code