Fix UnicodeEncodeError for non-ASCII file paths (#1086)

[emeryberger]

Fixes #1086.

Root cause

pywhere.cpp converted code->co_filename with PyUnicode_AsASCIIString on three paths: the stack walker used by the allocator interposer, on_stack(), and the settrace line callback. For any path containing a non-ASCII character — e.g. a module at .../überschüsse.py — that call fails, returning NULL and leaving a UnicodeEncodeError set on the thread.

Since these run from the native allocator hook and from trace callbacks, the stray exception surfaced later at an arbitrary, unrelated Python line — which is why the reporter saw it land in sysconfig.get_path on one run and inside pydantic on the next, and why the failure looked flaky. Two of the sites also fed the NULL straight into PyBytes_AsString/strstr, giving the reported SIGSEGV.

The Python side compounded it: ScaleneMapFile.get_str() decoded the native sample records as ASCII, so any allocation attributed to such a file also blew up the sample-draining loop.

Only --memory runs were affected, matching the reporter's observation that --cpu-only worked — that's the mode that loads the native interposer.

Changes

• src/source/pywhere.cpp — new encodeFilename() helper encoding as UTF-8 with surrogateescape. That's the exact inverse of how CPython decodes filesystem paths, so every path Python can represent — including undecodable bytes, which appear as lone surrogates — round-trips. It clears any exception on failure and returns nullptr; all three call sites now NULL-check. Also removes a dead encode-and-discard in trace_func.
• src/include/traceconfig.hpp — the package-path encode used "strict" and leaked a pending exception on failure; now surrogateescape + PyErr_Clear().
• scalene/scalene_mapfile.py — new decode_sample() using the matching UTF-8/surrogateescape pair, replacing the ASCII decodes.

Test

tests/test_issue1086_non_ascii_paths.py is picked up by the existing python3 -m pytest step in tests.yml, so it runs on Ubuntu + macOS across Python 3.9–3.14. Three cases:

1. decode_sample directly — deterministic, including lone surrogates.
2. • 3. End-to-end scalene run --memory on a program importing a non-ASCII module, with the non-ASCII component in the filename and in a parent directory. Asserts no Unicode error, the program ran to completion, exit code 0, and the path comes back through the profile intact (NFC-normalized comparison so an NFD-normalizing filesystem doesn't false-fail).

The crash assertions are hard on every attempt; only the "file appears in the profile" check retries against the usual sampling flake.

Verification

• Reproduced the issue on macOS / Python 3.14: 3/3 runs died with 'ascii' codec can't encode character '\xfc', same traceback shape as the report.
• With the fix, the profile is complete (überschüsse.py: 20 MB malloc, 89% CPU); --memory --stacks also carries the non-ASCII path correctly through the native stack buffer.
• Reverted only the native fix and rebuilt → the two end-to-end tests fail as intended; restored → pass.
• Full tests/ suite: 445 passed, 13 skipped. ruff clean; mypy shows only the pre-existing scalene_signal_manager.py error also present on master.

🤖 Generated with Claude Code

[emeryberger]

Added a directory-only layout to the end-to-end test (90cabf7), covering the reporter's exact framing — "a module in its import tree which is at a path with non-ascii characters in it," where the umlaut is not in the filename at all.

The three layouts are now explicit:

id	path	non-ASCII component
leaf-only	program/überschüsse.py	filename
directory-only	optionale_verlängerung/pakete/workload.py	interior directory only — module and its immediate parent are ASCII
both	prögramm/überschüsse.py	both

The native code encoded the whole path, so a single non-ASCII character anywhere in it was enough. Verified directory-only against the parent commit's native code:

'ascii' codec can't encode character '\xe4' in position 130: ordinal not in range(128)
  File ".../optionale_verlängerung/pakete/workload.py", line 6, in f

That \xe4 is the ä from the directory, on an all-ASCII workload.py. All four tests pass with the fix.