Retry EINTR on poll/select so profiling signals don't break native blocking calls (#1060) - #1062
Merged
Merged
Conversation
Scalene profiles wall-clock time by default (use_virtual_time is False), so it arms ITIMER_REAL and delivers SIGALRM periodically -- including while the process is blocked inside a native syscall. Scalene already installs handlers with SA_RESTART (siginterrupt(sig, False)), which resumes *restartable* calls, but poll(2)/select(2)/ppoll(2)/pselect(2)/epoll_wait(2) are never restarted regardless of SA_RESTART (signal(7)) -- they always fail with EINTR when a handler runs. CPython retries its own I/O on EINTR (PEP 475), but native libraries that don't break. The reported case (#1060) is the Microsoft ODBC Driver 18: its TCP provider does a non-blocking connect() + poll()/select() with a login timeout, and a SIGALRM landing on that wait surfaces as WSAEINTR (0x2714 / 10004) / SQLSTATE 08S01 and a failed connection -- only on Linux, since Windows samples via threads. Because libscalene is LD_PRELOADed whenever memory profiling is enabled (the default), interpose the non-restartable wait calls and retry them on EINTR, recomputing the remaining timeout so deadlines are preserved. Only EINTR caused by Scalene's own timer signal is absorbed: the guard wraps whichever timer signal handler scalene installs (learned by interposing setitimer) with a shim that bumps a generation counter, and the retry loop continues only when that counter advanced across the interrupted call. An EINTR from any other signal (SIGINT/Ctrl-C, SIGTERM, ...) is returned to the caller unchanged. Set SCALENE_DISABLE_EINTR_RETRY=1 to opt out (pure pass-through). - src/include/eintr_retry.hpp: the interposer logic (Linux uses dlsym(RTLD_NEXT); macOS calls the libc symbol directly, as Heap-Layers' memcpy wrapper does, since __interpose only redirects other images). - src/source/libscalene.cpp: exported interpose entry points + MAC_INTERPOSE. - tests/test_eintr_retry.py: builds a tiny interposer from the real header and verifies poll/select survive the timer while SIGINT still interrupts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 19, 2026
The EINTR-retry interposer is POSIX-only: on Windows Scalene samples via threads, not signals, so blocking syscalls are never interrupted and the issue #1060 failure mode cannot occur. Add the positive counterpart test plus a dedicated Windows CI job: - tests/test_eintr_retry.py: split the module-level skip into per-test marks (POSIX test keeps its skip; a new Windows-only test is added). The Windows test profiles a blocking TCP round-trip (200 connect/recv cycles while burning CPU so the sampler is active) under `scalene run --cpu-only` and asserts it completes cleanly. - .github/workflows/tests.yml: add a scoped `run-tests-windows` job (windows-latest, Python 3.11 + 3.13) that builds Scalene and runs this test. It's a separate job rather than a matrix entry because the full pytest suite is not yet Windows-clean. Verified locally: the workload completes (ROUNDTRIP_OK 200) standalone and under `scalene run --cpu-only`; pytest collects both tests with correct per-platform skips; YAML and ruff pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ping CI surfaced a SIGSEGV under `scalene run --profile-interval=2` on macOS (and the same crash would hit Linux): wrapping Scalene's timer-signal handler with a generation-counting shim collided with Scalene's own native `--stacks` unwinder (scalene_signal_unwinder chains via g_prev_action and runs _Unwind_Backtrace from signal context). The extra wrapped frame plus re-entrant signal delivery during unwinding overflowed into a stack-guard page and crashed in dyld. Replace the whole sigaction-wrapping mechanism with a side-effect-free signal: Scalene's CPU sampler re-arms its one-shot interval timer from inside the signal handler on every tick (ScaleneSignalManager.restart_timer -> setitimer). Since libscalene already interposes setitimer, just bump the generation counter there when a CPU-sampling timer (ITIMER_REAL/VIRTUAL/PROF) is armed. The poll/select retry loops are unchanged: they retry only when the counter advanced across the interrupted call, so SIGINT/Ctrl-C still propagates. No signal handler is touched anymore, so Scalene's signal machinery (including the --stacks unwinder) is left completely intact. Verified locally with a rebuilt libscalene.dylib: the previously-crashing command now passes 5/5, as do `--stacks`, `--stacks --profile-interval` with memory profiling, and a blocking-socket workload under default profiling. The POSIX regression test was updated to re-arm the timer from its handler (matching Scalene) and to disarm it before the SIGINT phase; it passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1060.
Problem
Profiling with Scalene breaks SQL Server connections made via pyodbc / ODBC
Driver 18 on Linux (reported on Azure Batch):
connect()fails with WSAEINTR(
0x2714/10004) / SQLSTATE08S01. The same code works without Scalene,and works under Scalene on Windows.
Root cause
Scalene profiles wall-clock time by default (
use_virtual_time=False), so itarms
ITIMER_REALand deliversSIGALRMperiodically — including while theprocess is blocked inside a native syscall. Scalene already installs its handlers
with
SA_RESTART(siginterrupt(s, False)), which auto-resumes restartablesyscalls. But per
signal(7),poll(2)/select(2)/ppoll(2)/pselect(2)/epoll_wait(2)are never restarted regardless ofSA_RESTART— they alwaysfail with
EINTRwhen a handler runs.CPython retries its own I/O on
EINTR(PEP 475), but native libraries that don'tbreak. The ODBC Driver 18 TCP provider does a non-blocking
connect()followedby
poll()/select()with a login timeout; aSIGALRMlanding on that waitsurfaces as WSAEINTR and a failed connection. It's Linux-only because Windows
samples via threads, not signals — and a slower (containerized / cross-network)
connection widens the window, matching the report.
No signal-handler flag can fix this:
poll/selectsimply aren't restartable.The interrupted wait has to be retried.
Fix
libscaleneis alreadyLD_PRELOADed whenever memory profiling is enabled (thedefault), so it interposes the non-restartable wait calls and retries them on
EINTR, recomputing the remaining timeout so deadlines are preserved.Only Scalene-induced
EINTRis absorbed, soSIGINT/Ctrl-C and other signalsstill propagate. Scalene's CPU sampler re-arms its one-shot interval timer from
inside the signal handler on every tick (
ScaleneSignalManager.restart_timer→setitimer), andlibscalenealready interposessetitimer— so the interposerjust bumps a generation counter there. The retry loop continues only when that
counter advanced across the interrupted call. No signal handler is touched,
so Scalene's signal machinery (including the
--stacksnative unwinder) is leftcompletely intact.
Opt out with
SCALENE_DISABLE_EINTR_RETRY=1(pure pass-through).Files
src/include/eintr_retry.hpp— interposer logic. Linux usesdlsym(RTLD_NEXT); macOS calls the libc symbol directly (as Heap-Layers'memcpy wrapper does), because
__interposeonly redirects other images anddlsym(RTLD_NEXT)would otherwise return our own shim and recurse.src/source/libscalene.cpp— exported interpose entry points +MAC_INTERPOSE.tests/test_eintr_retry.py— POSIX regression test (builds a tiny interposerfrom the real header; verifies
poll/selectsurvive the timer whileSIGINTstill interrupts) plus a Windows test that profiles a blocking TCP round-trip
under
scalene runand asserts it completes cleanly..github/workflows/tests.yml— adds a scopedrun-tests-windowsjob thatbuilds Scalene and runs the test on
windows-latest(Python 3.11 + 3.13).Testing
Verified with a rebuilt
libscalene:polldies withEINTRin ~6 ms.poll/selectrun their full timeout despite the timer;SIGINTstill interrupts;SCALENE_DISABLE_EINTR_RETRY=1restores pass-through.scalene run(default + memory),--stacks, and--stacks --profile-intervalall run clean (the earlier handler-wrapping approach crashed here; the
setitimer-based design does not). Measured no profiling overhead vs master
(2.5 s memory-profiling run, identical to baseline).
CI: smoketests pass on all platforms incl. Windows;
run-testspasses onLinux + macOS (the regression test runs and passes there); the new
run-tests-windowsjobs pass.🤖 Generated with Claude Code