Skip to content

Retry EINTR on poll/select so profiling signals don't break native blocking calls (#1060) - #1062

Merged
emeryberger merged 3 commits into
masterfrom
fix/issue-1060-eintr-retry
Jun 21, 2026
Merged

Retry EINTR on poll/select so profiling signals don't break native blocking calls (#1060)#1062
emeryberger merged 3 commits into
masterfrom
fix/issue-1060-eintr-retry

Conversation

@emeryberger

@emeryberger emeryberger commented Jun 19, 2026

Copy link
Copy Markdown
Member

Fixes #1060.

Problem

Profiling with Scalene breaks SQL Server connections made via pyodbc / ODBC
Driver 18 on Linux (reported on Azure Batch): connect() fails with WSAEINTR
(0x2714 / 10004) / SQLSTATE 08S01. The same code works without Scalene,
and works under Scalene on Windows.

Root cause

Scalene profiles wall-clock time by default (use_virtual_time=False), so it
arms ITIMER_REAL and delivers SIGALRM periodically — including while the
process is blocked inside a native syscall. Scalene already installs its handlers
with SA_RESTART (siginterrupt(s, False)), which auto-resumes restartable
syscalls. But per signal(7), poll(2)/select(2)/ppoll(2)/pselect(2)/
epoll_wait(2) are never restarted regardless of SA_RESTART — they always
fail with EINTR when a handler runs.

CPython retries its own I/O on EINTR (PEP 475), but native libraries that don't
break. The ODBC Driver 18 TCP provider does a non-blocking connect() followed
by poll()/select() with a login timeout; a SIGALRM landing on that wait
surfaces as WSAEINTR and a failed connection. It's Linux-only because Windows
samples via threads, not signals — and a slower (containerized / cross-network)
connection widens the window, matching the report.

No signal-handler flag can fix this: poll/select simply aren't restartable.
The interrupted wait has to be retried.

Fix

libscalene is already LD_PRELOADed whenever memory profiling is enabled (the
default), so it interposes the non-restartable wait calls and retries them on
EINTR, recomputing the remaining timeout so deadlines are preserved.

Only Scalene-induced EINTR is absorbed, so SIGINT/Ctrl-C and other signals
still propagate. Scalene's CPU sampler re-arms its one-shot interval timer from
inside the signal handler on every tick (ScaleneSignalManager.restart_timer
setitimer), and libscalene already interposes setitimer — so the interposer
just bumps a generation counter there. The retry loop continues only when that
counter advanced across the interrupted call. No signal handler is touched,
so Scalene's signal machinery (including the --stacks native unwinder) is left
completely intact.

Opt out with SCALENE_DISABLE_EINTR_RETRY=1 (pure pass-through).

Files

  • src/include/eintr_retry.hpp — interposer logic. Linux uses
    dlsym(RTLD_NEXT); macOS calls the libc symbol directly (as Heap-Layers'
    memcpy wrapper does), because __interpose only redirects other images and
    dlsym(RTLD_NEXT) would otherwise return our own shim and recurse.
  • src/source/libscalene.cpp — exported interpose entry points + MAC_INTERPOSE.
  • tests/test_eintr_retry.py — POSIX regression test (builds a tiny interposer
    from the real header; verifies poll/select survive the timer while SIGINT
    still interrupts) plus a Windows test that profiles a blocking TCP round-trip
    under scalene run and asserts it completes cleanly.
  • .github/workflows/tests.yml — adds a scoped run-tests-windows job that
    builds Scalene and runs the test on windows-latest (Python 3.11 + 3.13).

Testing

Verified with a rebuilt libscalene:

  • Baseline (no interposer) reproduces the bug — poll dies with EINTR in ~6 ms.
  • With the interposer, poll/select run their full timeout despite the timer;
    SIGINT still interrupts; SCALENE_DISABLE_EINTR_RETRY=1 restores pass-through.
  • scalene run (default + memory), --stacks, and --stacks --profile-interval
    all run clean (the earlier handler-wrapping approach crashed here; the
    setitimer-based design does not). Measured no profiling overhead vs master
    (2.5 s memory-profiling run, identical to baseline).

CI: smoketests pass on all platforms incl. Windows; run-tests passes on
Linux + macOS (the regression test runs and passes there); the new
run-tests-windows jobs pass.

🤖 Generated with Claude Code

Scalene profiles wall-clock time by default (use_virtual_time is False), so it
arms ITIMER_REAL and delivers SIGALRM periodically -- including while the
process is blocked inside a native syscall. Scalene already installs handlers
with SA_RESTART (siginterrupt(sig, False)), which resumes *restartable* calls,
but poll(2)/select(2)/ppoll(2)/pselect(2)/epoll_wait(2) are never restarted
regardless of SA_RESTART (signal(7)) -- they always fail with EINTR when a
handler runs.

CPython retries its own I/O on EINTR (PEP 475), but native libraries that don't
break. The reported case (#1060) is the Microsoft ODBC Driver 18: its TCP
provider does a non-blocking connect() + poll()/select() with a login timeout,
and a SIGALRM landing on that wait surfaces as WSAEINTR (0x2714 / 10004) /
SQLSTATE 08S01 and a failed connection -- only on Linux, since Windows samples
via threads.

Because libscalene is LD_PRELOADed whenever memory profiling is enabled (the
default), interpose the non-restartable wait calls and retry them on EINTR,
recomputing the remaining timeout so deadlines are preserved. Only EINTR caused
by Scalene's own timer signal is absorbed: the guard wraps whichever timer
signal handler scalene installs (learned by interposing setitimer) with a shim
that bumps a generation counter, and the retry loop continues only when that
counter advanced across the interrupted call. An EINTR from any other signal
(SIGINT/Ctrl-C, SIGTERM, ...) is returned to the caller unchanged.

Set SCALENE_DISABLE_EINTR_RETRY=1 to opt out (pure pass-through).

- src/include/eintr_retry.hpp: the interposer logic (Linux uses
  dlsym(RTLD_NEXT); macOS calls the libc symbol directly, as Heap-Layers'
  memcpy wrapper does, since __interpose only redirects other images).
- src/source/libscalene.cpp: exported interpose entry points + MAC_INTERPOSE.
- tests/test_eintr_retry.py: builds a tiny interposer from the real header and
  verifies poll/select survive the timer while SIGINT still interrupts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread tests/test_eintr_retry.py Fixed
emeryberger and others added 2 commits June 19, 2026 12:37
The EINTR-retry interposer is POSIX-only: on Windows Scalene samples via
threads, not signals, so blocking syscalls are never interrupted and the
issue #1060 failure mode cannot occur. Add the positive counterpart test plus
a dedicated Windows CI job:

- tests/test_eintr_retry.py: split the module-level skip into per-test marks
  (POSIX test keeps its skip; a new Windows-only test is added). The Windows
  test profiles a blocking TCP round-trip (200 connect/recv cycles while
  burning CPU so the sampler is active) under `scalene run --cpu-only` and
  asserts it completes cleanly.
- .github/workflows/tests.yml: add a scoped `run-tests-windows` job
  (windows-latest, Python 3.11 + 3.13) that builds Scalene and runs this test.
  It's a separate job rather than a matrix entry because the full pytest suite
  is not yet Windows-clean.

Verified locally: the workload completes (ROUNDTRIP_OK 200) standalone and under
`scalene run --cpu-only`; pytest collects both tests with correct per-platform
skips; YAML and ruff pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ping

CI surfaced a SIGSEGV under `scalene run --profile-interval=2` on macOS (and the
same crash would hit Linux): wrapping Scalene's timer-signal handler with a
generation-counting shim collided with Scalene's own native `--stacks` unwinder
(scalene_signal_unwinder chains via g_prev_action and runs _Unwind_Backtrace
from signal context). The extra wrapped frame plus re-entrant signal delivery
during unwinding overflowed into a stack-guard page and crashed in dyld.

Replace the whole sigaction-wrapping mechanism with a side-effect-free signal:
Scalene's CPU sampler re-arms its one-shot interval timer from inside the signal
handler on every tick (ScaleneSignalManager.restart_timer -> setitimer). Since
libscalene already interposes setitimer, just bump the generation counter there
when a CPU-sampling timer (ITIMER_REAL/VIRTUAL/PROF) is armed. The poll/select
retry loops are unchanged: they retry only when the counter advanced across the
interrupted call, so SIGINT/Ctrl-C still propagates. No signal handler is
touched anymore, so Scalene's signal machinery (including the --stacks unwinder)
is left completely intact.

Verified locally with a rebuilt libscalene.dylib: the previously-crashing
command now passes 5/5, as do `--stacks`, `--stacks --profile-interval` with
memory profiling, and a blocking-socket workload under default profiling. The
POSIX regression test was updated to re-arm the timer from its handler (matching
Scalene) and to disarm it before the SIGINT phase; it passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@emeryberger emeryberger changed the title Fix native EINTR failures under wall-clock profiling (#1060) Retry EINTR on poll/select so profiling signals don't break native blocking calls (#1060) Jun 21, 2026
@emeryberger
emeryberger merged commit 8e2b56f into master Jun 21, 2026
57 of 59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ODBC connect() fails with 08S01 / WSAEINTR (10004) when profiling under scalene (Running Docker image on Azure Batch)

2 participants