Fix(wasi): enable configurable stack overflow protection

[aayush-kapoor]

What

This PR makes JS_SetMaxStackSize() work on WASI builds.

WASI still defaults to no stack guard, so existing users are unaffected. Embedders can now opt into a limit and receive a catchable JavaScript stack overflow instead of a WASM memory trap.

It also adds WASI tests covering:

• Catchable recursive stack overflow
• Runtime usability after the overflow
• The unchanged default behavior

Why

The Run SDK wants to use quickjs-wasi. the Run SDK currently supports a configurable maxStackSizeBytes limit. That limit currently cannot be enforced because QuickJS ignores JS_SetMaxStackSize() on WASI.

This change unblocks stack-limit support in quickjs-wasi, which can then be used by run

Related run PRs:

• feat: use quickjs-wasi for runtime vercel-labs/run#42
• feat: pass maxStackSize to QuickJS.create vercel-labs/run#43

[aayush-kapoor]

@bnoordhuis could i please bother you for a review? facing a downstream issue 🙏

[bnoordhuis]

The CI failure looks very much related:

Error: failed to run main module `build/qjs`

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
    0:  0x80154 - qjs!JS_DetectModule
    1:   0x1c1f - qjs!main
    2:  0xf0c71 - qjs!__main_void
    3:   0x11ff - qjs!_start
note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
    2: memory fault at wasm address 0xffffffec in linear memory of size 0x80000
    3: wasm trap: out of bounds memory access

[bnoordhuis]

Just curious, was this picked at random or is there a logic to it? 64 KiB because that's the size of one memory page?

[aayush-kapoor]

this is random lol; just meant to be a placeholder. feel free to suggest a better alternative if there is

[bnoordhuis]

64 seems a little on the low side. 256 or 512?

[josefguenther]

There is a constraint that decides this, and the linker default turns out to be exactly the number under discussion.

With no -z stack-size, wasm-ld reserves one page — 65,536 bytes. Measured by reading the __stack_pointer global's initializer: a no-flag build is byte-identical to -z stack-size=65536, on both wasi-sdk 25 (LLD 19.1.5) and wasi-sdk 29 (LLD 21.1.4, which CI installs today). lld/wasm/Driver.cpp corroborates — zStackSize defaults to WasmDefaultPageSize.

update_stack_limit() computes stack_top - stack_size, and stack_top is necessarily below the top of that region: JS_NewRuntime2's prologue is global.get 0; i32.const 32; i32.sub with JS_UpdateStackTop inlined, and any real host adds its own frames on top. So a 64 KiB constant against a 64 KiB reservation puts stack_limit below __data_end, by the host's own call depth. Measured across host depths 0-24, it was below the floor at every one.

Below the floor is ordinary writable static data, and wasm guards nothing there — the shadow stack simply keeps descending into it. Built at df836d1 with this PR, no -z stack-size, with a zeroed .bss canary immediately under __data_end:

reservation	stack_limit vs floor	canary corrupted	JS result
default (64 KiB), sdk 25	−76	224 bytes	RangeError at depth 203
default (64 KiB), sdk 29	−72	224 bytes	RangeError at depth 203
-z stack-size=1048576	+982964	none	RangeError at depth 203

The RangeError is identical in all three, which is the part worth flagging: the JS-visible behaviour looks correct while 224 bytes of static data have already been overwritten. The guard is not failing to fire — it fires after the damage, because the budget it was handed is the whole reservation.

Sweeping the candidates against that default reservation: 32 KiB is safe, with the limit inside the region and no corruption. 64 KiB is the pathological value, because it equals the reservation. 256 and 512 KiB make the subtraction wrap (stack_limit = 4294881808 and 4294619664), after which sp < stack_limit is true immediately and the top-level eval itself throws stack-overflow — nothing runs at all.

So no fixed constant is correct while the WASI target sets no reservation: the safe ceiling is reservation − host_depth, and the reservation is a link-time choice quickjs.h cannot observe.

Suggestion: mirror Emscripten and drop the constant entirely. CMakeLists.txt:348 already gives Emscripten -sSTACK_SIZE=2097152 against the generic 1 MiB JS_DEFAULT_STACK_SIZE — 2:1 — with a comment that the platform default was too small; Windows does the same at /STACK:8388608. Giving the WASI target -z stack-size=2097152 puts it on the identical footing and removes the need for the #ifdef __wasi__ in quickjs.h at all: WASI would take the same 1 MiB default as every other platform, with the headroom Emscripten has been shipping. That also settles 256-vs-512 by making the question go away.

We run the smaller version of that same ratio in production — -Wl,-z,stack-size=1048576 with the host calling JS_SetMaxStackSize(rt, 512 * 1024) — where recursion inside try/catch is caught as RangeError: Maximum call stack size exceeded at depth 2179 with the runtime usable afterwards. We hit this trap independently and made the same update_stack_limit() change; we had missed the rt->stack_size = 0 in JS_NewRuntime2() that this PR also removes.

If you would rather keep a WASI-specific constant, anything at or below half the reservation is fine — but the reservation has to be set first, or the constant is only correct for builds that happen to match the linker default.