Each crawler builds its own AutoscaledPool and SystemStatus, so several crawlers in one process oversubscribe the host - there is no way to cap their combined concurrency. JS v4 split the scaling logic out into a ConcurrencySystem that can be constructed once and passed to several crawlers.
JS implementation: apify/crawlee#3917
✍️ Drafted by Claude Code
Each crawler builds its own
AutoscaledPoolandSystemStatus, so several crawlers in one process oversubscribe the host - there is no way to cap their combined concurrency. JS v4 split the scaling logic out into aConcurrencySystemthat can be constructed once and passed to several crawlers.JS implementation: apify/crawlee#3917
✍️ Drafted by Claude Code