Repository navigation
ThrottlingRequestManager does not stop a 429 from retiring the session #2143
Description
Activity
One decision is split across two layers:
_raise_for_session_blocked_status_coderecords the domain delay, then falls through to Session.is_blocked_status_code, where 429 sits in _DEFAULT_BLOCKED_STATUS_CODES — so the throttler's mitigation is instantly undone by a retirement that burns a max_session_rotations slot, the exact churn #1437 complains about. The clean fix is a three-state verdict per response: retry-in-place (throttler owns it: record delay, retry same session), rotate-session, or fail. Only statuses the throttler cannot absorb should ever reach the session layer — and because a domain that rate-limits forever would then retry in place indefinitely, this needs the per-domain stall timeout the report calls for (max_domain_stall_secs plus the keep_alive exemption), in the same PR:if throttler.handles(status): await throttler.backoff(); return RETRY_SAME_SESSION
This decision split is a core case in my published resilient-agent-toolkit. Want me to take a crack at it? Happy to open a PR with working code.
Hi @chenhz01, thanks for taking a look at this
Want me to take a crack at it? Happy to open a PR with working code.
I'm sorry, but this task involves many important components. It will be implemented by one of the core maintainers.
Reacted by Vlada Dusek- added a commit that references this issue
on Sep 29, 2026
_raise_for_session_blocked_status_coderecords the domain delay and then falls through toSession.is_blocked_status_code, and 429 is in_DEFAULT_BLOCKED_STATUS_CODES. So on a domain thethrottler covers, a 429 still raises
SessionError, still retires the session and still spends amax_session_rotationsslot — the churn #1437 was filed about. Onlyignore_http_error_status_codesavoids it, and nothing says so.
This wants fixing together with a stall timeout, in one PR:
_should_retry_requestcapsSessionErroratmax_session_rotationsand refusesHttpClientStatusCodeErroroutright, so today such a domain drains and the crawl ends. Make athrottled 429 free and it never terminates —
run()hangs while the statistics report nothing,which is what the TS port walked into.
has been rate-limiting for that long, and a
keep_aliveexemption.max_domain_stall_secsis aconstructor argument, so landing it separately ships a release where the knob does nothing.
Both halves are in apify/crawlee#3741. From #1762.