Repository navigation
ThrottlingRequestManager has no effect behind RequestManagerTandem #2238
Description
BasicCrawler finds the throttler with a bare isinstance(self._request_manager, ThrottlingRequestManager) check in three places: the respect_robots_txt_file startup warning (_basic_crawler.py:708), the 429 recording (:1644), and the robots.txt crawl-delay (:1710).
RequestLoader.to_tandem() wraps the manager in a RequestManagerTandem, so await loader.to_tandem(throttler) silently drops the per-domain backoff and the crawl-delay. Both warnings then say the crawler is not using ThrottlingRequestManager, while one is configured.
RequestManagerTandem exposes no accessor for the manager it wraps, so a fix needs new public API. #2158 left this out of scope on purpose.
✍️ Drafted by Claude Code
Activity
chenhz01 commentedon Sep 22, 2026
The bug generalizes past the three call sites: every
isinstance(self._request_manager, ThrottlingRequestManager)check breaks the moment the manager gets wrapped, so to_tandem() silently disables throttling, the robots warning, and 429 recording in one stroke. The durable fix is to stop asking for the concrete class and expose the capability on the protocol — athrottlerproperty that unwraps through any wrapper — so decoration layers stay transparent:@Property
def throttler(self): return self._inner.throttler # Tandem delegates inward
I would pair that with a mechanical eval gate: a regression test that builds every documented manager combination and asserts throttling stays effective through each one. Want me to take a crack at it? Happy to open a PR with working code.
Mantisus commentedon Sep 22, 2026
Hi @chenhz01, thanks for taking a look at this
All work on this issue has been postponed until v2, when the
RequestManagerwill be refactored.