You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Stop purging the request queue on repeated run() calls #2305
A repeated run() on the same crawler purges the default request queue (run(purge_request_queue=True) in _basic_crawler.py), so the second run re-crawls everything locally. JS v4 stopped doing that: a repeated run continues where the previous one left off.
Proposed solution
Remove the purge_request_queue argument of run() and the implicit purge. purge() stays on the request manager for anyone who wants to start over.
When a crawl processes nothing because its request manager holds only handled requests, log a warning that says why. This also catches a second crawler on a shared queue and a queue that a previous process worked through. Starting against handled requests stays silent, since that is what resuming looks like.
Document the alternatives for re-crawling: an explicit purge(), or a fresh alias per run.
Activity
fix(crawlers)!: Do not purge the request queue between `run()` calls (#…
vdusek commentedon Oct 27, 2026
resolved by #2343