Skip to content

Stop purging the request queue on repeated run() calls #2305

Closed

Description

@vdusek

A repeated run() on the same crawler purges the default request queue (run(purge_request_queue=True) in _basic_crawler.py), so the second run re-crawls everything locally. JS v4 stopped doing that: a repeated run continues where the previous one left off.

Proposed solution

  • Remove the purge_request_queue argument of run() and the implicit purge. purge() stays on the request manager for anyone who wants to start over.
  • When a crawl processes nothing because its request manager holds only handled requests, log a warning that says why. This also catches a second crawler on a shared queue and a queue that a previous process worked through. Starting against handled requests stays silent, since that is what resuming looks like.
  • Document the alternatives for re-crawling: an explicit purge(), or a fresh alias per run.

This will be a breaking change.

JS implementation: apify/crawlee#4056 (related apify/crawlee#3367)

✍️ Drafted by Claude Code

Activity

  1. added this to the 2.0 milestone on Oct 13, 2026
  2. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    debtCode quality improvement or decrease of technical debt.
    on Oct 13, 2026
  3. self-assigned this
    on Oct 19, 2026
  4. added a commit that references this issue on Oct 27, 2026
    0ed0abd
  5. vdusek commented on Oct 27, 2026

    @vdusek
    CollaboratorAuthor

    resolved by #2343

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

debtCode quality improvement or decrease of technical debt.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions