Repository navigation
How can I re-queue a URL? #942
Description
Crawlee optimizes requests by suppressing redundant URLs and giving up on a URL after reaching a configurable retry limit. This is great, but I encountered an edge case.
Situation: You're extracting data from a page and realize that it wasn’t downloaded properly.
Ideal Goal: Re-queue the URL while decrementing its retry counter.
Alternative: Add the URL back to the queue with a fresh retry counter.
How can I achieve either of these?
Activity
B4nan commentedon Jan 29, 2025
You can enqueue it again with a different
unique_key, that's what we use for deduplication (and it defaults to the URL itself).vdusek commentedon Jan 30, 2025
I am afraid this is not possible.
You can use the
always_enqueue=True:which will generate "unique"
unique_keyunder the hood. And add the request to the request queue.