Conversation
… crawlee 1.10.1 crawlee 1.10.1 made extra_name a required keyword-only argument of crawlee._utils.try_import.try_import. Every call site in apify.scrapy omitted it, so importing the package raised TypeError at module load and broke every fresh apify[scrapy] install. Bump crawlee in uv.lock to 1.10.1 as well, so CI installs the version that exposes the break instead of the pinned 1.10.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1130 +/- ##
==========================================
+ Coverage 92.75% 92.82% +0.06%
==========================================
Files 53 54 +1
Lines 3519 3553 +34
==========================================
+ Hits 3264 3298 +34
Misses 255 255
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ify[scrapy] Reusing crawlee's private try_import made the hint tell users to run pip install 'crawlee[scrapy]'. crawlee has no scrapy extra, so that command installs nothing and leaves the user stuck. Move the import-guard machinery into apify._try_import, which names the apify distribution. This also stops a signature change in a private crawlee module from breaking apify.scrapy at import time, as 1.10.1 did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
l2ysho
marked this pull request as ready for review
September 18, 2026 14:09
Author
|
@vdusek can you pls take a look if this make sense? Either way after last release CLI pipeline is red, if this is BS and I should fix downstream, let me know. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
TL;DR while fixing CLI pipeline claude brought me here. This is just one-shot fix, probably just starting point (?). It make sense to fix it here as version before was working, or should I fix downstream rather?
The break
crawlee #2208 changed the signature, shipped in 1.10.1 on 2026-09-16:
https://github.com/apify/crawlee-python/blob/c120fdf279bdc7db712302ca60613ae9e7d3ad9e/src/crawlee/_utils/try_import.py#L15
We allow
crawlee>=1.8.0,<2.0.0, andcrawlee._utilsis private, so nothing stopped it reaching users:The Actor dies before it can write anything. It has been red on every run and every matrix leg in apify-cli CI since 2026-09-16 14:33 UTC, after passing on every run before 2026-09-16 01:05 UTC (last pass, first fail). The
python-scrapyActor template is broken the same way.Why CI did not see it
uv.lockpinned crawlee 1.10.0, so no job here ever installed 1.10.1. This PR bumps it, which turns the lock into the regression guard: on 1.10.1 with the source fix reverted,tests/unit/scrapyfails with 52 collection errors.Verification
uv run pytest tests/unit— 586 passed on crawlee 1.10.1.uv run ruff format --check && uv run ruff check— clean.src/apify/scrapy/__init__.pyon this branch — 52 errors, allTypeError: try_import() missing 1 required keyword-only argument.python-scrapyActor template: on crawlee 1.10.0 the crawl completes and exits 0 with 10 pages scraped; on 1.10.1 without this fix the Actor never starts and writes no dataset.Open question — the hint names the wrong package
_get_install_hinthardcodescrawleeas the distribution:https://github.com/apify/crawlee-python/blob/c120fdf279bdc7db712302ca60613ae9e7d3ad9e/src/crawlee/_utils/try_import.py#L32-L37
So with
extra_name='scrapy', a user without scrapy installed now gets:crawlee has no
scrapyextra — it is not inprovides_extra— anduv pip install 'crawlee[scrapy]'silently installs nothing, so the advice leaves the user exactly where they started. The correct advice isapify[scrapy].Two ways to settle it, and I did not want to pick one unilaterally:
try_importin the SDK, which removes the coupling for good.Left as a draft for that reason — the code change itself is ready.
🤖 Generated with Claude Code