Skip to content

fix(scrapy): restore apify.scrapy imports on crawlee 1.10.1 - #1130

Open
l2ysho wants to merge 2 commits into
masterfrom
fix/scrapy-try-import-extra-name
Open

l2ysho wants to merge 2 commits into
masterfrom
fix/scrapy-try-import-extra-name

Conversation

@l2ysho

@l2ysho l2ysho commented Sep 18, 2026

Copy link
Copy Markdown

Note

TL;DR while fixing CLI pipeline claude brought me here. This is just one-shot fix, probably just starting point (?). It make sense to fix it here as version before was working, or should I fix downstream rather?

The break

crawlee #2208 changed the signature, shipped in 1.10.1 on 2026-09-16:

https://github.com/apify/crawlee-python/blob/c120fdf279bdc7db712302ca60613ae9e7d3ad9e/src/crawlee/_utils/try_import.py#L15

We allow crawlee>=1.8.0,<2.0.0, and crawlee._utils is private, so nothing stopped it reaching users:

File ".../site-packages/apify/scrapy/__init__.py", line 8, in <module>
    with _try_import(__name__, 'run_scrapy_actor'):
TypeError: try_import() missing 1 required keyword-only argument: 'extra_name'

The Actor dies before it can write anything. It has been red on every run and every matrix leg in apify-cli CI since 2026-09-16 14:33 UTC, after passing on every run before 2026-09-16 01:05 UTC (last pass, first fail). The python-scrapy Actor template is broken the same way.

Why CI did not see it

uv.lock pinned crawlee 1.10.0, so no job here ever installed 1.10.1. This PR bumps it, which turns the lock into the regression guard: on 1.10.1 with the source fix reverted, tests/unit/scrapy fails with 52 collection errors.

Verification

  • uv run pytest tests/unit — 586 passed on crawlee 1.10.1.
  • uv run ruff format --check && uv run ruff check — clean.
  • Reverting only src/apify/scrapy/__init__.py on this branch — 52 errors, all TypeError: try_import() missing 1 required keyword-only argument.
  • End to end with the python-scrapy Actor template: on crawlee 1.10.0 the crawl completes and exits 0 with 10 pages scraped; on 1.10.1 without this fix the Actor never starts and writes no dataset.

Open question — the hint names the wrong package

_get_install_hint hardcodes crawlee as the distribution:

https://github.com/apify/crawlee-python/blob/c120fdf279bdc7db712302ca60613ae9e7d3ad9e/src/crawlee/_utils/try_import.py#L32-L37

So with extra_name='scrapy', a user without scrapy installed now gets:

No module named 'scrapy'. Install the optional 'scrapy' extra to use it: pip install 'crawlee[scrapy]'

crawlee has no scrapy extra — it is not in provides_extra — and uv pip install 'crawlee[scrapy]' silently installs nothing, so the advice leaves the user exactly where they started. The correct advice is apify[scrapy].

Two ways to settle it, and I did not want to pick one unilaterally:

  1. Land this as the hotfix now, and change crawlee to take the distribution name (or a ready-made hint) so downstream users get a correct message.
  2. Stop using crawlee's private util here and keep a small local try_import in the SDK, which removes the coupling for good.

Left as a draft for that reason — the code change itself is ready.

🤖 Generated with Claude Code

… crawlee 1.10.1

crawlee 1.10.1 made extra_name a required keyword-only argument of
crawlee._utils.try_import.try_import. Every call site in apify.scrapy
omitted it, so importing the package raised TypeError at module load and
broke every fresh apify[scrapy] install.

Bump crawlee in uv.lock to 1.10.1 as well, so CI installs the version
that exposes the break instead of the pinned 1.10.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.82%. Comparing base (9657922) to head (faed446).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1130      +/-   ##
==========================================
+ Coverage   92.75%   92.82%   +0.06%     
==========================================
  Files          53       54       +1     
  Lines        3519     3553      +34     
==========================================
+ Hits         3264     3298      +34     
  Misses        255      255              
Flag Coverage Δ
e2e 32.42% <0.00%> (-0.32%) ⬇️
integration 52.82% <0.00%> (-0.52%) ⬇️
unit 89.44% <100.00%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ify[scrapy]

Reusing crawlee's private try_import made the hint tell users to run
pip install 'crawlee[scrapy]'. crawlee has no scrapy extra, so that
command installs nothing and leaves the user stuck.

Move the import-guard machinery into apify._try_import, which names the
apify distribution. This also stops a signature change in a private
crawlee module from breaking apify.scrapy at import time, as 1.10.1 did.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@l2ysho l2ysho changed the title fix(scrapy): pass extra_name to try_import so apify.scrapy imports on crawlee 1.10.1 fix(scrapy): restore apify.scrapy imports on crawlee 1.10.1 Sep 18, 2026
@l2ysho
l2ysho marked this pull request as ready for review September 18, 2026 14:09
@l2ysho
l2ysho requested a review from vdusek September 18, 2026 14:09
@apify-service-account apify-service-account added the tested Temporary label used only programatically for some analytics. label Sep 18, 2026
@apify-service-account apify-service-account added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 18, 2026
@l2ysho

l2ysho commented Sep 18, 2026

Copy link
Copy Markdown
Author

@vdusek can you pls take a look if this make sense? Either way after last release CLI pipeline is red, if this is BS and I should fix downstream, let me know.

@l2ysho l2ysho added the adhoc Ad-hoc unplanned task added during the sprint. label Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants