Repository navigation
Conversation
…tra_packages `adk deploy` only copied each --extra_packages entry to /app and put /app on PYTHONPATH, so a wheel, sdist archive, or pyproject.toml/setup.py package was staged but never installed and failed with ModuleNotFoundError at runtime. Detect installable entries (.whl/.tar.gz/.tgz/.zip, or a directory with pyproject.toml or setup.py), copy them to /tmp/extra_packages, and pip install them in exec form; plain modules and folders keep the existing COPY + PYTHONPATH behavior. PYTHONPATH is only set when a plain entry is present so a flat-layout package cannot shadow its own site-packages install. Also reject basenames containing quotes, backslashes, separators, or line breaks before they reach the Dockerfile COPY instruction. Close google#7750
The --help text and the run/to_agent_engine/to_gke docstrings only described the copy + PYTHONPATH path. Say which entries are installed with pip and which are placed on PYTHONPATH, and document in _get_extra_packages_copy why installable entries are kept out of /app.
copybara-service Bot
pushed a commit
that referenced
this pull request
Oct 29, 2026
…tra_packages Merge #7751 **Please ensure you have read the [contribution guide](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) before creating a pull request.** ### Link to Issue or Description of Change **1. Link to an existing issue (if applicable):** - Closes: #7750 - Related: #3936, #4980, #6860 (introduced and extended `--extra_packages`), #2044 (user report of a `.whl` passed to `extra_packages` failing with `ModuleNotFoundError`) **2. Or, if no issue exists, describe the change:** **Problem:** `adk deploy` (`agent_engine`, `cloud_run`, `gke`) stages each `--extra_packages` entry into the build context, `COPY`s it to `/app/<basename>`, and adds `/app` to `PYTHONPATH`. That works for a plain module or a flat folder of `.py` files, but silently fails at runtime for standard Python distribution formats: - a wheel (`.whl`) or sdist archive (`.tar.gz`, `.tgz`, `.zip`) is copied into `/app` as an opaque file and is never importable; - a project directory with a `pyproject.toml` or `setup.py` (notably a `src/`-layout package, or a flat-layout project root without a root `__init__.py`) is copied but never installed, so neither its modules nor its declared dependencies are available. The deploy succeeds and the container then crashes with `ModuleNotFoundError`. **Solution:** In `_get_extra_packages_copy`, classify each staged entry: | Entry | Dockerfile output | | --- | --- | | Directory containing `pyproject.toml` or `setup.py` (without a root `__init__.py`) | `COPY` to `/tmp/extra_packages/<base>` + `pip install` | | File ending in `.whl`, `.tar.gz`, `.tgz`, `.zip` | `COPY` to `/tmp/extra_packages/<base>` + `pip install` | | Any other file or directory (including a package folder with a root `__init__.py` that may also carry a `pyproject.toml` for tool config) | `COPY` to `/app/<base>` + `ENV PYTHONPATH="/app:$PYTHONPATH"` | - The `pip install` is emitted in exec form via the existing `_to_exec_form` helper, so nothing user-controlled reaches a shell. - Only directories that have `pyproject.toml` or `setup.py` **and no root `__init__.py`** are treated as installable project roots. A plain Python package folder with a root `__init__.py` (e.g. `common/__init__.py` + `common/helper.py` + `common/pyproject.toml` for `[tool.ruff]`) stays on `PYTHONPATH` at `/app/<base>` instead of failing `pip install` flat-layout discovery. - Installable entries are copied to `/tmp/extra_packages` rather than `/app`, and `ENV PYTHONPATH="/app:$PYTHONPATH"` is only emitted when at least one plain entry exists. Otherwise a flat-layout package such as `my_pkg/pyproject.toml` + `my_pkg/my_pkg/__init__.py` would be found at `/app/my_pkg` as a PEP 420 namespace package and shadow its own `site-packages` install. - New `_validate_extra_package_name` rejects basenames containing whitespace, quotes, backslashes, dollar signs, path separators, or line breaks before they are interpolated into the `COPY` instruction (the `RUN` was already safe via exec form; the `COPY` was not). - No CLI flag changes. Existing plain-folder behavior from #4980 / #6860 is byte-for-byte unchanged. Example rendered block for `--extra_packages my_schemas --extra_packages utils-1.0.0-py3-none-any.whl --extra_packages legacy-2.0.tar.gz --extra_packages common` (where `my_schemas/` has a `pyproject.toml` without a root `__init__.py`, and `common/` is a plain folder): ```dockerfile COPY --chown=myuser:myuser "my_schemas/" "/tmp/extra_packages/my_schemas/" COPY --chown=myuser:myuser "utils-1.0.0-py3-none-any.whl" "/tmp/extra_packages/utils-1.0.0-py3-none-any.whl" COPY --chown=myuser:myuser "legacy-2.0.tar.gz" "/tmp/extra_packages/legacy-2.0.tar.gz" COPY --chown=myuser:myuser "common/" "/app/common/" ENV PYTHONPATH="/app:$PYTHONPATH" RUN ["pip", "install", "/tmp/extra_packages/my_schemas", "/tmp/extra_packages/utils-1.0.0-py3-none-any.whl", "/tmp/extra_packages/legacy-2.0.tar.gz"] ``` ### Testing Plan **Unit Tests:** - [x] I have added or updated unit tests for my change. - [x] All unit tests pass locally. New tests in `tests/unittests/cli/utils/test_cli_deploy.py`: - `test_extra_packages_pip_installs_installable_packages_and_wheels` — end-to-end through `to_gke` with a `pyproject.toml` package, a `.whl`, and a plain folder in one deploy; asserts each `COPY` destination, that `PYTHONPATH` is still set for the plain folder, and the exact exec-form `RUN ["pip", "install", ...]`. - `test_get_extra_packages_copy_pip_installs_setup_py_package` — `setup.py` package is installed and `PYTHONPATH` is **not** emitted when there is no plain entry. - `test_get_extra_packages_copy_keeps_package_with_root_init_on_pythonpath` — a package directory with a root `__init__.py` (plus `pyproject.toml` or `setup.py`) stays on `PYTHONPATH` at `/app/<base>` and is not `pip install`ed. - `test_get_extra_packages_copy_pip_installs_source_archives` — parametrized over `.tar.gz`, `.tgz`, `.zip`. - `test_get_extra_packages_copy_rejects_unsafe_basename` — parametrized over a newline/`RUN` injection, an embedded quote, whitespace, `$VAR`, and `../`. ```text $ uv run pytest tests/unittests/cli/utils/test_cli_deploy.py tests/unittests/cli/utils/test_cli_deploy_to_cloud_run.py -q 229 passed in 2.12s $ uv run mypy src/google/adk/cli/cli_deploy.py Success: no issues found in 1 source file $ uv run pre-commit run --files src/google/adk/cli/cli_deploy.py tests/unittests/cli/utils/test_cli_deploy.py all hooks passed (ruff, isort, pyink, addlicense, ADK compliance checks, codespell, ...) ``` **Manual End-to-End (E2E) Tests:** Not run against Cloud Run / GKE / Agent Engine, as that requires billable cloud resources. The unit tests build the real staging directory and assert on the exact generated Dockerfile, which is the only artifact this change affects. The rendered block above is the actual output of `_get_extra_packages_copy` on this branch. To verify locally without cloud access: ```bash adk deploy cloud_run --project=$PROJECT --region=$REGION \ --extra_packages ./libs/my_schemas \ --extra_packages ./dist/utils-1.0.0-py3-none-any.whl \ --temp_folder /tmp/adk_build path/to/my_agent # then inspect /tmp/adk_build/Dockerfile for the COPY / RUN pip install lines ``` ### Checklist - [x] I have read the [CONTRIBUTING.md](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) document. - [x] I have performed a self-review of my own code. - [x] I have commented my code, particularly in hard-to-understand areas. - [x] I have added tests that prove my fix is effective or that my feature works. - [x] New and existing unit tests pass locally with my changes. - [ ] I have manually tested my changes end-to-end. - [x] Any dependent changes have been merged and published in downstream modules. ### Additional context - The `--extra_packages` `--help` text (all three deploy targets) and the `run` / `to_agent_engine` / `to_gke` docstrings are updated to say which entries are pip-installed and which are placed on `PYTHONPATH`; `_get_extra_packages_copy` documents why installable entries are kept out of `/app`. A matching `adk-docs` update will follow once this lands. - Independent of #7181 (which reorders `requirements.txt` install for layer caching); both touch the extra-packages region of the Dockerfile but do not conflict. COPYBARA_INTEGRATE_REVIEW=#7751 from juliays:feat/deploy-installable-extra-packages b32c743 PiperOrigin-RevId: 1009646787
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
--extra_packages), [Reasoning Engine] ModuleNotFoundError - During ADK Agent deployment despite correct packaging #2044 (user report of a.whlpassed toextra_packagesfailing withModuleNotFoundError)2. Or, if no issue exists, describe the change:
Problem:
adk deploy(agent_engine,cloud_run,gke) stages each--extra_packagesentry into the build context,COPYs it to/app/<basename>, and adds/apptoPYTHONPATH. That works for a plain module or a flat folder of.pyfiles, but silently fails at runtime for standard Python distribution formats:.whl) or sdist archive (.tar.gz,.tgz,.zip) is copied into/appas an opaque file and is never importable;pyproject.tomlorsetup.py(notably asrc/-layout package) is copied but never installed, so neither its modules nor its declared dependencies are available.The deploy succeeds and the container then crashes with
ModuleNotFoundError.Solution:
In
_get_extra_packages_copy, classify each staged entry:pyproject.tomlorsetup.pyCOPYto/tmp/extra_packages/<base>+pip install.whl,.tar.gz,.tgz,.zipCOPYto/tmp/extra_packages/<base>+pip installCOPYto/app/<base>+ENV PYTHONPATH="/app:$PYTHONPATH"pip installis emitted in exec form via the existing_to_exec_formhelper, so nothing user-controlled reaches a shell./tmp/extra_packagesrather than/app, andENV PYTHONPATH="/app:$PYTHONPATH"is only emitted when at least one plain entry exists. Otherwise a flat-layout package such asmy_pkg/pyproject.toml+my_pkg/my_pkg/__init__.pywould be found at/app/my_pkgas a PEP 420 namespace package and shadow its ownsite-packagesinstall._validate_extra_package_namerejects basenames containing quotes, backslashes, path separators, or line breaks before they are interpolated into theCOPYinstruction (theRUNwas already safe via exec form; theCOPYwas not).Example rendered block for
--extra_packages my_schemas --extra_packages utils-1.0.0-py3-none-any.whl --extra_packages legacy-2.0.tar.gz --extra_packages common(wheremy_schemas/has apyproject.tomlandcommon/is a plain folder):Testing Plan
Unit Tests:
New tests in
tests/unittests/cli/utils/test_cli_deploy.py:test_extra_packages_pip_installs_installable_packages_and_wheels— end-to-end throughto_gkewith apyproject.tomlpackage, a.whl, and a plain folder in one deploy; asserts eachCOPYdestination, thatPYTHONPATHis still set for the plain folder, and the exact exec-formRUN ["pip", "install", ...].test_get_extra_packages_copy_pip_installs_setup_py_package—setup.pypackage is installed andPYTHONPATHis not emitted when there is no plain entry.test_get_extra_packages_copy_pip_installs_source_archives— parametrized over.tar.gz,.tgz,.zip.test_get_extra_packages_copy_rejects_unsafe_basename— parametrized over a newline/RUNinjection, an embedded quote, and../.Manual End-to-End (E2E) Tests:
Not run against Cloud Run / GKE / Agent Engine, as that requires billable cloud resources. The unit tests build the real staging directory and assert on the exact generated Dockerfile, which is the only artifact this change affects. The rendered block above is the actual output of
_get_extra_packages_copyon this branch.To verify locally without cloud access:
Checklist
Additional context
--extra_packages--helptext (all three deploy targets) and therun/to_agent_engine/to_gkedocstrings are updated to say which entries are pip-installed and which are placed onPYTHONPATH;_get_extra_packages_copydocuments why installable entries are kept out of/app. A matchingadk-docsupdate will follow once this lands.requirements.txtinstall for layer caching); both touch the extra-packages region of the Dockerfile but do not conflict.