Skip to content

feat(cli): pip install wheels and installable packages passed to --extra_packages - #7751

Closed
juliays wants to merge 2 commits into
google:mainfrom
juliays:feat/deploy-installable-extra-packages
Closed

juliays wants to merge 2 commits into
google:mainfrom
juliays:feat/deploy-installable-extra-packages

Conversation

@juliays

@juliays juliays commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

2. Or, if no issue exists, describe the change:

Problem:

adk deploy (agent_engine, cloud_run, gke) stages each --extra_packages entry into the build context, COPYs it to /app/<basename>, and adds /app to PYTHONPATH. That works for a plain module or a flat folder of .py files, but silently fails at runtime for standard Python distribution formats:

  • a wheel (.whl) or sdist archive (.tar.gz, .tgz, .zip) is copied into /app as an opaque file and is never importable;
  • a directory with a pyproject.toml or setup.py (notably a src/-layout package) is copied but never installed, so neither its modules nor its declared dependencies are available.

The deploy succeeds and the container then crashes with ModuleNotFoundError.

Solution:

In _get_extra_packages_copy, classify each staged entry:

Entry Dockerfile output
Directory containing pyproject.toml or setup.py COPY to /tmp/extra_packages/<base> + pip install
File ending in .whl, .tar.gz, .tgz, .zip COPY to /tmp/extra_packages/<base> + pip install
Any other file or directory (unchanged) COPY to /app/<base> + ENV PYTHONPATH="/app:$PYTHONPATH"
  • The pip install is emitted in exec form via the existing _to_exec_form helper, so nothing user-controlled reaches a shell.
  • Installable entries are copied to /tmp/extra_packages rather than /app, and ENV PYTHONPATH="/app:$PYTHONPATH" is only emitted when at least one plain entry exists. Otherwise a flat-layout package such as my_pkg/pyproject.toml + my_pkg/my_pkg/__init__.py would be found at /app/my_pkg as a PEP 420 namespace package and shadow its own site-packages install.
  • New _validate_extra_package_name rejects basenames containing quotes, backslashes, path separators, or line breaks before they are interpolated into the COPY instruction (the RUN was already safe via exec form; the COPY was not).
  • No CLI flag changes. Existing plain-folder behavior from Enable common folder ouside of adk-agent for code reusability #4980 / feat(cli) support extra packages in cloud deployments #6860 is byte-for-byte unchanged.

Example rendered block for --extra_packages my_schemas --extra_packages utils-1.0.0-py3-none-any.whl --extra_packages legacy-2.0.tar.gz --extra_packages common (where my_schemas/ has a pyproject.toml and common/ is a plain folder):

COPY --chown=myuser:myuser "my_schemas/" "/tmp/extra_packages/my_schemas/"
COPY --chown=myuser:myuser "utils-1.0.0-py3-none-any.whl" "/tmp/extra_packages/utils-1.0.0-py3-none-any.whl"
COPY --chown=myuser:myuser "legacy-2.0.tar.gz" "/tmp/extra_packages/legacy-2.0.tar.gz"
COPY --chown=myuser:myuser "common/" "/app/common/"
ENV PYTHONPATH="/app:$PYTHONPATH"
RUN ["pip", "install", "/tmp/extra_packages/my_schemas", "/tmp/extra_packages/utils-1.0.0-py3-none-any.whl", "/tmp/extra_packages/legacy-2.0.tar.gz"]

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

New tests in tests/unittests/cli/utils/test_cli_deploy.py:

  • test_extra_packages_pip_installs_installable_packages_and_wheels — end-to-end through to_gke with a pyproject.toml package, a .whl, and a plain folder in one deploy; asserts each COPY destination, that PYTHONPATH is still set for the plain folder, and the exact exec-form RUN ["pip", "install", ...].
  • test_get_extra_packages_copy_pip_installs_setup_py_package — setup.py package is installed and PYTHONPATH is not emitted when there is no plain entry.
  • test_get_extra_packages_copy_pip_installs_source_archives — parametrized over .tar.gz, .tgz, .zip.
  • test_get_extra_packages_copy_rejects_unsafe_basename — parametrized over a newline/RUN injection, an embedded quote, and ../.
$ uv run pytest tests/unittests/cli/utils/test_cli_deploy.py tests/unittests/cli/utils/test_cli_deploy_to_cloud_run.py -q
227 passed in 2.12s

$ uv run mypy src/google/adk/cli/cli_deploy.py
Success: no issues found in 1 source file

$ uv run pre-commit run --files src/google/adk/cli/cli_deploy.py tests/unittests/cli/utils/test_cli_deploy.py
all hooks passed (ruff, isort, pyink, addlicense, ADK compliance checks, codespell, ...)

Manual End-to-End (E2E) Tests:

Not run against Cloud Run / GKE / Agent Engine, as that requires billable cloud resources. The unit tests build the real staging directory and assert on the exact generated Dockerfile, which is the only artifact this change affects. The rendered block above is the actual output of _get_extra_packages_copy on this branch.

To verify locally without cloud access:

adk deploy cloud_run --project=$PROJECT --region=$REGION \
  --extra_packages ./libs/my_schemas \
  --extra_packages ./dist/utils-1.0.0-py3-none-any.whl \
  --temp_folder /tmp/adk_build path/to/my_agent
# then inspect /tmp/adk_build/Dockerfile for the COPY / RUN pip install lines

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

  • The --extra_packages --help text (all three deploy targets) and the run / to_agent_engine / to_gke docstrings are updated to say which entries are pip-installed and which are placed on PYTHONPATH; _get_extra_packages_copy documents why installable entries are kept out of /app. A matching adk-docs update will follow once this lands.
  • Independent of fix(cli): install agent deps before copying source in deploy Dockerfiles #7181 (which reorders requirements.txt install for layer caching); both touch the extra-packages region of the Dockerfile but do not conflict.

…tra_packages

`adk deploy` only copied each --extra_packages entry to /app and put /app on
PYTHONPATH, so a wheel, sdist archive, or pyproject.toml/setup.py package
was staged but never installed and failed with ModuleNotFoundError at
runtime. Detect installable entries (.whl/.tar.gz/.tgz/.zip, or a directory
with pyproject.toml or setup.py), copy them to /tmp/extra_packages, and
pip install them in exec form; plain modules and folders keep the existing
COPY + PYTHONPATH behavior. PYTHONPATH is only set when a plain entry is
present so a flat-layout package cannot shadow its own site-packages install.
Also reject basenames containing quotes, backslashes, separators, or line
breaks before they reach the Dockerfile COPY instruction.

Close google#7750
The --help text and the run/to_agent_engine/to_gke docstrings only described
the copy + PYTHONPATH path. Say which entries are installed with pip and which
are placed on PYTHONPATH, and document in _get_extra_packages_copy why
installable entries are kept out of /app.
copybara-service Bot pushed a commit that referenced this pull request Oct 29, 2026
…tra_packages

Merge #7751

**Please ensure you have read the [contribution guide](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) before creating a pull request.**

### Link to Issue or Description of Change

**1. Link to an existing issue (if applicable):**

- Closes: #7750
- Related: #3936, #4980, #6860 (introduced and extended `--extra_packages`), #2044 (user report of a `.whl` passed to `extra_packages` failing with `ModuleNotFoundError`)

**2. Or, if no issue exists, describe the change:**

**Problem:**

`adk deploy` (`agent_engine`, `cloud_run`, `gke`) stages each `--extra_packages` entry into the build context, `COPY`s it to `/app/<basename>`, and adds `/app` to `PYTHONPATH`. That works for a plain module or a flat folder of `.py` files, but silently fails at runtime for standard Python distribution formats:

- a wheel (`.whl`) or sdist archive (`.tar.gz`, `.tgz`, `.zip`) is copied into `/app` as an opaque file and is never importable;
- a project directory with a `pyproject.toml` or `setup.py` (notably a `src/`-layout package, or a flat-layout project root without a root `__init__.py`) is copied but never installed, so neither its modules nor its declared dependencies are available.

The deploy succeeds and the container then crashes with `ModuleNotFoundError`.

**Solution:**

In `_get_extra_packages_copy`, classify each staged entry:

| Entry | Dockerfile output |
| --- | --- |
| Directory containing `pyproject.toml` or `setup.py` (without a root `__init__.py`) | `COPY` to `/tmp/extra_packages/<base>` + `pip install` |
| File ending in `.whl`, `.tar.gz`, `.tgz`, `.zip` | `COPY` to `/tmp/extra_packages/<base>` + `pip install` |
| Any other file or directory (including a package folder with a root `__init__.py` that may also carry a `pyproject.toml` for tool config) | `COPY` to `/app/<base>` + `ENV PYTHONPATH="/app:$PYTHONPATH"` |

- The `pip install` is emitted in exec form via the existing `_to_exec_form` helper, so nothing user-controlled reaches a shell.
- Only directories that have `pyproject.toml` or `setup.py` **and no root `__init__.py`** are treated as installable project roots. A plain Python package folder with a root `__init__.py` (e.g. `common/__init__.py` + `common/helper.py` + `common/pyproject.toml` for `[tool.ruff]`) stays on `PYTHONPATH` at `/app/<base>` instead of failing `pip install` flat-layout discovery.
- Installable entries are copied to `/tmp/extra_packages` rather than `/app`, and `ENV PYTHONPATH="/app:$PYTHONPATH"` is only emitted when at least one plain entry exists. Otherwise a flat-layout package such as `my_pkg/pyproject.toml` + `my_pkg/my_pkg/__init__.py` would be found at `/app/my_pkg` as a PEP 420 namespace package and shadow its own `site-packages` install.
- New `_validate_extra_package_name` rejects basenames containing whitespace, quotes, backslashes, dollar signs, path separators, or line breaks before they are interpolated into the `COPY` instruction (the `RUN` was already safe via exec form; the `COPY` was not).
- No CLI flag changes. Existing plain-folder behavior from #4980 / #6860 is byte-for-byte unchanged.

Example rendered block for `--extra_packages my_schemas --extra_packages utils-1.0.0-py3-none-any.whl --extra_packages legacy-2.0.tar.gz --extra_packages common` (where `my_schemas/` has a `pyproject.toml` without a root `__init__.py`, and `common/` is a plain folder):

```dockerfile
COPY --chown=myuser:myuser "my_schemas/" "/tmp/extra_packages/my_schemas/"
COPY --chown=myuser:myuser "utils-1.0.0-py3-none-any.whl" "/tmp/extra_packages/utils-1.0.0-py3-none-any.whl"
COPY --chown=myuser:myuser "legacy-2.0.tar.gz" "/tmp/extra_packages/legacy-2.0.tar.gz"
COPY --chown=myuser:myuser "common/" "/app/common/"
ENV PYTHONPATH="/app:$PYTHONPATH"
RUN ["pip", "install", "/tmp/extra_packages/my_schemas", "/tmp/extra_packages/utils-1.0.0-py3-none-any.whl", "/tmp/extra_packages/legacy-2.0.tar.gz"]
```

### Testing Plan

**Unit Tests:**

- [x] I have added or updated unit tests for my change.
- [x] All unit tests pass locally.

New tests in `tests/unittests/cli/utils/test_cli_deploy.py`:

- `test_extra_packages_pip_installs_installable_packages_and_wheels` — end-to-end through `to_gke` with a `pyproject.toml` package, a `.whl`, and a plain folder in one deploy; asserts each `COPY` destination, that `PYTHONPATH` is still set for the plain folder, and the exact exec-form `RUN ["pip", "install", ...]`.
- `test_get_extra_packages_copy_pip_installs_setup_py_package` — `setup.py` package is installed and `PYTHONPATH` is **not** emitted when there is no plain entry.
- `test_get_extra_packages_copy_keeps_package_with_root_init_on_pythonpath` — a package directory with a root `__init__.py` (plus `pyproject.toml` or `setup.py`) stays on `PYTHONPATH` at `/app/<base>` and is not `pip install`ed.
- `test_get_extra_packages_copy_pip_installs_source_archives` — parametrized over `.tar.gz`, `.tgz`, `.zip`.
- `test_get_extra_packages_copy_rejects_unsafe_basename` — parametrized over a newline/`RUN` injection, an embedded quote, whitespace, `$VAR`, and `../`.

```text
$ uv run pytest tests/unittests/cli/utils/test_cli_deploy.py tests/unittests/cli/utils/test_cli_deploy_to_cloud_run.py -q
229 passed in 2.12s

$ uv run mypy src/google/adk/cli/cli_deploy.py
Success: no issues found in 1 source file

$ uv run pre-commit run --files src/google/adk/cli/cli_deploy.py tests/unittests/cli/utils/test_cli_deploy.py
all hooks passed (ruff, isort, pyink, addlicense, ADK compliance checks, codespell, ...)
```

**Manual End-to-End (E2E) Tests:**

Not run against Cloud Run / GKE / Agent Engine, as that requires billable cloud resources. The unit tests build the real staging directory and assert on the exact generated Dockerfile, which is the only artifact this change affects. The rendered block above is the actual output of `_get_extra_packages_copy` on this branch.

To verify locally without cloud access:

```bash
adk deploy cloud_run --project=$PROJECT --region=$REGION \
  --extra_packages ./libs/my_schemas \
  --extra_packages ./dist/utils-1.0.0-py3-none-any.whl \
  --temp_folder /tmp/adk_build path/to/my_agent
# then inspect /tmp/adk_build/Dockerfile for the COPY / RUN pip install lines
```

### Checklist

- [x] I have read the [CONTRIBUTING.md](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) document.
- [x] I have performed a self-review of my own code.
- [x] I have commented my code, particularly in hard-to-understand areas.
- [x] I have added tests that prove my fix is effective or that my feature works.
- [x] New and existing unit tests pass locally with my changes.
- [ ] I have manually tested my changes end-to-end.
- [x] Any dependent changes have been merged and published in downstream modules.

### Additional context

- The `--extra_packages` `--help` text (all three deploy targets) and the `run` / `to_agent_engine` / `to_gke` docstrings are updated to say which entries are pip-installed and which are placed on `PYTHONPATH`; `_get_extra_packages_copy` documents why installable entries are kept out of `/app`. A matching `adk-docs` update will follow once this lands.
- Independent of #7181 (which reorders `requirements.txt` install for layer caching); both touch the extra-packages region of the Dockerfile but do not conflict.

COPYBARA_INTEGRATE_REVIEW=#7751 from juliays:feat/deploy-installable-extra-packages b32c743
PiperOrigin-RevId: 1009646787
@adk-bot

adk-bot commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Thank you @juliays for your contribution! 🎉

Your changes have been successfully imported and merged via Copybara in commit e3d0ec9.

Closing this PR as the changes are now in the main branch.

@adk-bot adk-bot added the merged [Status] This PR is merged label Oct 29, 2026
@adk-bot adk-bot closed this Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merged [Status] This PR is merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cli): pip install wheels, source distributions, and installable packages passed to --extra_packages in adk deploy

3 participants