Migrate development and CI to uv and remove Upstream checks by Billyzhang1229 · Pull Request #1352 · sgkit-dev/sgkit

Billyzhang1229 · 2026-09-27T16:09:16Z

This PR moves development and CI dependency management to uv and removes the Upstream dependency checks. These changes are split into two commits.

Changes included:

Remove the Upstream workflow, its installation script, the README badge, and related contributor documentation.
Replace the requirements files with dependency groups in pyproject.toml and a committed uv.lock.
Provide test, docs, lint, and packaging groups, with an aggregate dev group for local development.
Update contributor instructions to use uv sync and uv run.
Use locked dependencies in standard CI, including scheduled runs, and document periodic, reviewed lockfile updates.
Keep cd docs && make html using the shared .venv. Docs workflows install dask==2026.8.0 for task-graph rendering with NumPy 2, and the Makefile uses uv run --no-sync to preserve that override. Document local setup and restoring locked dependencies before testing.
Migrate package builds and wheel installation checks to uv.
Retain Cubed and Zarr compatibility testing. Limit Cubed upgrades to the experimental packages and any necessary dependency changes.

Existing runtime dependency ranges, optional extras, and Python support are preserved.

Left for follow-up PRs:

Runtime dependency upgrades and relaxed version bounds, including newer Dask, xarray, numba and SciPy versions.
Address any compatibility changes those upgrades require, including a previously identified tajimas_D issue.
Remove the temporary documentation Dask override once a common version passes both tests and documentation builds.
Upgrading the bio2zarr/Zarr stack and broader Zarr 3 support.
Revisit the disabled GPU workflow.

Validation: local tests passed with 100% coverage; strict documentation builds and package smoke tests passed. Build, Windows, Check docs, and Cubed CI passed. The upstream-only Wheels workflow and GPU workflow were not exercised.

Delete the Upstream workflow and its installation script, and remove the README badge and contributor documentation describing the workflow. Fixes sgkit-dev#1351.

Replace requirements files with dependency groups and a committed uv.lock. Use locked environments for standard CI and preserve existing runtime dependency ranges and Python support. Retain the documentation make target and use the shared .venv, with an explicit Dask override for documentation builds. Document local setup and restoration of locked dependencies, plus periodic reviewed lockfile updates. Limit Cubed upgrades to its experimental packages.

timothymillar · 2026-10-07T21:45:47Z

+  "aiohttp",
+  "requests",
+  "graphviz",
+  # Exercise both optional I/O formats with `uv run pytest`.


Can these duplicated deps be replaced with sgkit[plink,bgen] ?

timothymillar · 2026-10-07T21:48:50Z

Thanks @Billyzhang1229, I'm not an expert on uv but this looks great. A couple of minor questions from me:

I think sgkit/_version.py needs to be added to .gitignore
We can probably remove setuptools from the core dependencies with this PR?

Billyzhang1229 · 2026-10-11T22:19:00Z

Thanks for reviewing my PR! I checked all three points and agree with your suggestions. I also audited the current pyproject.toml and pushed the following changes:

Replaced duplicated test dependencies with sgkit[plink,bgen].
Added the generated sgkit/_version.py to .gitignore.
Removed the runtime setuptools dependency while retaining the build dependency, and replaced pkg_resources in the Glow validation script with importlib.metadata.
Configured package discovery to include sgkit’s subpackages and exclude tests, resolving setuptools’ missing-package warnings.
Migrated to SPDX license metadata and raised the build-time setuptools minimum to 77, as required by the setuptools documentation (https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html).
Removed the outdated PLINK dependency comment about pip’s old resolver.
Updated the old validation script to use sgkit.io.plink.read_plink, with a TODO for the later bio2zarr/vcztools migration.
Removed unused isort entries for pkg_resources and setuptools, and mypy overrides for callee, dask_ml, and setuptools.
Removed the obsolete zip-safe setting, following the setuptools documentation (https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#setuptools-specific-configuration).

timothymillar · 2026-10-13T20:40:05Z

Thanks @Billyzhang1229! If your happy then hit ready for review. It looks good to me but would be good to get a second opinion for this one.

Billyzhang1229 · 2026-10-14T12:11:16Z

Thanks, Tim! I’ll mark it ready for review. @jeromekelleher would you mind taking a second look?

jeromekelleher

This looks great, thanks @Billyzhang1229 ! I'd like to do one more pass of making things more "uv native" so that we don't bake in the old complexity, and we try to make all the different dependency combinations visible within pyproject.toml.

jeromekelleher · 2026-10-14T15:03:54Z

-          python -m pip install --pre 'zarr${{ matrix.zarr }}'
-          python -m pip uninstall -y bio2zarr  # TODO: remove when bio2zarr supports Zarr Python 3
+          uv pip install --python .venv/bin/python --pre 'zarr${{ matrix.zarr }}'
+          uv pip uninstall --python .venv/bin/python bio2zarr  # TODO: remove when bio2zarr supports Zarr Python 3


bio2zarr supports Zarr v3 now

You’re right that newer bio2zarr supports Zarr 3, but our locked version still has compatibility issues. Upgrading it involves changes across Zarr, xarray and rechunker, so I’ve restored the workaround; I think these are best addressed together in a broader dependency upgrade.

jeromekelleher · 2026-10-14T15:05:03Z

+        uv sync --locked --all-extras --group docs --group lint --no-default-groups --python "${{ matrix.python-version }}"
+        # Use newer Dask for task-graph rendering with NumPy 2.
+        uv pip uninstall --python .venv/bin/python dask-expr
+        uv pip install --python .venv/bin/python 'dask==2026.8.0'


Why don't we pin this directly in the docs section of pyproject.toml? We should avoid this manual faffing where at all possible.

Pinning Dask 2026.8.0 in the docs group conflicts with the core upper bound. It passes the docs build but breaks the Tajima’s D tests. We could fix that and relax the bound; otherwise, the docs environment still needs an explicit override. This is temporary, once we upgrade the core packages we could simply removing these two lines?

jeromekelleher · 2026-10-14T15:06:50Z

-        python -m pip install -U git+https://github.com/cubed-dev/cubed.git -U git+https://github.com/cubed-dev/cubed-xarray.git -U git+https://github.com/pydata/xarray.git
+        uv sync --locked --all-extras --group test --no-default-groups --python "${{ matrix.python-version }}"
+        # Update the experimental packages without upgrading unrelated dependencies.
+        uv pip install --python .venv/bin/python \


Likewise here, it would be better to declare an additional "cubed-test" dependency group which extends the tests one that declares these dependencies so that uv can resolve theml.

same here, uv cannot resolve it because the current Cubed Git version requires zarr>3.1.2, while sgkit requires zarr<3, so we still need an explicit override until those constraints are resolved.

jeromekelleher · 2026-10-14T15:07:08Z

-        pip install -r requirements.txt -r requirements-dev.txt -r requirements-doc.txt
-        pip install -U dask distributed  # need latest versions to successully build docs
+        uv sync --locked --all-extras --group docs --no-default-groups --python "3.11"
+        # Use newer Dask for task-graph rendering with NumPy 2.


Same point as above.

jeromekelleher · 2026-10-14T15:08:26Z

-          python sgkit-copy/.github/scripts/test_sgkit_plink.py
+          uv venv --python "${{ matrix.python-version }}"
+          uv pip install --python .venv/bin/python --only-binary scipy "${wheel}[bgen,plink]"
+          .venv/bin/python sgkit-copy/.github/scripts/test_sgkit.py


Wouldn't you use uv run python sgkit-copy/.github/scripts/test_sgkit.py or something?

jeromekelleher · 2026-10-14T15:10:01Z

+]
+packaging = [
+  "twine",
+]


So, for things like cubed tests do

cubed-tests = [ { include-group = "test" }, .. cubed deps ]```

Billyzhang1229 · 2026-10-15T08:10:19Z

Thanks, @jeromekelleher ! I’ve ~~removed the bio2zarr uninstall step and~~ update the wheel tests

But moving the docs and Cubed dependencies into groups, would hit conflicts with the core constraints: Dask 2026.8.0 passes the strict docs build but breaks the Tajima’s D tests, while the current Cubed Git version requires zarr>3.1.2 and sgkit requires zarr<3. Could we keep those temporary overrides in this PR and move them into dependency groups once the compatibility issues are addressed?

jeromekelleher · 2026-10-15T09:35:44Z

OK, let's merge this so. I think the next thing would be to move to Zarr >=3, and try to resolve some of these awkward dependencies automatically via uv.

jeromekelleher · 2026-10-15T09:42:53Z

In terms of next steps, I've opened two issues which I think will help untangle some of the dependency issues @Billyzhang1229 - it would be great if you could look at those: #1355 and #1360

Remove upstream dependency CI

b36d639

Delete the Upstream workflow and its installation script, and remove the README badge and contributor documentation describing the workflow. Fixes sgkit-dev#1351.

Billyzhang1229 changed the title ~~Uv upgrade~~ Migrate development and CI to uv and remove Upstream checks Sep 27, 2026

Billyzhang1229 force-pushed the uv-upgrade branch from 62b9776 to 2c28a71 Compare September 27, 2026 17:02

timothymillar reviewed Oct 7, 2026

View reviewed changes

Billyzhang1229 added 7 commits October 11, 2026 21:14

Reuse optional I/O extras in test dependencies

db1c091

Ignore generated version file

b52358a

Remove setuptools runtime dependency

90ddb03

Discover sgkit subpackages explicitly

176cccc

Use SPDX license metadata

0baf1ed

Remove outdated PLINK dependency comment

0739e23

Use sgkit PLINK reader in validation script

4cb2d00

Billyzhang1229 mentioned this pull request Oct 11, 2026

SciPy 1.15.3 fails to load on macOS 27 #1354

Open

Billyzhang1229 added 3 commits October 11, 2026 22:59

Remove unused isort module entries

7c01384

Remove unused mypy overrides

1e0de1c

Remove obsolete zip-safe setting

8935805

Billyzhang1229 marked this pull request as ready for review October 14, 2026 12:11

jeromekelleher reviewed Oct 14, 2026

View reviewed changes

Billyzhang1229 force-pushed the uv-upgrade branch 4 times, most recently from 1854d8a to a32ddff Compare October 15, 2026 08:32

Standardize uv environment targeting and wheel test execution

de63ad6

Billyzhang1229 force-pushed the uv-upgrade branch from a32ddff to de63ad6 Compare October 15, 2026 08:34

jeromekelleher merged commit b73c37b into sgkit-dev:main Oct 15, 2026
14 checks passed

Conversation

Billyzhang1229 commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timothymillar commented Oct 7, 2026

Uh oh!

Billyzhang1229 commented Oct 11, 2026

Uh oh!

timothymillar commented Oct 13, 2026

Uh oh!

Billyzhang1229 commented Oct 14, 2026

Uh oh!

jeromekelleher left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Billyzhang1229 commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeromekelleher commented Oct 15, 2026

Uh oh!

Uh oh!

jeromekelleher commented Oct 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Billyzhang1229 commented Sep 27, 2026 •

edited

Loading

Billyzhang1229 commented Oct 15, 2026 •

edited

Loading