Fix flaky tests by databyjp · Pull Request #2208 · weaviate/weaviate-python-client

databyjp · 2026-10-16T12:37:27Z

Fix currently flaky tests

Fix flaky Embedded Weaviate tests:

integration_embedded/test_client.py:
- Dynamically assign ports for embedded Weaviate spin-up
Add new tests test/test_embedded.py
- Check ports being assigned w correct patterns 2x pairs, 1 off from each other
- Check that when port assignment fails it correctly progresses to assign different ports
weaviate/embedded.py
- Add helper functions to reliably return open ports
- Hold sets of ports (with _hold_consecutive_ports) before assigning

Fix flaky tokenization test (`integration/test_tokenize.py`):

Use CollectionFactory to prevent collision

Move / refactor test for alias creation (`integration/test_rbac.py`):

Since Authz bugfix sweep weaviate#13324 disallowed empty alias names, create test_create_role_with_empty_alias() containing old test & new test to see if empty alias name now raises
Update pinned WEAVIATE_139 test patch ver to 1.39.12

Fix flaky auth test due to identity provider rate limit (`integration/test_auth.py`):

Add backoff & retry specifically for 429 errors

The 1.39 pin stayed at 1.39.0 after the 2026-08-11 version bump. Later 1.39 patches include backup reliability fixes, and test_hnsw_with_rq4c needs 1.39.2 or later to run at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

recipe_collection created and deleted a fixed collection name. Under pytest -n auto --dist loadgroup, the two tests that use it run on different workers at once and fail with 422 class already exists or 404. collection_factory names the collection after the test node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ssip port start() took one random port for gossip and assumed the next three were free for the data, Raft, and Raft RPC ports. When one of them was taken, Weaviate exited at startup with "bind: address already in use" and the client reported a 30-second timeout (seen in CI on PR #2197). _get_cluster_ports() checks that gossip + 1 is free for the data port, which the server requires, and asks the OS for the Raft and Raft RPC ports. All probe sockets stay open until the four ports are chosen, so they are distinct. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The embedded tests passed fixed HTTP and gRPC ports, several inside the Linux ephemeral port range. In CI, Weaviate exited at startup because another socket held the gRPC port (50040, 50150, 50152). Each test now asks for distinct ports that are free when chosen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The five integration jobs request Okta tokens for the same user at the same time and sometimes exceed its rate limit. _get_access_token now retries a 429 three times with 2, 4, and 8 second backoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

orca-security-eu

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0 0 0 0	View in Orca
Passed	SAST	0 0 0 0	View in Orca
Passed	Secrets	0 0 0 0	View in Orca
Passed	Vulnerabilities	0 0 0 0	View in Orca

This reverts commit 261c878. Weaviate 1.39.10 and later reject an empty alias name in role permissions, which fails test_create_role. The pin will return in a separate PR with that test fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

With CLUSTER_IN_LOCALHOST, Weaviate dials a peer's RPC server at its Raft port + 1 and ignores RAFT_INTERNAL_RPC_PORT (cluster/resolver/rpc.go). Choosing the RPC port independently left nothing listening at Raft + 1, so the node never joined its own cluster and every embedded start timed out on Linux. macOS hands out sequential ports, which hid the bug locally. _get_cluster_ports() now holds two consecutive pairs: gossip/data and Raft/Raft RPC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

codecov-commenter · 2026-10-16T13:26:49Z

Codecov Report

❌ Patch coverage is 90.80460% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.04%. Comparing base (74eed97) to head (9dede35).
⚠️ Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
integration/test_rbac.py	73.68%	5 Missing ⚠️
integration/test_auth.py	75.00%	2 Missing ⚠️
weaviate/embedded.py	94.73%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2208      +/-   ##
==========================================
+ Coverage   88.98%   89.04%   +0.05%     
==========================================
  Files         308      309       +1     
  Lines       24396    24526     +130     
==========================================
+ Hits        21709    21838     +129     
- Misses       2687     2688       +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Weaviate PR #13324 applies the collection-name rule to alias patterns, so 1.38.19+, 1.39.10+, and 1.40.0+ answer 422 "'' is not a valid alias name" where older servers stored an empty alias as "*". The empty-alias case moves out of test_create_role into its own test that expects whichever behaviour the server version has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Restores the pin removed in b0f9556, now that test_rbac.py handles the empty-alias validation added in 1.39.10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

PROFILING_PORT was still drawn from get_random_port() and released before _get_cluster_ports() ran, so a cluster port could land on the just-freed profiling port and Weaviate would exit at startup. Hold all five ports in one ExitStack (_get_server_ports) so the OS guarantees they are distinct. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Only the token POST backed off on 429; the discovery GET hits the same rate-limited provider and failed as a KeyError on token_endpoint. Route both requests through one backoff helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Pinning the server's full error wording across three release branches fails the test on any rewording; matching "alias" keeps the intent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…tion

databyjp and others added 5 commits October 16, 2026 13:13

ci: pin Weaviate 1.39.12

261c878

The 1.39 pin stayed at 1.39.0 after the 2026-08-11 version bump. Later 1.39 patches include backup reliability fixes, and test_hnsw_with_rq4c needs 1.39.2 or later to run at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

orca-security-eu Bot reviewed Oct 16, 2026

View reviewed changes

databyjp and others added 2 commits October 16, 2026 14:08

Revert "ci: pin Weaviate 1.39.12"

b0f9556

This reverts commit 261c878. Weaviate 1.39.10 and later reject an empty alias name in role permissions, which fails test_create_role. The pin will return in a separate PR with that test fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

databyjp and others added 8 commits October 16, 2026 14:31

ci: pin Weaviate 1.39.12

9dede35

Restores the pin removed in b0f9556, now that test_rbac.py handles the empty-alias validation added in 1.39.10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test(rbac): match a stable fragment of the alias rejection message

a66778e

Pinning the server's full error wording across three release branches fails the test on any rewording; matching "alias" keeps the intent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Docstrings

3cba13d

Merge branch 'main' into fix/20261016-flaky-tests

af1c461

Ruff

308dfab

databyjp marked this pull request as ready for review October 16, 2026 18:04

databyjp requested a review from a team as a code owner October 16, 2026 18:04

databyjp changed the title ~~[DRAFT] Fix/20261016 flaky tests~~ Fix flaky tests Oct 16, 2026

databyjp added 2 commits October 16, 2026 19:53

fix(auth): improve backoff handling for HTTP requests and rename func…

c691331

…tion

ruff

4487e19

databyjp requested a review from tsmith023 October 20, 2026 12:07

tsmith023 approved these changes Oct 20, 2026

View reviewed changes

Navigation Menu

Repository navigation

Fix flaky tests - #2208

Fix flaky tests#2208
databyjp wants to merge 17 commits into
mainfrom
fix/20261016-flaky-tests

databyjp commented Oct 16, 2026 •

edited

Loading

Uh oh!

orca-security-eu Bot left a comment •

edited

Loading

Uh oh!

codecov-commenter commented Oct 16, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

databyjp commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fix currently flaky tests

Fix flaky Embedded Weaviate tests:

Fix flaky tokenization test (integration/test_tokenize.py):

Move / refactor test for alias creation (integration/test_rbac.py):

Fix flaky auth test due to identity provider rate limit (integration/test_auth.py):

Uh oh!

orca-security-eu Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Orca Security Scan Summary

Uh oh!

codecov-commenter commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

databyjp commented Oct 16, 2026 •

edited

Loading

Fix flaky tokenization test (`integration/test_tokenize.py`):

Move / refactor test for alias creation (`integration/test_rbac.py`):

Fix flaky auth test due to identity provider rate limit (`integration/test_auth.py`):

orca-security-eu Bot left a comment •

edited

Loading

codecov-commenter commented Oct 16, 2026 •

edited

Loading