[dkropachev]

Summary

Alternative to scylladb#1036 for scylladb#1007. Missing-host reconciliation must retire hosts and clean up pools, policies, listeners, and reconnectors. Serializing the whole refresh across callbacks can deadlock when a listener calls synchronous refresh while another thread waits for that listener's host lock.

Publish topology changes in short transactions, then drain ordered lifecycle operations without topology or host callback locks. Check exact Host identity before stale queued work can take effect, reserve new hosts until their addition initializes, and discard refresh snapshots overtaken by a newer accepted refresh. Removal notifications precede replacement addition notifications; callback failures do not strand the queue.

This draft explores option 3 from the scylladb#1036 design discussion: separate publication and notification coordination instead of extending topology-first locking across lifecycle callbacks. The upstream PR remains unchanged.

Behavior and compatibility

• refresh_nodes() synchronously publishes node-list and token metadata. If a lifecycle notification is already running, it can return before queued notifications finish. Pool availability can lag publication.
• An active callback can finish after its Host is retired. Stale queued UP, DOWN, and ADD work is discarded. Nested refresh appends notifications instead of delivering them recursively.
• DC/rack changes are queued with removal from the old policy location and addition at the new location. They can lag node-list publication. Newer queued location changes supersede older desired locations, and location publication replaces the token-map cache owner.
• Notifications share an ordered dispatcher. A slow listener delays later notifications, but does not hold the topology publication lock. Listeners must not wait for a later notification from the same cluster.
• Refresh ordering is client-side; it does not establish a server-side topology version. No wire-protocol change is intended.

Decisions

• Choice: Separate topology publication from ordered lifecycle notification delivery. Run notifications without topology or host callback locks so listeners can refresh synchronously without lock inversion. Source: user selected option 3 as a separate draft alternative to cluster: coordinate topology publication and host lifecycle scylladb/python-driver#1036.
• Choice: Synchronous refresh guarantees node-list and token publication, not completion of queued lifecycle notifications or DC/rack policy transitions. An active callback may finish after retirement; subsequent stale work is discarded. This permits callback re-entry while preserving terminal Host identity and notification order.

Refs scylladb#1007, scylladb#921, scylladb#1036.

Branch layout

Targets dk/lifecycle-coordination-base, a snapshot of upstream master at e152e3a11631f07b5ba0bb29b0f8c11e41b446ab. Fork master is behind upstream; the dedicated base keeps this diff limited to the alternative without updating fork master.

Validation

• Full unit run: 1,587 passed, 134 skipped, 931 subtests passed. The final location-event adjustment was followed by the focused suite: 227 passed.
• Cython translation of cluster.py and git diff --check passed.
• Independent correctness findings were fixed; focused verification of the fixes found no remaining findings.
• Live-cluster integration and remote CI have not been run for this fork draft. Required integration scenarios include node removal/rebirth, endpoint movement, and listener-triggered refresh during lifecycle changes.

Pre-review checklist

• I have split my patch into logically separate commits.
• All commit messages clearly explain what they change and why.
• I added relevant tests for new features and bug fixes.
• All commits compile, pass static checks and pass test.
• PR description sums up the changes and reasons why they should be introduced.
• I have provided docstrings for the public items that I want to introduce. Existing listener and refresh contracts are documented.
• I have adjusted the documentation in ./docs/source/. This repository uses docs/api/.
• I added appropriate Fixes: annotations to PR description. This fork draft references upstream issues without closing them.