Repository navigation
Retire PolicyFingerprint as StreamingConfig's routing key #785
Description
PolicyFingerprint describes itself as legacy, in its own module doc:
//! Legacy routing wrapper for a deployed stored output.
//!
//! An explicit `PrecomputeMaterialization::stored_output_id` takes precedence.
//! Otherwise the compiler allocates a deterministic default from the existing
//! policy fields (including pane layout and cadence). This identifier is not
//! semantic identity: `SummaryDefinitionId` hashes the versioned semantic
//! definition, and several deployed outputs may share that definition.
Its two original jobs have already been split out: semantic identity belongs to SummaryDefinitionId, and explicit routing belongs to stored_output_id.
The migration is real but cannot finish on its current path
stored_output_id adoption is climbing through the stack:
| branch | stored_output_id occurrences |
|---|---|
main |
157 |
issue-752 (#761) |
194 |
test/physical-execution-e2e (#775) |
196 |
test/issue754-level3 (#759) |
196 |
But PolicyFingerprint occurrences do not fall. Counted across every open PR branch, they sit at 300-302 against main's 301, and several branches are at 302. No open PR deletes crates/asap_types/src/policy_fingerprint.rs.
The reason is a loop in PolicyFingerprint::from_config:
pub fn from_config(cfg: &PrecomputeMaterialization) -> Self {
if let Some(output) = cfg.stored_output_id {
return output.fingerprint(); // the explicit id becomes a fingerprint too
}
...
}Adopting an explicit id does not remove a fingerprint; it changes how that fingerprint is derived. The computation is being retired. The type is not, because it is still the routing key.
What actually blocks retirement
StreamingConfig keys its materializations by the fingerprint as a bare u64:
HashMap<u64, PrecomputeMaterialization>StreamingConfig is what the control plane POSTs to the data plane as YAML (control_plane/src/backend_client.rs). So the key is part of a published configuration format, and changing it is a format change rather than an internal refactor.
Proposed work
- Key
StreamingConfig's materializations byStoredOutputIdinstead of a bareu64. - Require
stored_output_id, sofrom_config's hashing fallback has no callers, then delete the fallback and the module. - Bump the published schema version, which now has a single owner and a test enforcing that every producer agrees (
WORKLOAD_SNAPSHOT_VERSION, added in fix(control-plane): give the snapshot schema version one owner #782).
Step 3 is the reason this is worth doing now rather than later: #782 made a schema bump one edit plus whatever its test reports, instead of a literal that some producers follow and others quietly do not.
Not in scope
SummaryDefinitionId stays. It is semantic identity, a different thing from routing, and several deployed outputs may legitimately share one definition.
Related
Separately worth checking while in this area: plan_id is computed with std::collections::hash_map::DefaultHasher (control_plane/src/physical/compiler.rs), whose algorithm the standard library does not guarantee across Rust releases, yet plan_id is compared across processes and contract tests reject a changed installed generation. A toolchain upgrade would change it.
Activity
zzylol commentedon Sep 28, 2026
Correcting the blocker I described above: I had not checked how the key is serialized, and it is not.
StreamingConfig's materialization map carries#[serde(skip)]. It is a process-local index, rebuilt byfrom_precompute_planwhen the config is decoded. So re-keying it is an internal refactor, not a change to the published configuration format, and step 3 above (bump the schema version) is not required.Two further facts make the first two steps cheaper than I said:
StoredOutputIdandPolicyFingerprintare the sameu64wearing two type names, with free conversions in both directions:Both are
#[serde(transparent)], so wherever either is serialized the bytes are identical.And
PolicyFingerprintnever appears as a serde field on any struct. The only place its value reaches a published document is inside a formatted string:That
schema_iddoes travel inPrecomputePlan.schemasand is validated on install, but since the two types hold the same number, passing aStoredOutputIdproduces the identical string.So the work is: re-key the map and give it a name that says what it holds, take
StoredOutputIdinstate_schema_id, then makestored_output_idrequired so the hashing fallback infrom_confighas no callers and the module can go. No format change, no version bump, and nothing aboutSummaryDefinitionIdmoves.refactor: route precompute outputs on StoredOutputId, not a bare u64 (#…