Skip to content

fix: compare the schema behind a changed $ref - #668

Open
josephaxisa wants to merge 1 commit into
pb33f:mainfrom
josephaxisa:fix/663-ref-change-compares-content
Open

josephaxisa wants to merge 1 commit into
pb33f:mainfrom
josephaxisa:fix/663-ref-change-compares-content

Conversation

@josephaxisa

@josephaxisa josephaxisa commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #663.

When a $ref was replaced by an inline schema, an inline schema by a $ref, or a $ref pointed at another schema, CompareSchemas recorded the non-breaking $ref change and returned. The new schema was never compared, so the repro in #663 ($ref: Pet becoming an inline schema with required: [name]) reported 1 non-breaking change, while the same edit between two inline schemas is breaking.

The $ref change is now kept, and the two schemas are then compared like any other two schemas, as e935b3e did. Comparison stops at the $ref change (keeping it rather than returning nil) when:

  • either side doesn't resolve (a missing target, or an external one with SkipExternalRefResolution). Otherwise everything would be reported as removed.
  • either side is on a reference loop, or the document wasn't checked for circular references (SkipCircularReferenceCheck), since a loop can't be recognised there. Rolodex gets a small HasCheckedForCircularReferences() getter for this.
  • a 3.1 $ref with sibling keywords is involved. Its internal allOf form doesn't line up with the referenced schema field by field. That case needs its own fix.

Three helpers that recognise equivalent forms now look through a reference, so a change made through a $ref is reported like the same change made inline:

Existing tests that asserted a single change now also assert the change behind the $ref, e.g. TestCompareSchemas_PropertyRefChange_Fail (int → string, breaking). In TestCheckExplodedFileCheck, both references change and c-alt.yaml has a new description, so the test now expects 3 non-breaking changes. The three CheckRefChangeCircular tests keep the $ref change instead of expecting nil.

New tests: the #663 repro both ways and checked against the inline equivalent, a retargeted $ref, an identical inline copy (still no change), the simple allOf view, the nullable wrap, siblings and circular targets stopping at the $ref, unresolved targets, a skipped circular check, and the equivalent scalar union. The touched functions are at 100% coverage. The full suite and openapi-changes' tests pass.

Things worth knowing:

  • Following a changed $ref costs what following an identical external $ref already costs: a deep graph of shared schemas is compared once per path. I checked a 25-level diamond where every level is renamed. Memoizing per schema pair would make the comparison cheap, but the report would still list the change once per path, so I left that for a separate discussion.
  • Changes found behind a $ref point at the target schema's lines, as they do when an external $ref is followed today.
  • oneOf/anyOf pairing scores ref-vs-ref candidates on their content now, instead of one $ref change each.
  • Real-world spec check (stripe.yaml 2024-01 → 2026-09 history, with fix: only treat schemas on a reference loop as circular #667): Time and memory are unchanged, and what-changed catches +117 additional breaking changes (407 → 524 breaking) behind retargeted $refs.

Related: #662. In documents with any circular reference, this only takes effect with that fix.

@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (3406d80) to head (ffb70f0).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #668   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          309       309           
  Lines        39279     39298   +19     
=========================================
+ Hits         39279     39298   +19     
Flag Coverage Δ
unittests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@daveshanley daveshanley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right direction. A breaking change hidden behind a $ref swap is exactly what openapi-changes should catch. One blocker inline.

Also needs #667 first. On Stripe with quick hash on, this branch alone still reports 0 of 300 breaking changes.

if base.CheckSchemaProxyForCircularRefs(l) || base.CheckSchemaProxyForCircularRefs(r) {
// if we have a circular reference, we can't do any more work here.
return nil
if !canCompareBehindRefChange(l, r) || low.AreEqual(l.Schema(), r.Schema()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This goes exponential on shared schema graphs. Every path to a schema pair gets compared again, and for local refs main never did that (identical local $refs short-circuit, I checked: 1 change at every depth).

Repro: a diamond where Ln has a and b both pointing at Ln+1, and the right side renames every level to Rn.

depth main this PR
8 19 changes, <1ms 785, 6ms
12 27 12,313, 42ms
16 35 196,641, 710ms

×4 every two levels, so depth 25 is hundreds of millions of changes. Renaming a family of schemas (v1 → v2) is a normal thing to do, so this will bite.

Memoize per (l, r) schema pair for the comparison: follow the ref the first time, record only the $ref change after that. That keeps the #663 fix and bounds the work by the number of schema pairs.

@daveshanley

Copy link
Copy Markdown
Member

#667 shipped in v0.41.7, so rebase onto main and that dependency is sorted.

On your open question about reporting per path: report the full diff behind a retargeted $ref once per schema pair. Every other path to that pair records the $ref change only. That's the memo from my inline comment, and it fixes both problems: the work stays bounded, and the report stays readable.

When a schema $ref was replaced by an inline schema, an inline schema by
a $ref, or a $ref pointed somewhere else, CompareSchemas recorded a
non-breaking $ref change and returned, so changes in the new schema were
never reported. A $ref to a schema with type: int becoming an inline
type: string was a single non-breaking change.

Keep the $ref change, then compare the two schemas as it would compare
two inline schemas. It still stops at the $ref change, now keeping that
change instead of returning nil, when either side doesn't resolve, is on
a reference loop, or comes from a document that was not checked for
circular references (a loop can't be recognised there), and when a 3.1
$ref with sibling keywords is involved, since its internal allOf form
doesn't line up with the referenced schema field by field. Rolodex gains
HasCheckedForCircularReferences for that check.

The helpers that recognise equivalent forms now look through a
reference, so a change made through a $ref is reported like the same
inline change: the object composition handling from pb33f#650 ($ref: Thing
becoming anyOf: [$ref: Thing, type: 'null']), the simple allOf view, and
a type array matching an anyOf of scalar types.

Fixes pb33f#663
@josephaxisa
josephaxisa force-pushed the fix/663-ref-change-compares-content branch from bc53102 to ffb70f0 Compare October 25, 2026 00:35
@josephaxisa

Copy link
Copy Markdown
Contributor Author

Thanks for the review @daveshanley. Rebased onto main (with #667).

I tried the per-pair memo you suggested, scoped to CompareDocuments. The first path to a schema pair follows the $ref, and later paths record only the $ref change. Sibling properties are compared in goroutines, so once the comparison finishes the full diff moves to the path that comes first in the document, which keeps the output the same from run to run. It fixes your diamond (depth 16: 67 changes, 4 per level after the first) and keeps stripe.yaml at 300/300 breaking.

It breaks union matching, though. buildSchemaCompositionPairCandidateMatrix calls CompareSchemas on every anyOf/oneOf/allOf candidate pair and keeps the pairing with the fewest changes. With the memo, any candidate after the first scores as a single $ref change, so the pairing goes wrong.

For example, anyOf [$ref A, $ref B, $ref X] -> [$ref B2, $ref A2] pairs A with B2 and reports invented property changes instead of the two real type changes. With several such unions in one schema, the result also depends on goroutine timing. A package-level memo also can't tell two concurrent comparisons of the same left document apart.

The way I see to fix both is an unexported comparison context passed through the schema comparison. CompareSchemas keeps its signature, candidate scoring runs without the memo, and each CompareDocuments call gets its own. That touches a fair amount of what-changed/model, so I wanted to check you're happy with that shape before pushing it. Or is there a better place for per-comparison state?

On a separate note, I noticed that SchemaChanges.TotalBreakingChanges sums AllOfChanges twice, so breaking changes inside an allOf are double counted on main. Happy to send that as its own PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Breaking changes are missed when a schema $ref is replaced or retargeted

2 participants