Skip to content

Keep calc_forces!' per-panel loads out of line so Julia 1.12 gets linearize's ForwardDiff Jacobian right on AVX-512 Zen - #378

Closed
1-Bort-1 wants to merge 7 commits into
mainfrom
agent/360-flaky-polar-matrices-forwarddiff-test-on
Closed

1-Bort-1 wants to merge 7 commits into
mainfrom
agent/360-flaky-polar-matrices-forwarddiff-test-on

Conversation

@1-Bort-1

@1-Bort-1 1-Bort-1 commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

TL;DR

On Julia 1.12 with --check-bounds=yes (as Pkg.test runs), LLVM 18's SLP vectorizer miscompiles calc_forces!'s per-panel force assembly for 10-partial Duals on AMD Zen 4/5 CPUs with AVX-512 (JuliaLang/julia#62368). linearize's default ForwardDiff Jacobian came out about 5% off there, and whether CI drew such a runner was the whole #360 flake. The loads and body moment now come from a @noinline panel_body_loads, which keeps the Jacobian exact there, and a new check makes sure the Jacobian does not depend on the ForwardDiff chunk size.

This replaces #378, cut down as asked on #360: only the workaround and the chunk-size test. The POLAR_MATRICES check stays on main's ram-air wing; the affine tables are gone.

What was wrong

Diagnostic CI ran repro379.jl (in #379) against main on 16 Julia 1.12.7 runners per round. It computes linearize's Jacobian with chunk 10 and with chunk 5, which must agree to rounding:

runner --check-bounds=yes default bounds checks --check-bounds=yes, SLP off
EPYC 9V74 (znver4) / 9V45 (znver5), +avx512f rel. 0.042 or 0.050: 7 of 7 in the rounds that logged CPU features 0 3e-19
EPYC 9V74 with AVX-512 masked 3e-19 0 3e-19
EPYC 7763, Intel Sapphire/Granite Rapids, Ice Lake 3e-19 0 3e-19
  • JULIA_LLVM_ARGS=-vectorize-slp=false makes it exact, which places it in Julia 1.12 and LLVM 18 vectorization defect JuliaLang/julia#62368. That issue has the LLVM fix identified (llvm/llvm-project@5d7cf504), still waiting on a backport to Julia 1.12's LLVM.
  • Julia 1.13 (LLVM 20) was correct on the same CPUs. Ordinary runs, without --check-bounds=yes, are not affected. The finite-difference Jacobian was right everywhere.
  • Comparing one Dual{10} solve with two stitched Dual{5} solves put the first wrong value at sol.f_body_3D in calc_forces!. Recompiling copies of it with single statements removed: leaving out either the moment_dist[i] line or the m_body_3D store fixes it. The same loop in a function of its own, or in a standalone script, compiles correctly, so there is no package-free reproducer.

What changed

  • panel_body_loads(axes, dirs, q_dyn, cl, cd, cm, c_span, arm) (src/panel_aerodynamics.jl, @noinline) returns panel_loads plus body_moment, the moment about the reference point. calc_forces! and calculate_results both call it; calculate_results carried the same moment line, so the formula has one source now. Results do not move: norm_fwd was bit-identical with and without the change.
  • test/solver/test_forwarddiff.jl: a nested testset on the POLAR_MATRICES case, AutoForwardDiff(chunksize=5) matches the default chunk to 1e-12. That is the class this bug belongs to, checked with no finite-difference noise.
  • Changelog: a Fixed entry under Unreleased.

Where I would push back

Verification

  • Reproduced first (on Keep calc_forces!' per-panel loads out of line so Julia 1.12 gets linearize's ForwardDiff Jacobian right on AVX-512 Zen #378's history): PR run 35666027777, ubuntu 1.12: 0.04894348185682458 < 0.0001. Control runs on main (35730023185, 35736962099): repro379.jl rel. 0.042 / 0.050 on every AVX-512 Zen runner.
  • Workaround proven on the failing hardware: run 35740722868, 16/16 green, three AVX-512 Zen runners (2× 9V74, 1× 9V45) give repro379.jl rel. 0 under --check-bounds=yes. That run carried the affine test; the src/ change is the same one here.
  • After merging main (e62fc15): the 11 solver and body-aerodynamics test files, 5243/5243 pass locally (juliaserver, Julia 1.13.0). There is no local red: this box's i5-13500 has no AVX-512.
  • Local CI mirror (agent ci-local, Julia 1.13, full Pkg.test): PASS in 6 min · GitHub CI: PASS (all five cells, docs, setup test)
  • Docs build clean (CI) · no REUSE in this repo · up to date with main
  • Risk: another force loop that LLVM 18 SLP-vectorizes the same way, which only an AVX-512 Zen runner under --check-bounds=yes would show.

Scope

+34 / −10 across 6 files. src/ is +21 / −10: the new 13-line function, and its two callers trading their inline moment line for the call. The test gains the 8-line chunk-size testset; the docs page and the changelog take one entry each.

🤖 Generated with Claude Code

Closes #360 · task VortexStepMethod.jl-360

…il's

The ram-air wing's tip sections are ~70%-thick slices (#361) on which
NeuralFoil returns Cd=1 at every alpha, and those outputs differ with the
runner's CPU, so the LOOP solve lands on one of two different solutions
depending on the runner. Tables that are affine in (alpha, delta) make the
bilinear interpolation exact, keep every panel attached, and are the same
bits on every machine.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@1-Bort-1 1-Bort-1 added agent:running Agent task state agent:ci Agent task state and removed agent:running Agent task state labels Sep 21, 2026

@1-Bort-1 1-Bort-1 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (advisory)

Verdict: APPROVE WITH COMMENTS · 2 inline, 0 off the diff

Good

  • The change goes after the cause: affine tables make bilinear interpolation exact, so no grid knot can be a kink, and the tables no longer depend on the CPU; the three-dispatch CI result in the card, with norm_fwd bit-identical across 15 cells, backs this
  • The solver settings, operating point, input indices, rtol and the < 1e-4 assertion are unchanged (the unchanged context lines confirm it), so no test was weakened
  • The knot-offset comment block is gone: it described a mechanism the card shows was wrong (the runners land on two solutions 11° apart, not near a knot)
  • No existing helper duplicates this one: rg finds only ram_air_matrix_wing in test/test_data_utils.jl (NeuralFoil) and a local two-section flat_wing closure in test_backend_comparison.jl
  • The card names the ram_* → matrix_* rename and scopes it to the testset's locals; src/ is untouched and the diff stays at one idea

Not good

  • test/solver/test_forwarddiff.jl:8 — This is a two-line comment block explaining why the tables are affine, which is the kind of why §3/§5 route to the PR card. At most keep one line, e.g. 'affine tables: bilinear interpolation is exact'.
  • test/solver/test_forwarddiff.jl:79 — The Windows ForwardDiff norm of 2.0576, which matched neither converged solution while norm_fd was bit-identical to ubuntu, is left unexplained, and this swap is what stops it being exercised. Under §5 that needs its own issue linked from the PR, or a possible Dual-path bug goes silent with the flake.
  • The card's 'Where I would push back' items (test_results.jl has the same NeuralFoil exposure, fail-fast: false in CI.yml) are noted but not linked to an issue; #361 covers the first only by implication
  • The check now runs only on smooth attached flow with affine tables, so it no longer tests ForwardDiff through a real table's curvature or near stall; the card names this as the risk, but no other test picks that coverage up
  • affine_matrix_wing(n_panels) has a single caller that passes 8; the argument is harmless but a literal inside the helper would say the same thing

claude, rubric CLEAN_CODE.md. A different lab from the implementer
on purpose: a reviewer sharing its blind spots would not flag its mistakes.

Comment thread test/solver/test_forwarddiff.jl Outdated

relative_error(jac, reference) = maximum(abs.(jac .- reference)) / maximum(abs, reference)

# Four-section arc whose POLAR_MATRICES tables are affine in (alpha, delta), so the

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MINOR: This is a two-line comment block explaining why the tables are affine, which is the kind of why §3/§5 route to the PR card. At most keep one line, e.g. 'affine tables: bilinear interpolation is exact'.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 190694d: one line.

Comment thread test/solver/test_forwarddiff.jl Outdated
)
ram_body = BodyAerodynamics([ram_wing])
ram_solver = Solver(ram_wing.n_panels, ram_wing.n_unrefined_sections;
matrix_wing = affine_matrix_wing(8)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MINOR: The Windows ForwardDiff norm of 2.0576, which matched neither converged solution while norm_fd was bit-identical to ubuntu, is left unexplained, and this swap is what stops it being exercised. Under §5 that needs its own issue linked from the PR, or a possible Dual-path bug goes silent with the flake.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #379 for it, with the lost curved-table coverage; linked from the PR.

…ide it

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@1-Bort-1

1-Bort-1 commented Sep 21, 2026 •

Copy link
Copy Markdown
Member Author

Local full suite: FAIL (0 min, Julia 1.13.0, one cell of the matrix)

ijl_toplevel_eval at /cache/build/builder-amdci4-4/julialang/julia-ci/src/toplevel.c:754
ijl_toplevel_eval_in at /cache/build/builder-amdci4-4/julialang/julia-ci/src/toplevel.c:799
eval at ./boot.jl:489
exec_options at ./client.jl:310
_start at ./client.jl:577
jfptr__start_59126.1 at /home/agent/.julia/juliaup/julia-1.13.0+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/builder-amdci4-4/julialang/julia-ci/src/julia.h:2318 [inlined]
true_main at /cache/build/builder-amdci4-4/julialang/julia-ci/src/jlapi.c:985
jl_repl_entrypoint at /cache/build/builder-amdci4-4/julialang/julia-ci/src/jlapi.c:1152
main at /cache/build/builder-amdci4-4/julialang/julia-ci/cli/loader_exe.c:58
unknown function (ip: 0x7fcfba534ca7) at /lib/x86_64-linux-gnu/libc.so.6
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8) at /workspace/srcdir/glibc-2.17/csu/../sysdeps/x86_64/start.S
unknown function (ip: (nil)) at (unknown file)
Allocations: 282270965 (Pool: 282266269; Big: 4696); GC: 117
2026-09-26T10:50:58Z  KILLED by SIGTERM after 5m
ERROR: The Project.toml of the package being tested must have a name and a UUID entry
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types ~/.julia/juliaup/julia-1.13.0+0.x64.linux.gnu/share/julia/stdlib/v1.13/Pkg/src/Types.jl:68
  [2] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}})
    @ Pkg.API ~/.julia/juliaup/julia-1.13.0+0.x64.linux.gnu/share/julia/stdlib/v1.13/Pkg/src/API.jl:578
  [3] kwcall(::@NamedTuple{io::IOContext{IO}}, ::typeof(Pkg.API.test), ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec})
    @ Pkg.API ~/.julia/juliaup/julia-1.13.0+0.x64.linux.gnu/share/julia/stdlib/v1.13/Pkg/src/API.jl:562
  [4] test(pkgs::Vector{PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{})
    @ Pkg.API ~/.julia/juliaup/julia-1.13.0+0.x64.linux.gnu/share/julia/stdlib/v1.13/Pkg/src/API.jl:172
  [5] test(pkgs::Vector{PackageSpec})
    @ Pkg.API ~/.julia/juliaup/julia-1.13.0+0.x64.linux.gnu/share/julia/stdlib/v1.13/Pkg/src/API.jl:161
  [6] test(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::PackageMode, subdir::Nothing, kwargs::@Kwargs{})
    @ Pkg.API ~/.julia/juliaup/julia-1.13.0+0.x64.linux.gnu/share/julia/stdlib/v1.13/Pkg/src/API.jl:189
  [7] test()
    @ Pkg.API ~/.julia/juliaup/julia-1.13.0+0.x64.linux.gnu/share/julia/stdlib/v1.13/Pkg/src/API.jl:178
  [8] top-level scope
    @ none:1
  [9] eval(m::Module, e::Any)
    @ Core ./boot.jl:489
 [10] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:310
 [11] _start()
    @ Base ./client.jl:577

@1-Bort-1 1-Bort-1 added agent:queued Agent task state agent:running Agent task state and removed agent:running Agent task state agent:queued Agent task state labels Sep 21, 2026
@1-Bort-1 1-Bort-1 added agent:waiting-human Agent task state agent:queued Agent task state and removed agent:running Agent task state agent:waiting-human Agent task state labels Sep 22, 2026
@1-Bart-1

Copy link
Copy Markdown
Member

This is not the solution right and should be closed.

@1-Bort-1 1-Bort-1 added agent:running Agent task state and removed agent:queued Agent task state labels Sep 22, 2026
1-Bort-1 and others added 3 commits September 22, 2026 14:49
…ompiles for Dual{10} on Zen 4/5

panel_body_loads (@noinline) now returns panel_loads plus the moment about the
reference point, for calc_forces! and for the body_aerodynamics results loop,
which carried the same moment line. The POLAR_MATRICES testset also checks that
the Jacobian does not depend on the ForwardDiff chunk size.

Refs #360, #379

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

1-Bort-1 and others added 2 commits September 22, 2026 15:56
…bounds=yes

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@1-Bort-1 1-Bort-1 added agent:waiting-human Agent task state and removed agent:running Agent task state labels Sep 22, 2026
@1-Bort-1
1-Bort-1 requested a review from 1-Bart-1 September 22, 2026 15:05
@1-Bort-1 1-Bort-1 added agent:queued Agent task state agent:running Agent task state and removed agent:waiting-human Agent task state agent:queued Agent task state labels Sep 26, 2026
@1-Bort-1 1-Bort-1 closed this Sep 26, 2026
@1-Bort-1 1-Bort-1 changed the title Check POLAR_MATRICES ForwardDiff against finite differences on affine tables, not NeuralFoil's ram-air polars Keep calc_forces!' per-panel loads out of line so Julia 1.12 gets linearize's ForwardDiff Jacobian right on AVX-512 Zen Sep 26, 2026
@1-Bort-1

Copy link
Copy Markdown
Member Author

Not closed yet. This came in two minutes after your A on #360, and #378 has since picked up that work (06f88f3, 2576866): the @noinline workaround that makes the Jacobian exact on AVX-512 Zen, plus the chunk-size test. Asked on #360 whether to close it anyway.

@1-Bort-1 1-Bort-1 added agent:ci Agent task state agent:done Agent task state agent:queued Agent task state and removed agent:running Agent task state agent:ci Agent task state agent:done Agent task state agent:queued Agent task state labels Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:done Agent task state

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky POLAR_MATRICES forwarddiff test on Windows / Julia 1.12 (knot-proximity in piecewise-linear polar interpolation)

2 participants