Skip to content

P0 SMV SSOT reliability: concurrent snapshot safety and fail-closed START - #248

Closed
masarray wants to merge 18 commits into
fix/studio-smart-serial-reconnect-20261019from
fix/studio-smv-ssot-concurrent-publisher-20261019
Closed

masarray wants to merge 18 commits into
fix/studio-smart-serial-reconnect-20261019from
fix/studio-smv-ssot-concurrent-publisher-20261019

Conversation

@masarray

Copy link
Copy Markdown
Owner

Audit scope (GUI + ESP32-P4 + PTP AUTO)

Verified source at PR #247 HEAD f3e4456:

  • GUI START derives 8 enabled/quality/amplitude/angle channel setpoints plus frequency and CT shape from the displayed ListModels, ordered into a 19-command exclusive serial batch; queue admission alone is NOT semantic ACK or wire parity.
  • Firmware publishes each accepted SET to an immutable generation via SvLiveSignalBank, read by 4 kHz publisher and console diagnostics. Physical values, PTP lock time and network signal parity require fresh PCAP and hardware logs.
  • Source AUTO PTP intentionally requires 1 Hz phase observations + live MAC descriptor evidence, 3 clean epochs; code-level gates do not certify <10 s on board.

P0 defects found and corrected

  1. Bank's single reader_index_ was only safe if all snapshot readers serialized; firmware uses concurrent publisher and console diagnostics, so the second reader can replace first reader protection while writer republishes to a bank still copied by the first. Replace with 4 bank slots + per-bank atomic reader pin counts. No mutex, allocation or I/O in the 4kHz snapshot path.
  2. Firmware STOP did not clear pending g_start_request. Explicit STOP now cancels pending START; publisher checks cancellation before and after build.
  3. START could set g_running=true and claim START accepted, then fail profile/template preparation while staying logically RUNNING with no SV. Publisher now live_control_force_stop() + GPTimer teardown on all profile/template failures.

Regression

  • Portable multithreaded 20k-generation stress test with simultaneous publisher/console readers asserting coherent generation/8 channel tuples.
  • Qt structural guard for SSOT bank readers and fail-closed START; GCC/Clang/MSVC tests plus ESP-IDF build required.
  • No unmeasured smpSynch=1, no loosened clock gate, no silent GUI claim of SV wire parity. PR remains unmerged pending physical 4I4V and PTP acceptance.

Remaining P0 hardening

Serial SETs are independent firmware updates. A fully atomic, firmware-confirmed scenario generation commit + CRC/profile hash and independent capture-side exact comparison is needed to prove every START signal matches visible table; don't call that solved by queue exclusivity. Use physical tests to verify 4000 fps, zero gaps, RMS and phase, VLAN/APPID/svID/quality, AUTO stable LOCAL <=10s, hot-plug and reboot.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T02:15:00.222003Z — PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +786 to +788
if (!live_tx_running()) {
stop_sample_clock(sample_clock);
continue;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Publish the running state before exposing the START intent

If the publisher is already awake while handle_line() executes START, it can consume g_start_request after that flag is stored but before the subsequent g_running=true store. This new check then observes the old stopped state, stops the clock, and discards the only request; the control task afterwards marks the device RUNNING and reports START accepted, but no timer is started and the publisher may block indefinitely. Publish these fields as one state transition, or at least make the running state visible before the request can be consumed.

AGENTS.md reference: AGENTS.md:L74-L78

Useful? React with 👍 / 👎.

Comment on lines 153 to +154
static constexpr std::uint32_t kNoReader = 3U;
std::array<SvLiveSignalState, 3> banks_{};
mutable std::atomic<std::uint32_t> reader_index_{kNoReader};
std::array<SvLiveSignalState, 4> banks_{};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reserve an out-of-range sentinel for the fourth bank

Index 3 is now a valid bank, but it remains the kNoReader sentinel. When bank 3 is the only inactive unpinned slot—for example, active bank 2 while two concurrent readers still pin banks 0 and 1—the loop selects 3 and the following sentinel check returns false, so the advertised spare bank is never usable and a valid signal update is dropped. Use banks_.size() or another out-of-range value as the sentinel.

AGENTS.md reference: AGENTS.md:L120-L127

Useful? React with 👍 / 👎.

Comment thread tests/test_live_signal.cpp Outdated
publisher.join();
diagnostics.join();
CHECK(committed > 100U);
CHECK(read_count.load() > 100U);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Synchronize the readers before asserting their read count

The main thread can finish all 20,000 publishes and set done before either newly created reader receives meaningful CPU time; each reader then performs only its mandatory first iteration and this assertion fails with a count of 2. Thread scheduling provides no minimum progress guarantee here, so the regression test is flaky and also does not deterministically exercise reads overlapping writes. Add a startup barrier and require a controlled number of reader iterations before ending the test.

AGENTS.md reference: AGENTS.md:L230-L234

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Additional concrete GUI-vs-firmware semantic defect discovered during end-to-end audit (19 October 2026): QML validFrequency(), StudioDeviceController::setFrequency(), and DeviceController::startWithScenario()/setFrequency() allowed 0 < Hz < 1.0, but portable generator's SvLiveSignalBank::validate() rejects 0 < frequency_millihz < 1000. Consequently a GUI displaying 0.5 Hz could queue FREQ 500, firmware would reject it, and the subsequent START could still use previous frequency. Current PR HEAD now has one C++ host validity rule Hz == 0 || 1 <= Hz <= 1000 shared by both controller paths and matching QML validation, with negative/positive regression at -1/0/0.5/1/50/1000/1000.001. Source HEAD 14db47a0f4c8dcaf4b0fe912d04c390d2613690c; earlier CI runs are superseded and must not be cited for this final HEAD. This specifically prevents a silent injection mismatch rather than cosmetic UI change. Outstanding: firmware applied-state hash/ack and wire capture comparer remain mandatory for full parity acceptance; no physical <=10s LOCAL claim is made.

masarray added a commit that referenced this pull request Oct 22, 2026
Replace single-reader pin with per-bank atomic pins and four banks, fix sentinel to 4 so last bank can be selected, and add 2-reader/1-writer regression. Keep realtime path lock-free.

Copy link
Copy Markdown
Owner Author

Stage-main housekeeping · 2026-10-23 · SUPERSEDED, NOT MERGED

Original source SHA: 832db687fb9cb6c15c2934c61d43057e47a67fb2 (branch remains intact). Canonical integration: Draft #286 inspected at 042df0d40ddd3f061ebeebc81c662c585827e346; target is main but no merge authorized or performed. Source head is diverged from #286, so closure documents verified successor behavior, not commit ancestry or identical patches.

Code/regression replacement evidence: Four-bank, per-bank atomic reader pin counts in live_signal_state.hpp; portable test_live_signal.cpp multi-reader stress with 20,000 publishes; STOP clears pending start request and firmware live_control_force_stop() fail-closed; current CI build retained. Source feature-level parity, not branch-history parity.

Exact-head software check: Native Preview 39882032904 SUCCESS (ESP32 firmware + native Windows preview/session harness). The larger Draft-routed CI matrix was SKIPPED; success is not physical STOP safety, real 4I4V waveform parity or SIPROTEC acceptance.

Open acceptance ownership: #286 release gate; #42 physical live-setpoint operation, no START/STOP/PTP reset and no smpCnt gaps; #154 selected IED PTP lock/smpSynch/4I4V relay evidence, sustained SOAK and STOP/START. The separate #253 LIVE CFG candidate remains OPEN pending exact hardware behavior; #284 external-GM actuator remains OPEN. No source branch removed, no PR merged, no firmware or UI code changed, no main update.

This source PR no longer needs a competing review/merge route because new production code owns its implemented functionality. Historical source/candidate remains recoverable in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant