[tanishqbaweja]

Summary

Fixes #3821

Fsmn_vad_online (runtime/python/onnxruntime/funasr_onnx/vad_bin.py) failed when a call, or a whole stream, was too short for a feature frame. The PR has two commits.

1. Calls without feature frames (#3821)

Fsmn_vad_online.__call__ gets vad_scorer from param_dict only inside if feats.size != 0:, but always runs param_dict.update({"frontend": frontend, "vad_scorer": vad_scorer}) afterwards. When WavFrontendOnline.extract_fbank returns no frames for a chunk, the call fails:

UnboundLocalError: cannot access local variable 'vad_scorer' where it is not associated with a value

(on Python 3.10: local variable 'vad_scorer' referenced before assignment). This happens:

• On the first call, when the first chunk is shorter than 720 samples (45 ms at 16 kHz). With lfr_m=5 the frontend pads the left context with two copies of the first fbank frame, so the first LFR frame needs three 25 ms fbank frames (400 + 2 * 160 = 720 samples). Streaming 10, 20 or 30 ms chunks therefore fails at once. A first chunk of 719 samples fails and one of 720 works.
• Later in the stream, on a chunk that does not complete a new 400-sample fbank window together with the samples cached from earlier chunks. After 1600-sample chunks, 320 samples are waiting, so a 40-sample packet fails, and so does the second of two 80-sample packets.

The README and demo_vad_online.py stream 1600-sample (100 ms) chunks, so they do not hit this error: every non-final chunk completes new fbank frames, and the final call flushes the frames cached so far.

The first commit (a45faf9) moves the param_dict.get("vad_scorer", ...) line out of the if, so the scorer is looked up (or created) on every call, like the frontend. A call that yields no features now returns [] and keeps the frontend and scorer in param_dict. The frontend already holds that chunk's samples in its caches, so they are scored on a later call and the segments match those of 1600-sample chunks.

2. The final call of a stream shorter than 720 samples

Such a stream produces no LFR frame before its final call, and that call still failed or left state behind:

• Under 400 samples there is no fbank frame, and the final flush in extract_fbank called np.stack on the empty LFR splice cache: ValueError: need at least one array to stack (for example a single 320-sample first-and-final call).
• With 400-719 samples, a final call that completes a new fbank frame returned before cache_reset(). With the first commit alone it returned [], but the frontend kept its samples, so the next stream on the same param_dict reported its segments 30 ms late (after a single 480-sample final call).
• A final call that completes no new fbank frame flushed the one or two cached frames, but the waveform passed with them was shorter than one 25 ms window, so E2EVadModel.GetFrameState had no decibel value: IndexError: list index out of range (for example 480 samples followed by an empty final call). The scorer was then never reset.

The second commit (f903065) changes two helpers:

• WavFrontendOnline.extract_fbank skips the flush when no fbank frame is cached, and its not-enough-frames branch no longer returns before the final cache_reset().
• E2EVadModel.GetFrameState scores a frame that has no decibel value as silence. FsmnVADStreaming.GetFrameState (funasr/models/fsmn_vad_streaming/model.py) has the same check, added for Non-default 'decibel_thres' value causes FSMN VAD model crash #2780.

So for a stream shorter than 720 samples, every call returns [], and its final call resets the frontend and the scorer like any final call, so the same param_dict can start the next stream. Streams of 720 samples or more give the same results as before.

Tests

tests/test_funasr_onnx_vad_online_short_chunks.py builds the model with __new__ (no model files) and replaces only infer with a stub that scores loud frames as speech. __call__, WavFrontendOnline and E2EVadModel run unchanged, with the frontend_conf of funasr/models/fsmn_vad_streaming/template.yaml. The first three tests stream 4 s of audio (noise at 0.5-1.5 s and 2.5-3.2 s) and compare each chunking with the 1600-sample result, [[[[220, -1]]], [[[-1, 1660]]], [[[2220, -1]]], [[[-1, 3360]]]]. Three more cover the final call of a short stream:

• test_short_first_and_final_call: a single first-and-final call of 0, 160, 320 or 399 samples returns [].
• test_empty_final_call_after_partial_chunk: 400, 480, 560 or 719 samples, then an empty final call, return [] twice.
• test_param_dict_reuse_after_short_stream: after the short streams [320], [480, 0] and [480] (samples per call), the same param_dict gives the 1600-sample result for the next stream.

I added the file to test-funasr-onnx-package.yml (both path filters, and a step after the empty split-final check), so CI runs it against the built wheel.

Type of change

• Bug fix
• Documentation
• Example or demo
• Runtime or deployment
• Benchmark or evaluation
• Model/training change

Validation

• python -m compileall funasr examples tests
• Docs or links checked
• Runtime/deployment command tested

Windows, Python 3.11.17 (and 3.12.15 for both commits), numpy 1.26.4, onnxruntime 1.31.1, kaldi-native-fbank 1.22.5, librosa 0.11.0. I built wheels from runtime/python/onnxruntime at main (8710e1c), a45faf9 and f903065, installed each into a fresh venv and ran the test scripts as the workflow does:

• python tests/test_funasr_onnx_vad_online_short_chunks.py (6 tests, 16 subtests):
  • main: FAILED (errors=16). Every subtest fails: 11 with the UnboundLocalError, and the four first-and-final calls and [320] with the ValueError. test_demo_chunk_size passes.
  • a45faf9: FAILED (failures=1, errors=10). The first three tests pass, and all 11 subtests of the new ones fail: ValueError for the four first-and-final calls and [320], IndexError for the four partial chunks and [480, 0], and after [480] the next stream gives [[[[250, -1]]], [[[-1, 1690]]], [[[2250, -1]]], [[[-1, 3390]]]].
  • f903065: Ran 6 tests ... OK on 3.11 and 3.12. With pytest: 6 passed, 16 subtests passed.
• tests/test_funasr_onnx_release.py (5 tests), tests/test_funasr_onnx_installed.py (6), tests/test_funasr_onnx_paraformer_online_overlap.py (3) and tests/test_funasr_onnx_paraformer_online_empty_final.py (5): OK with all three wheels.
• black --line-length=100 (24.4.0, the pre-commit pin) leaves the test and e2e_vad.py unchanged. On frontend.py it only reports the existing load_cmvn formatting, as on main. git diff --check is clean.

Outside the test, with the same stub, I compared every call's result between a45faf9 and this branch:

• Every stream length from 0 to 1000 samples, in several chunkings, with and without an empty final call: under 720 samples, every call now returns [] and the final call leaves the frontend, scorer and cache reset; on a45faf9 each of these streams raised or left frontend state behind. From 720 samples on, the segments are identical.
• 300 random streams of 720 samples to 4 s cut from the test audio, in random chunks of 1-1999 samples, half of them ending with an empty final call: identical.
• Random streams under 720 samples, each followed by a 1600-sample stream on the same param_dict: the second stream always gives the 1600-sample result and starts from a fresh cache; on a45faf9 the short streams raised or changed the next stream's segments.
• WavFrontendOnline.extract_fbank alone, with the LFR settings of the VAD and of the streaming Paraformer (300 random streams each): every call that returned features returns the same features and waveform.

On this branch, every chunk size from 1 to 2000 samples still returns exactly the 1600-sample result. I did not run a real FSMN-VAD ONNX model. The stub replaces only the session call.

User impact

Real-time callers of funasr_onnx.Fsmn_vad_online that send chunks shorter than 45 ms (for example 10, 20 or 30 ms frames) or packets of irregular size get VAD segments instead of an UnboundLocalError. A stream that ends before 45 ms of audio, for example a single empty final call, now returns [] instead of raising (UnboundLocalError or ValueError on main), and a param_dict reused after any stream starts the next one like a fresh param_dict. Output for 100 ms chunks is unchanged.

Notes for reviewers

• The UnboundLocalError came in with 修复Fsmn_vad_online多线程调用报错 #2528, which moved the frontend and scorer from instance attributes into param_dict for concurrent calls. Before it, a call without features returned []. vad_bin.py, frontend.py and e2e_vad.py are the same in the funasr-onnx-v0.4.3 release tag as on main.
• The GetFrameState check only changes calls that raised IndexError before. In the runs above, that happened only in streams under 720 samples.
• A final call now also clears in_cache, as the PyTorch model (init_cache when is_final) and the C++ FsmnVadOnline (Reset() when input_finished) do. Single streams are unaffected, because the reset runs after their last call; only a reused param_dict changes, and it now starts like a fresh one.
• WavFrontendOnline is shared with the streaming Paraformer, and its calls that return features are unchanged (see above). A first Paraformer chunk too short for an LFR frame already raises IndexError in Paraformer.__call__ (feats.shape[1] on the empty 1-D array). A first-and-final chunk under 400 samples now fails there as well, with the frontend reset, instead of in np.stack. I left Paraformer's short-chunk handling out of this PR.
• The package version stays at 0.4.3, which the workflow checks.

[LauraGPT]

Thanks for the clear reproducer and the narrowly scoped change. I independently ran the exact three-file head at a45faf98febc93101c15f383801078fe4b654d69 against base 8710e1cc64d45b0170b6818b66fa3ee1d272ef38.

The proposed non-final behavior is verified: the contributor test reports five expected UnboundLocalErrors on base and all three tests pass on this head. However, two terminal short-stream paths described in the PR still crash on the exact head:

• A first and final 320-sample chunk raises ValueError: need at least one array to stack in WavFrontendOnline.extract_fbank.
• A 480-sample non-final chunk now returns [], but an empty final flush then raises IndexError: list index out of range in E2EVadModel.GetFrameState.

Please include the final-flush contract in this fix, with regressions for (1) a short first/final call, (2) a partial cached chunk followed by an empty final call, and (3) reuse of the same param_dict after that terminal short stream so frontend/scorer/cache state cannot leak into the next stream. Returning [] and resetting terminal state would match the pre-#2528 no-feature behavior described in the PR; if a different public contract is intended, please make it explicit and test it.

The validation used the exact base/head vad_bin.py blobs and the submitted test blob under Python 3.12 with the existing isolated funasr-onnx dependencies. It exercises the real frontend and E2EVadModel, while stubbing only the ONNX session that neither failing path reaches. This is CPU frontend/state-machine validation, not pretrained VAD, GPU, ASR-quality, or full-suite evidence. There are currently no hosted checks/statuses on the head, so CI has not been counted as passing.

[tanishqbaweja]

Thanks for checking the terminal paths. I pushed two new commits on top of a45faf9: f903065 and 2700fa5. They take the contract you suggested: a stream shorter than 720 samples returns [] on every call, and every final call resets the frontend, the scorer and in_cache.

1. Short first-and-final call (test_short_first_and_final_call: 0, 160, 320 and 399 samples). WavFrontendOnline.extract_fbank now skips the final flush when no fbank frame is cached, so the call returns [] instead of raising ValueError: need at least one array to stack.
2. Partial cached chunk, then an empty final call (test_empty_final_call_after_partial_chunk: 400, 480, 560 and 719 samples, then 0). The flushed frames come with less than one 25 ms window of waveform, so they have no decibel value. E2EVadModel.GetFrameState now scores such a frame as silence, the same check FsmnVADStreaming.GetFrameState has (added for Non-default 'decibel_thres' value causes FSMN VAD model crash #2780). Both calls return [], and the scorer resets through its normal final path instead of raising IndexError.
3. Reusing the param_dict (test_param_dict_reuse_after_short_stream: after [320], [480, 0] and [480]). in_cache is empty after the short stream and the next stream gives the same segments as with a fresh param_dict. The test stub now changes the cache it returns, so a carried-over cache would show; without 2700fa5, [480, 0] fails there, because its final call runs the model once. [480] covers one more path: a final call that completes a new fbank frame but no LFR frame returned before cache_reset(), so the next stream's segments came 30 ms late. That branch now falls through to the reset.

Red/green, with wheels built from each commit, installed into fresh venvs and run as the workflow does (Windows, Python 3.11 and 3.12):

• a45faf9: FAILED (failures=1, errors=10). All 11 subtests of the three new tests fail (5 ValueError, 5 IndexError, and the 30 ms shift after [480]); the three existing tests pass.
• 2700fa5: 6 passed, 16 subtests passed (f903065 alone fails only the in_cache check for [480, 0]).
• The release, installed, Paraformer overlap and empty split-final tests pass with both wheels.

With the same stub, I also compared every call's result between the two commits, over every stream length from 0 to 1000 samples (four chunkings each) and 300 random streams of 720 samples to 4 s. Streams of 720 samples or more give identical results. Shorter ones now return [] throughout and end with the frontend, scorer and cache reset. The PR description has the contract and these runs.

Still out of scope:

• The streaming Paraformer shares WavFrontendOnline. Its calls that return features are unchanged, and its own handling of a first chunk too short for an LFR frame (IndexError in Paraformer.__call__) is not part of this PR.
• As before, no real FSMN-VAD ONNX model was run; the stub replaces only the session call.

[LauraGPT]

I re-reviewed the five-file revision at 2700fa5aba4ea19d80b88c9393928bedf13b53ff, including the shared frontend and terminal state reset. It addresses the two exceptions and cross-stream cache leak identified in my earlier review. The missing-decibel guard also matches the native FSMN implementation.

Validation on this exact revision:

• 21 focused CPU behavioral tests passed, including the submitted six methods, the two previous terminal reproductions, and installed-wrapper/Paraformer regressions. A separate package-origin test initially failed because my source-only audit overlay lacked distribution metadata; after building and installing the actual exact-head wheel, that test passed too. The initial failure is retained, not counted as a product regression.
• 21 real L20Z CUDA boundary cases passed, against 19 failing cases on the previous head. This uses an explicit fixed analytic ONNX scoring/cache graph and the actual wrapper, frontend and E2EVadModel. ORT profiles show 6,816 CUDA operator events on the new revision. FP32, batch 1; no pretrained model or new weights. This is runtime-contract coverage, not pretrained VAD/ASR quality or a full-repository suite pass.
• Both new-head workflows succeeded. Python 3.11/3.12 package wheel/sdist bytes match all 13 exact-head source files, and both jobs' three overlap, five empty-final, and six short-VAD test methods were checked in the actual logs. Both Python 3.10/3.12 runtime-evaluation JUnit/dependency/source-commit artifacts were also checked.

No protection was relaxed or CI bypassed. The historical full-repository offline/model-hub limitations remain explicitly separate; they are not reported as a new full-suite pass. No package release is being claimed by this review.

[LauraGPT]

Thanks for explicitly calling out the separate streaming Paraformer short-packet
failure. It is now fixed separately in #3827 and available in standalone
funasr-onnx 0.4.5, together with
your VAD fixes from this PR:

python -m pip install --upgrade funasr-onnx==0.4.5

Upgrading only the root funasr package does not upgrade this standalone wrapper.
Version 0.4.4 included the VAD repair but predates the additional Paraformer fix.

The Paraformer change checks feats.size before indexing its shape. Frontend
audio accumulation, terminal reset, and the existing cached-final flushing path
remain intact. The new regressions cover short initial packets, accumulated
small packets, empty final calls with buffered features/cache, and stream reuse.

The validation attached to #3827 includes 71 related CPU checks and 32 real L20Z
FP32/batch-1 synthetic runtime cases, with unchanged outputs/cache fingerprints
for seven previously successful controls. These are runtime-contract checks,
not pretrained ASR accuracy, a full repository suite, or independent validation
of your Windows/pretrained-model workload. Those unchanged-code results were
reused for packaging, not rerun or represented as a new model experiment.

The 0.4.5 release
and official PyPI wheel/sdist were actually downloaded and verified to contain
the same accepted source bytes; the two distribution channels also match in
archive SHA256 and size.