Repository navigation
Conversation
The SME unit moves data 2-4x faster than NEON or the generic copies. In sme2_gemm_copy.h, s2c_rows() uses predicated streaming vectors and s2c_cols() transposes through ZA, for pieces of 512 elements or more; smaller pieces and CPUs without SME2 stay on NEON. The new B-side copies (S/DGEMMONCOPY, S/DGEMMOTCOPY) use the same helpers with the layout of the generic copies (UNROLL_N, then 8/4/2/1). Part of OpenMathLib#6146
…ement tests The diagonal block goes through the panel copy; then each row gets its zero side and its (unit or inverted) diagonal element. Part of OpenMathLib#6146
s2_c_store_alpha() (C = alpha ZA in the TRMM kernel, C += alpha ZA in the GEMM kernel for alpha other than 1 and -1) read ZA one slice at a time. It now reads four slices per MOVA, as s2_c_store(). This makes the TRMM kernel about 1.5x (n = 1024) to 2.4x (n = 256) faster. Part of OpenMathLib#6147
In the 64-row block on the diagonal of a left TRMM, the A panel rows of tile Q are zero for VL * Q (lower: the last) steps. The kernel now leaves out the outer products of those tiles; about 4% (fp32) and 12% (fp64) faster at n = 256. Part of OpenMathLib#6147
The triangle copies zeroed with memset and fixed the diagonal block with scalar stores on the core, between two streaming copies. The kernel then had to read these lines back from the core's L1. Now a copy of more than 512 elements is one streaming-mode call: the kept part with s2c_rows_s/s2c_cols_s, the zero part and the diagonal block with predicated streaming stores. Part of OpenMathLib#6147
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #6147. Stacked on #6154 (its two commits are the first two here); only the last three commits are new.
Before: on VORTEXM4 (with #6154), one thread, STRMM at n = 1024 ran at about 830-1000 GFLOPS against about 1550 for Accelerate, and at n = 256 at 340-370 against about 1350. Timing the parts separately showed the TRMM kernel as the main cost: for fp32 at n = 256 it needed about 4x the time per flop of the GEMM kernel, although the code is the same apart from the store. The difference is
s2_c_store_alpha()(C = alpha AB, used by the TRMM kernel), which read ZA one slice at a time.After: all eight STRMM/DTRMM variants are faster at every size measured, by 1.3-3.8x; the left side reaches 0.95x of Accelerate at n = 1024 (fp32).
How:
sme2_gemm_tile.h:s2_c_store_alpha()reads four ZA slices per MOVA, ass2_c_store()does. This is the main fix; it also helps the GEMM kernel when alpha is not +-1.gemm_kernel_sme2.c: for left TRMM, the depth steps at the diagonal block skip the outer products of tiles whose rows of A are zero (s2p_mac_diag()). The block functions get the diagonal position as one more argument, unused in the GEMM kernel.trmm_[nt]copy_sme2.c,sme2_gemm_copy.h: a TRMM panel copy (kept part, zero part and diagonal fix) is one streaming-mode call above 512 elements, with predicated stores for the zero side and the unit diagonal, instead of core stores between streaming-mode calls.Only VORTEXM4 files change. Not included: the outer TRMM copies (right side, B panels) are fixed to the generic files in
kernel/Makefile.L3. A streaming-mode version of them gives another 1.15-1.7x on the right side (fp32, n = 64-1024), but needs optional[SD]TRMM??COPY_Nvariables inkernel/Makefile.L3andkernel/CMakeLists.txt, like the existing_Mones; I can propose that separately if it is wanted.Results
M4 Pro, one thread, non-unit, GFLOPS, best of two runs (noise about 3%). Base = develop + #6154. Columns: side (L/R), uplo (U/L), trans (N/T).
Accelerate on the same machine, STRMM: about 430-560 (n = 64), 1290-1400 (n = 256), 1520-1560 (n = 1024); DTRMM about 215-240, 390-405, 390-440. GEMM, SYMM, SYRK and TRSM stay within 2% on one thread and within noise with threads.
Tests
make: BLAS/CBLAS tests, utest 144/144 and 1564/1564;make lapack-test: 0 numerical errors.-DTARGET=VORTEXM4 -DNOFORTRAN=1 -DBUILD_TESTING=ON, ctest 14/14),DYNAMIC_ARCH=1 NO_FORTRAN=1(randomized check, 0 failures) and GCC 16 (TARGET=VORTEXM4; BLAS tests and utest pass).AI assistance
AI coding tools (Claude Code) helped to prepare this PR. The changes were reviewed, tested and benchmarked as described above.