[io] Compress buffers larger than 16 MiB in blocks of equal size

[rshrj]

This Pull request:

Changes or fixes:

Buffers larger than kMAXZIPBUF (2^24 − 1 bytes) are compressed in several blocks. So far all blocks but the last were kMAXZIPBUF bytes long, and if any block did not shrink the whole buffer was stored uncompressed. For a buffer just above a multiple of kMAXZIPBUF the last block is only a few bytes, which can never be compressed, so such buffers were always stored uncompressed (#23706): RNTuple pages of 2^24 bytes, every full page with a 32 MiB page size limit, and TTree baskets, TKey objects and TMessages of that size.

This PR cuts the buffer into the same number of blocks as before, but all of (nearly) the same size: ROOT::Internal::GetZipBlockSize(nbytes) in RZip.h, used by the five writers (RNTupleCompressor::Zip, TBasket::WriteBuffer, both TKey constructors, TMessage::Compress). No block is larger than kMAXZIPBUF, and no block is tiny.

The format does not change: all readers (TBasket::ReadBasketBuffers, TKey::UnzipBuffer, TTreeCacheUnzip, TMessage::Uncompress, RNTupleDecompressor::Unzip) take each block's size from its header. Older ROOT versions read the new files. Buffers up to kMAXZIPBUF bytes are compressed exactly as before; larger ones now compress to different, valid bytes.

Tests: RNTupleZip.LargeWithOutputBuffer asserted the old behaviour ("Trailing byte cannot be compressed, entire buffer returns uncompressed") and now checks that a kMAXZIPBUF + 1 buffer is compressed and read back. New: RNTupleZip.LargeJustAboveBlocks (sizes kMAXZIPBUF + 1, + 2 and 2^25, with ZLIB, LZMA, LZ4 and ZSTD) and, in io/io/test/ZipHeader.cxx, RZip.JustAboveBlockSize{TKey,TTree,Message}: an object, a basket and a message of exactly kMAXZIPBUF + 1 bytes must be compressed and read back.

Effect, with the reproducer of #23706 (ZSTD 5): RNTuple pages of 2^24, 2^24 + 1, + 19 and + 39 bytes take 566–574 bytes on storage instead of 16.8 MB; a TTree basket of 2^24 bytes takes 631 bytes instead of 16 777 281. On real data (CMS HGCAL hits written as RNTuple with 32 000 000-byte pages, 203 pages, 45 of them larger than one block), recompressing every page gives 312.9 instead of 346.5 MB with ZSTD 3, 308.3 instead of 341.8 MB with ZSTD 5 and 282.1 instead of 315.8 MB with LZMA 7; the difference is the pages that were stored uncompressed, the other pages hardly change.

On synthetic buffers (zeros, a compressible pattern and random data, 4 algorithms, sizes from kMAXZIPBUF − 1 to 10^8 bytes, 192 cases), 75 buffers that were stored uncompressed are now compressed, 94 are unchanged (all buffers up to kMAXZIPBUF and all random data), and 23 change by a few bytes or per cent in either direction. The largest increase is +5.5 % with LZMA, whose dictionary is capped at a quarter of each (now smaller) block.

Checklist:

• tested changes locally
• updated the docs (if necessary): not needed, no interface change

Tested on x86_64 (EL9, gcc 14.3) with two builds of 14a58f7 (master of 2026-10-12), with and without this change, -Dminimal=ON -Dtesting=ON: ctest -R '^gtest-(io|tree|net)' passes (100 of 100) in both; the new tests fail without the change.

This PR fixes #23706

[github-actions]

Test Results

    21 files      21 suites   3d 15h 12m 12s ⏱️
 3 888 tests  3 886 ✅ 0 💤 2 ❌
72 123 runs  72 121 ✅ 0 💤 2 ❌

For more details on these failures, see this check.

Results for commit 62cb787.

[pcanal]

What would happen if GetZipBlockSize::nblocks and nbuffers get out of sync? Should we either assert that (fObjlen + blockSize - 1) / blockSize == nbuffers or (probably better) pass nbuffers as a parameter? (or even return both nbuffers and blockSize for the function if we want to delegate the entire split to that function).