pinot: fix cold-cycle data loss, silent OOM partial load, and stop/start race

[KazukiKandaKK]

pinot/ currently fails all 43 queries (0/43) when run against the real ~74GB hits.tsv on a fresh instance. Found and fixed three separate bugs while tracking this down on a c6a.4xlarge (32 GiB RAM).

1. Cold-cycle data loss (root cause of 0/43). Pinot's benchmark.sh has effectively run with BENCH_DURABLE=yes since the BENCH_RESTARTABLE → BENCH_DURABLE rename in commit b282aa49a (2026-05-10) — the wrong setting for a system whose loaded state lives only in process memory. Commit b422b2d4e (2026-06-29) later deleted the explicit export BENCH_DURABLE=yes line, but that's a red herring: BENCH_DURABLE already defaults to yes in lib/benchmark-common.sh, so deleting the redundant explicit line changed nothing — Pinot's effective value was yes before that commit and stayed yes after it.

Pinot's own -dataDir persistence flag doesn't help either. The Quick Start docs describe it as reloading data across restarts, but I confirmed locally that the second QuickStart start throws IllegalStateException (Pinot 1.5.1: Preconditions.checkState(quickstartRunnerDir.mkdirs()) fails because the directory already exists — reproduced in Docker and confirmed against the Quickstart.java source).

A full CSV reload takes ~78 minutes and can't run before each of the 43 queries, so this PR sets BENCH_DURABLE=no and has load re-push the segment .tar.gz files already on disk from the first load, via Pinot's own Tar Push API docs, instead of re-parsing hits.tsv every cold cycle.

2. Silent partial segment generation under memory pressure. pinot-admin.sh defaults to -Xms4G with no -Xmx, so it falls back to the JVM's default heap ceiling (~25% of physical RAM). On the real 74GB/100-split hits.tsv this OOMed mid-job around segment 60/100, but LaunchDataIngestionJob still exited 0 and the >5GB data-size guard in bench_load didn't catch it either, since 60 segments already cleared 5GB. load now sizes the JVM heap dynamically instead of hard-coding one value, and compares the number of generated segment tars against the number of input splits, failing loudly on any mismatch.

Update (2026-09-21): this originally set a fixed JAVA_OPTS="-Xms4G -Xmx20G", sized for the c6a.4xlarge this PR was verified on. ClickBench's own machine:all CI run on this PR's follow-up (#2126) showed that fixed value breaking every smaller machine — c6a.large (4GiB), c6a.xlarge (8GiB), and t3a.small (2GiB) all failed to even start the JVM, since it can't reserve a 20G/4G heap on a host with less physical RAM than that. start/load now read /proc/meminfo and size the heap at 64% of RAM capped at 20G — 64% is anchored so c6a.4xlarge (32GiB, the only machine this fix has been end-to-end verified on: 37/43 queries, 3/3 cold-cycle trials) gets back exactly the already-proven 20G, while smaller machines scale down instead of over-requesting; larger machines stay capped at 20G rather than growing further, since Pinot keeps segments memory-mapped/off-heap and a larger heap would fight the OS page cache instead of helping. Re-verified end-to-end on a fresh c6a.large (4GiB): QuickStart now starts (-Xms2425m -Xmx2425m), completes bootstrap, answers queries, and stops cleanly — all of which failed outright with the old fixed value.

Note: c7a.metal-48xl's failure in the same CI run is a separate, unaddressed issue — c8g.metal-48xl (same 384GiB class) succeeded with the old fixed 20G heap, so an undersized heap isn't a sufficient explanation there. That needs its own investigation.

3. stop/start race corrupting the next cold cycle. stop only did pkill plus a fixed sleep 2, but the broker can keep answering for 7-8s after SIGTERM while the JVM shutdown hook is still running, and the embedded ZooKeeper's port teardown lags even further behind that. The next start then either treats the still-alive old broker as "already up" (skipping the real restart) or fails to bind ZooKeeper's port. stop now polls until the pinot-admin JVM process itself is fully gone. check similarly no longer treats broker-responsive as "fully started" — QuickStart only finishes bootstrapping (and registers its clean-shutdown hook) after printing "Quick start setup complete" to pinot.log, so check now greps for that line too, scoped to the current invocation via a recorded log offset.

Verified end-to-end on EC2 (c6a.4xlarge, real 74GB hits.tsv, 100 segments): 37/43 queries pass across all 3 cold-cycle trials, up from 0/43. The remaining 6 failures are a separate, pre-existing Pinot limitation (standard MIN()/MAX() rejects STRING columns) unrelated to this fix, reported upstream at apache/pinot#19603. A follow-up PR here addresses those once the query-side workaround is ready.