diff --git a/README.md b/README.md index 7416583..93970a3 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ npm install -g @openmixer/plugin-qualify The npm package ships the TypeScript command only, not the native tools. The CLAP prober, the LV2 measurer and the interposer come with the RPM or DEB, or from `make -C native` in a checkout; with the npm package alone, -point `OMX_CLAP_QUALIFY`, `OMX_LV2_MEASURE` and `OMX_RT_INTERPOSE` at them, or CLAP qualification and the LV2 +point `OMX_CLAP_QUALIFY`, `OMX_LV2_MEASURE` and `OMX_RT_INTERPOSE` at them (the probe runner is plugin-hostd's, `OMX_PROBE_RUNNER`), or CLAP qualification and the LV2 hosting dimensions refuse with "no CLAP prober at ...". The GitHub release also carries the packed tarball, `openmixer-plugin-qualify-.tgz`. To run from a @@ -68,7 +68,9 @@ plugin-qualify --help # every option and its default `--no-host-sweep` measures offline only, without mod-host. The `Containerfile` builds an image with lilv, mod-host and the ASan mod-host, if you do not want to install them yourself. -Exit code: `0` every plugin qualified, `1` some did not, `2` something could not be measured. +A named LV2 plugin is measured the same way as a CLAP one: through the probe runner, then judged from the probe +output it writes (`plugin-qualify judge` below). Exit code: `0` every plugin qualified, `1` some did not, `2` +something could not be measured. ## CLAP plugins: the RT-safety verdict @@ -82,6 +84,12 @@ plugin-qualify --out ./out ./my-fx.clap # and its RT-safety documents CLAP_PATH=~/my-clap plugin-qualify --all # every CLAP on the search path ``` +Each plugin is measured by plugin-hostd's probe runner, `plugin-hostd-probe` (from the `plugin-hostd` package, 0.4.0 or later, which +the RPM and DEB require; +`OMX_PROBE_RUNNER` names it when it is not on `PATH`). The runner starts the prober under limits (a wall clock +that kills the whole process tree, CPU time, memory, files), and plugin-qualify waits for the runner and reads +the document it writes, the probe output. Nothing in plugin-qualify times a prober out on its own any more. + `--all` walks the standard CLAP search path through the same scanner: each directory `CLAP_PATH` names (colon-separated), then `~/.clap`, then `/usr/lib/clap`, which is the system directory on every distribution and architecture. A plugin id found twice keeps its first place on that path. LV2 plugins @@ -100,15 +108,43 @@ count of its own: a chord, the pitch bent and a controller moved, pressure, the burst of 32 note ons in one block and their offs, all notes off, in the dialect its note input reads (CLAP's, else MIDI's). An allocation, lock or syscall while it plays is **unsuitable** (`hosting.rt-safety.violations-under-notes`); an instrument that could not be played under the -interposer is **unknown** (`hosting.rt-safety.notes-unmeasured`). `--out` writes one -document per plugin binary and host profile, named `..rt-safety.json`, so a -verdict never outlives the bytes it was measured on. Its schema is `plugin-qualify.rt-safety/2`: `/1` with -`measured.noteSweep` added and the note sweep's reasons in `verdict`; a plugin with no note input -reads the same verdict under both. Each line also carries the plugin's pin, an -`integrity` block: the `.clap`'s SHA-256 by its file name and the parameter-layout fingerprint -`omx-layout/1:`, both computed through plugin-hostd's `pin.h`, which is what a console +interposer is **unknown** (`hosting.rt-safety.notes-unmeasured`). `--out` writes the qualification records and the probe outputs they judged (see +"Judging a probe output" below), one pair per plugin binary and offered host extensions. Each plugin's pin is +an `integrity` block in the probe output: the `.clap`'s SHA-256 by its file name and the parameter-layout +fingerprint `omx-layout/1:`, both computed through plugin-hostd's `pin.h`, which is what a console hands plugin-hostd as `pin_set` before it adds the plugin. `OMX_CLAP_QUALIFY` and `OMX_RT_INTERPOSE` -name the prober and the interposer when they are not the installed ones or the checkout's own build. +name the prober and the interposer the runner starts when they are not the installed ones or the checkout's +own build. + +## Judging a probe output + +The probe output is the document `plugin-hostd-probe run` writes: one binary and one offer, the steps the +prober reached and how each ended, and the figures per rate and quantum, with no verdict in it. plugin-qualify +reads it with `readProbeOutput`, which refuses a document of a major version it does not know +(`probe.format.major-unknown`), a document that carries a verdict, and a document whose derived fields disagree +with the fields they are computed from (`probe.format.derived-mismatch`): `machine.class.cstatesLimited` and +`machine.profile.meetsProtocol` are recomputed from `cpuidle[]`, the governor and `unmet[]`, never trusted. + +``` +plugin-hostd-probe run --out out/plate.probe.json ./example-plate.clap org.example.plate +plugin-qualify judge --host-profile omx-clap-host --qualification-dir ./qualification out/plate.probe.json +``` + +`judge` writes, into the qualification directory (`--qualification-dir`, else `$XDG_STATE_HOME/omx/qualification`, +else `~/.local/state/omx/qualification`), the qualification record `@-..qualification.json` +and, beside it, a byte copy of the probe output it judged, `.probe.json`, because the probe directory +may be wiped. The record names that copy and its SHA-256 in `probe`, with the run's protocol, reason and machine +class, so a reference figure and a local one are told apart. The record holds, for each host profile, the verdict +(`rating`, `hosting`, the deciding reason code and every reason) and the cost reasons at that profile's rate and +quantum, and the figures as measured. Exit `0` every plugin qualified, `1` some did not, `2` something could not +be read or judged. + +The verdict is the same one `plugin-qualify` gives CLAP plugins today, the real-time-safety one, now for LV2 as +well; cost is reported beside it, not folded into it. Cost is read at the profile's rate AND quantum: a pair the +document does not hold reads `hosting.cost.unmeasured-at-rate-and-quantum`, and two quanta of one plugin that cost +differently give two different cost reasons. Only a measurement of the old format, which keeps one figure per +rate, is read per rate. A run that did not end `probe.ok` is `unknown`, with the run's outcome code as the reason. + The prober also scans: `OMX_CLAP_SCAN=native/build/clap-qualify` makes it the scanner, and its scan keeps what a console needs to draw a third-party plugin's controls: an enum parameter's labels (as diff --git a/bin/plugin-qualify.mjs b/bin/plugin-qualify.mjs index 4add868..0a32df7 100755 --- a/bin/plugin-qualify.mjs +++ b/bin/plugin-qualify.mjs @@ -5,32 +5,36 @@ * `plugin-qualify` — the entry point. Every decision is a * pure function in `src/qualify-cli.ts`; this file only reads argv, prints and exits. * - * CLAP targets are qualified here: the prober (`native/clap-qualify`) runs each under the interposer - * and every plugin gets its RT-safety verdict document, per binary hash and console. `--all` finds - * them on the standard CLAP search path (`CLAP_PATH`, then `~/.clap`, then `/usr/lib/clap`). - * - * NOT YET WIRED for LV2 (refused with exit 2, never a silent partial run): driving the offline pass - * (`tools/lv2-measure.mjs`) for a named subset and the mod-host hosting sweep from this entry - * point. Until then a run that would measure an LV2 refuses and says which step is missing. + * Every named plugin, CLAP or LV2, is measured by plugin-hostd's probe runner (`plugin-hostd-probe`, + * named by `OMX_PROBE_RUNNER` when it is not on `PATH`), which keeps the deadline and the limits, and is + * judged from the probe output it writes: the verdict and the qualification record, per host profile. + * `--all` finds the CLAP plugins on the standard CLAP search path (`CLAP_PATH`, then `~/.clap`, then + * `/usr/lib/clap`). `plugin-qualify judge` judges probe outputs already written. */ -import { closeSync, fsyncSync, mkdirSync, openSync, renameSync, writeFileSync, writeSync } from "node:fs"; -import { dirname, join } from "node:path"; +import { closeSync, fsyncSync, mkdirSync, openSync, readFileSync, renameSync, writeSync } from "node:fs"; +import { dirname } from "node:path"; import { ClapScanError, EXIT_CODE, clapLoader, clapPinEntries, + defaultQualificationDir, + judgeExitCode, + judgeFiles, + judgeHelpText, + parseJudgeArgs, discoverAllTargets, exitCodeFor, helpText, lv2Loader, parseQualifyArgs, pinsFile, - qualifyClapTargets, + qualificationRecordOf, + qualifyTargets, resolveHostProfiles, resolveTargets, - rtSafetyDocumentName, + writeQualification, textTable, } from "../dist/index.js"; @@ -58,6 +62,33 @@ function writePins(path, file) { process.stderr.write(`plugin-qualify: ${file.written.length} pin${file.written.length === 1 ? "" : "s"} written to ${path}\n`); } +if (process.argv[2] === "judge") { + const j = parseJudgeArgs(process.argv.slice(3)); + if (!j.ok) { + process.stderr.write(`plugin-qualify judge: ${j.error}\n\n${judgeHelpText()}`); + process.exit(EXIT_CODE.couldNotMeasure); + } + if (j.args.help) { + process.stdout.write(judgeHelpText()); + process.exit(EXIT_CODE.allQualified); + } + const hosts = resolveHostProfiles(j.args.hostProfiles); + if (!hosts.ok) { + for (const r of hosts.refused) process.stderr.write(`plugin-qualify judge: --host-profile ${r.spec}: ${r.problem}\n`); + process.exit(EXIT_CODE.couldNotMeasure); + } + const dir = j.args.qualificationDir ?? defaultQualificationDir(process.env); + const result = judgeFiles(j.args.files, { profiles: hosts.profiles, dir }); + for (const f of result.failures) process.stderr.write(`plugin-qualify judge: ${f.file}: ${f.problem}\n`); + for (const r of result.records) process.stderr.write(`plugin-qualify judge: record ${r.recordPath}\n`); + process.stdout.write( + j.args.format === "json" + ? `${JSON.stringify(result.records.map((r) => JSON.parse(readFileSync(r.recordPath, "utf8"))), null, 2)}\n` + : textTable(result.verdicts), + ); + process.exit(judgeExitCode(result)); +} + const parsed = parseQualifyArgs(process.argv.slice(2)); if (!parsed.ok) { process.stderr.write(`plugin-qualify: ${parsed.error}\n\n${helpText()}`); @@ -90,23 +121,25 @@ try { process.exit(EXIT_CODE.couldNotMeasure); } for (const p of plugins) process.stderr.write(`plugin-qualify: target ${p.uri}${p.bundle ? ` (${p.bundle})` : ""}\n`); -if (plugins.length > 0 && plugins.every((p) => p.format === "clap")) { - if (parsed.args.all) { - process.stderr.write("plugin-qualify: --all qualified the CLAP plugins on the search path (no LV2 bundle is on the LV2 path)\n"); - } - const pass = qualifyClapTargets(plugins, { rates: parsed.args.rates, profiles: profiles.profiles }); - for (const f of pass.failures) process.stderr.write(`plugin-qualify: ${f.uri}: ${f.problem}\n`); - if (parsed.args.out !== undefined) { - mkdirSync(parsed.args.out, { recursive: true }); - for (const doc of pass.documents) writeFileSync(join(parsed.args.out, rtSafetyDocumentName(doc)), `${JSON.stringify(doc, null, 2)}\n`); +if (plugins.length === 0) { + process.stderr.write("plugin-qualify: no plugin to measure\n"); + process.exit(EXIT_CODE.couldNotMeasure); +} +if (parsed.args.all) { + process.stderr.write("plugin-qualify: --all qualified the CLAP plugins on the search path (no LV2 bundle is on the LV2 path)\n"); +} +const pass = qualifyTargets(plugins, { rates: parsed.args.rates, profiles: profiles.profiles }); +for (const f of pass.failures) process.stderr.write(`plugin-qualify: ${f.uri}: ${f.problem}\n`); +if (parsed.args.out !== undefined) { + for (const run of pass.runs) { + const w = writeQualification(parsed.args.out, run.bytes, run.judgment); + process.stderr.write(`plugin-qualify: record ${w.recordPath}\n`); } - if (parsed.args.pins !== undefined) writePins(parsed.args.pins, pinsFile(clapPinEntries(pass.documents, profiles.profiles[0].id))); - process.stdout.write(parsed.args.format === "json" ? `${JSON.stringify(pass.documents, null, 2)}\n` : textTable(pass.verdicts)); - process.exit(exitCodeFor(pass.verdicts)); } -if (parsed.args.pins !== undefined) process.stderr.write(`plugin-qualify: --pins ${parsed.args.pins}: nothing to pin, this run measures no plugin yet\n`); -process.stderr.write( - "plugin-qualify: measuring from this entry point is not wired yet " + - "(offline pass: tools/lv2-measure.mjs; hosting sweep: mod-host) — refusing rather than rating on less\n", +if (parsed.args.pins !== undefined) writePins(parsed.args.pins, pinsFile(clapPinEntries(pass.documents, profiles.profiles[0].id))); +process.stdout.write( + parsed.args.format === "json" + ? `${JSON.stringify(pass.runs.map((r) => qualificationRecordOf(r.bytes, r.judgment)), null, 2)}\n` + : textTable(pass.verdicts), ); -process.exit(EXIT_CODE.couldNotMeasure); +process.exit(exitCodeFor(pass.verdicts)); diff --git a/debian/control b/debian/control index 7cbb7e0..a641cae 100644 --- a/debian/control +++ b/debian/control @@ -19,7 +19,7 @@ Rules-Requires-Root: no Package: plugin-qualify Architecture: any -Depends: ${misc:Depends}, ${shlibs:Depends}, nodejs (>= 18), python3 +Depends: ${misc:Depends}, ${shlibs:Depends}, nodejs (>= 18), python3, plugin-hostd (>= 0.4.0~) Recommends: python3-lilv, mod-host, omx-clap-host Description: find out whether an audio plugin is safe to run live A plugin that glitches on stage is worse than one you never installed. diff --git a/fixtures/probe-output/memory-limit-refused.json b/fixtures/probe-output/memory-limit-refused.json new file mode 100644 index 0000000..12238b6 --- /dev/null +++ b/fixtures/probe-output/memory-limit-refused.json @@ -0,0 +1,190 @@ +{ + "format": "plugin-hostd.probe-output", + "formatVersion": "1.0", + "run": { + "id": "r1-20261017T232540Z-652b62e", + "startedAt": "2026-10-17T23:25:40+02:00", + "elapsedMs": 2310, + "complete": false, + "outcome": "probe.memory-limit", + "reason": "add", + "protocol": { + "name": "plugin-hostd.probe-protocol", + "version": 1 + }, + "tools": { + "runner": "plugin-hostd-probe 0.1.0", + "child": "plugin-hostd-probe-clap 0.1.0", + "clapCore": "0.9.0", + "interposer": "rt-interpose 1" + }, + "via": "daemon" + }, + "plugin": { + "format": "clap", + "id": "org.example.big-sampler", + "name": "Big Sampler", + "vendor": "Example", + "version": "3.0.1" + }, + "binary": { + "path": "/home/op/.clap/big-sampler.clap", + "sha256": "652b62e0660ed65b624e9ef595a10ed43bd19021ed5a455d6f948f1308f1c849", + "files": [ + { + "path": "big-sampler.clap", + "sha256": "652b62e0660ed65b624e9ef595a10ed43bd19021ed5a455d6f948f1308f1c849" + } + ], + "layout": { + "scheme": "omx-layout/1", + "sha256": "37da45a15b9e009c001c4110c32fc534cfd1e7e4b5dd81d8de1ac6c53d8af4b0" + } + }, + "machine": { + "class": { + "arch": "x86_64", + "family": "amd", + "coreBand": "9-16", + "clockBand": "3.5-4.5GHz", + "preemptRt": false, + "cstatesLimited": true + }, + "profile": { + "governor": "performance", + "turbo": true, + "smt": true, + "preempt": "PREEMPT_DYNAMIC", + "cpuidle": [ + { + "cpu": 11, + "state": "POLL", + "latencyUs": 0, + "disabled": false, + "usageDelta": 0 + }, + { + "cpu": 11, + "state": "C1_ACPI", + "latencyUs": 1, + "disabled": false, + "usageDelta": 9 + }, + { + "cpu": 11, + "state": "C2_ACPI", + "latencyUs": 127, + "disabled": true, + "usageDelta": 0 + } + ], + "meetsProtocol": true + }, + "pinnedCpus": "11" + }, + "offer": { + "id": "omx-clap-host", + "clapExtensions": [ + "clap.audio-ports", + "clap.latency", + "clap.log", + "clap.params", + "clap.state", + "clap.thread-check", + "clap.preset-load/2", + "clap.track-info/1", + "clap.remote-controls/2" + ], + "realtime": "none" + }, + "limits": { + "set": "default", + "applied": [ + { + "limit": "wall-clock", + "value": 600, + "unit": "s", + "by": "supervisor" + }, + { + "limit": "cpu-time", + "value": 540, + "unit": "s", + "by": "rlimit" + }, + { + "limit": "memory", + "value": 2147483648, + "unit": "bytes", + "by": "cgroup" + }, + { + "limit": "tasks", + "value": 64, + "unit": "count", + "by": "cgroup" + }, + { + "limit": "core-dump", + "value": 0, + "unit": "bytes", + "by": "rlimit" + }, + { + "limit": "cpus", + "value": "11", + "by": "affinity" + }, + { + "limit": "no-new-privs", + "by": "prctl" + }, + { + "limit": "network", + "value": "none", + "by": "netns" + } + ], + "unavailable": [ + { + "limit": "filesystem", + "code": "probe.limit.landlock-absent" + } + ] + }, + "steps": [ + { + "step": "hash", + "outcome": "probe.ok", + "elapsedMs": 9 + }, + { + "step": "instantiate", + "outcome": "probe.memory-limit", + "elapsedMs": 2280, + "signal": "SIGKILL", + "limit": "memory", + "detail": "memory.events oom_kill 0 -> 1 while init() ran" + } + ], + "behaviour": { + "memory": { + "rssStartKb": 8800, + "peakBytes": 2147479552 + } + }, + "logs": [ + { + "name": "stderr", + "path": "probes/org.example.big-sampler@3.0.1-652b62e0660e.stderr.log", + "sha256": "f25743e01a8d2faa127a86adc728bf034cb23a158ec792ee9a43c5c2cfad4aab", + "bytes": 96 + }, + { + "name": "memory.events", + "path": "probes/org.example.big-sampler@3.0.1-652b62e0660e.memory.events", + "sha256": "fb7a4ca6b19e56a9d72a8e3a411fac55c0adc0963466e6c29f0b89319eb4795c", + "bytes": 84 + } + ] +} diff --git a/fixtures/probe-output/plugin-crash.json b/fixtures/probe-output/plugin-crash.json new file mode 100644 index 0000000..add7d52 --- /dev/null +++ b/fixtures/probe-output/plugin-crash.json @@ -0,0 +1,236 @@ +{ + "format": "plugin-hostd.probe-output", + "formatVersion": "1.0", + "run": { + "id": "r1-20261017T231004Z-0b858f1", + "startedAt": "2026-10-17T23:10:04+02:00", + "elapsedMs": 6120, + "complete": false, + "outcome": "probe.crashed", + "reason": "add", + "protocol": { + "name": "plugin-hostd.probe-protocol", + "version": 1 + }, + "tools": { + "runner": "plugin-hostd-probe 0.1.0", + "child": "plugin-hostd-probe-lv2 0.1.0", + "clapCore": "0.9.0", + "interposer": "rt-interpose 1" + }, + "via": "daemon" + }, + "plugin": { + "format": "lv2", + "id": "urn:example:zero-path", + "name": "zero-path", + "version": "0.0" + }, + "binary": { + "path": "/home/op/.lv2/zero-path.lv2", + "sha256": "0b858f1f53d802872c1b34d298b0ae4c81f5c1b51f5488f72a673fca54afeeb5", + "files": [ + { + "path": "manifest.ttl", + "sha256": "13038d7cea17afef4cfe3833884c73a0db990a60d35b49cf2589f8f139f00cbc" + }, + { + "path": "zero-path.so", + "sha256": "0b858f1f53d802872c1b34d298b0ae4c81f5c1b51f5488f72a673fca54afeeb5" + }, + { + "path": "zero-path.ttl", + "sha256": "a364f60c8ef436a160fc139c1575435d5653f70a4267bd08feb4f36782b61535" + } + ], + "layout": { + "scheme": "omx-layout/1", + "sha256": "8a66be4f19b6e12130e4a9f314b11a8485a5a9c4b7ef3d3b4117a176cde89f0e" + } + }, + "machine": { + "class": { + "arch": "x86_64", + "family": "amd", + "coreBand": "9-16", + "clockBand": "3.5-4.5GHz", + "preemptRt": false, + "cstatesLimited": false + }, + "profile": { + "governor": "powersave", + "turbo": true, + "smt": true, + "preempt": "PREEMPT_DYNAMIC", + "cpuidle": [ + { + "cpu": 11, + "state": "POLL", + "latencyUs": 0, + "disabled": false, + "usageDelta": 3 + }, + { + "cpu": 11, + "state": "C1_ACPI", + "latencyUs": 1, + "disabled": false, + "usageDelta": 812 + }, + { + "cpu": 11, + "state": "C2_ACPI", + "latencyUs": 127, + "disabled": false, + "usageDelta": 95 + } + ], + "meetsProtocol": false, + "unmet": [ + "probe.profile.governor", + "probe.profile.deep-idle-entered" + ] + }, + "pinnedCpus": "11", + "local": { + "hostname": "studio", + "kernel": "7.2.9-201.fc44.x86_64" + } + }, + "offer": { + "id": "omx-console-lv2", + "lv2Features": [ + "http://lv2plug.in/ns/ext/urid#map", + "http://lv2plug.in/ns/ext/urid#unmap", + "http://lv2plug.in/ns/ext/options#options", + "http://lv2plug.in/ns/ext/buf-size#boundedBlockLength", + "http://lv2plug.in/ns/ext/worker#schedule", + "http://lv2plug.in/ns/ext/log#log" + ], + "realtime": "none" + }, + "limits": { + "set": "default", + "applied": [ + { + "limit": "wall-clock", + "value": 600, + "unit": "s", + "by": "supervisor" + }, + { + "limit": "cpu-time", + "value": 540, + "unit": "s", + "by": "rlimit" + }, + { + "limit": "memory", + "value": 2147483648, + "unit": "bytes", + "by": "cgroup" + }, + { + "limit": "tasks", + "value": 64, + "unit": "count", + "by": "cgroup" + }, + { + "limit": "core-dump", + "value": 0, + "unit": "bytes", + "by": "rlimit" + }, + { + "limit": "cpus", + "value": "11", + "by": "affinity" + }, + { + "limit": "no-new-privs", + "by": "prctl" + }, + { + "limit": "network", + "value": "none", + "by": "netns" + }, + { + "limit": "filesystem", + "value": "read-only except probe_dir", + "by": "landlock" + } + ], + "unavailable": [] + }, + "steps": [ + { + "step": "hash", + "outcome": "probe.ok", + "elapsedMs": 2 + }, + { + "step": "instantiate", + "outcome": "probe.ok", + "elapsedMs": 12 + }, + { + "step": "lifecycle", + "outcome": "probe.ok", + "elapsedMs": 940 + }, + { + "step": "latency", + "rate": 48000, + "quantum": 128, + "outcome": "probe.ok", + "elapsedMs": 88 + }, + { + "step": "rt-safety", + "rate": 48000, + "quantum": 128, + "outcome": "probe.crashed", + "elapsedMs": 41, + "signal": "SIGSEGV", + "detail": "died during the control sweep, control 'path'" + } + ], + "grid": [ + { + "rate": 48000, + "quantum": 128, + "outcome": "probe.crashed", + "latency": { + "declaredFrames": 0, + "measuredFrames": 0, + "method": "impulse-onset" + } + } + ], + "behaviour": { + "lifecycle": { + "cycles": 30, + "instantiated": 30, + "failed": 0 + }, + "threads": { + "before": 2, + "afterFirstInstantiate": 2, + "afterAllFreed": 2 + } + }, + "crash": { + "signal": "SIGSEGV", + "frameObject": "zero-path.so" + }, + "logs": [ + { + "name": "stderr", + "path": "probes/urn_example_zero-path@0.0-0b858f1f53d8.stderr.log", + "sha256": "73aaa5071d9f861550d51469cbd89e0a6e5371486fd94c9675281d1cefbd5117", + "bytes": 1840 + } + ] +} diff --git a/fixtures/probe-output/success-msi.json b/fixtures/probe-output/success-msi.json new file mode 100644 index 0000000..0bf226e --- /dev/null +++ b/fixtures/probe-output/success-msi.json @@ -0,0 +1,363 @@ +{ + "format": "plugin-hostd.probe-output", + "formatVersion": "1.0", + "run": { + "id": "msi-20261017T221500Z-3569027", + "startedAt": "2026-10-17T22:15:00+02:00", + "elapsedMs": 184220, + "complete": true, + "outcome": "probe.ok", + "reason": "reference", + "protocol": { + "name": "plugin-hostd.probe-protocol", + "version": 1 + }, + "tools": { + "runner": "plugin-hostd-probe 0.1.0", + "child": "plugin-hostd-probe-clap 0.1.0", + "clapCore": "0.9.0", + "interposer": "rt-interpose 1" + }, + "via": "runner" + }, + "plugin": { + "format": "clap", + "id": "org.example.plate", + "name": "Plate", + "vendor": "Example", + "version": "1.4.2", + "ownerPackage": "example-plate-1.4.2-1.fc44.x86_64" + }, + "binary": { + "path": "/usr/lib64/clap/example-plate.clap", + "sha256": "356902794a4eec2122989e072271ef995968131145db918d9a3084ce582f244b", + "files": [ + { + "path": "example-plate.clap", + "sha256": "356902794a4eec2122989e072271ef995968131145db918d9a3084ce582f244b" + } + ], + "layout": { + "scheme": "omx-layout/1", + "sha256": "0a3e01399262321514e05af0575ebb0b207a35567ac262a060fde655ee069805" + } + }, + "descriptor": { + "path": "descriptors/2/clap/org.example.plate@1.4.2-356902794a4e.json", + "sha256": "cc312bdd5a9aa62796ebcb5a648c354c5ee45a6a99892425495ccb58a31bc44e", + "bytes": 9412 + }, + "machine": { + "class": { + "arch": "x86_64", + "family": "intel", + "coreBand": "17-32", + "clockBand": "4.5GHz+", + "preemptRt": false, + "cstatesLimited": true + }, + "profile": { + "governor": "performance", + "turbo": true, + "smt": true, + "preempt": "PREEMPT_DYNAMIC", + "cpuidle": [ + { + "cpu": 22, + "state": "POLL", + "latencyUs": 0, + "disabled": false, + "usageDelta": 118 + }, + { + "cpu": 22, + "state": "C1_ACPI", + "latencyUs": 1, + "disabled": false, + "usageDelta": 40210 + }, + { + "cpu": 22, + "state": "C2_ACPI", + "latencyUs": 127, + "disabled": true, + "usageDelta": 0 + }, + { + "cpu": 22, + "state": "C3_ACPI", + "latencyUs": 1048, + "disabled": true, + "usageDelta": 0 + } + ], + "meetsProtocol": true, + "declaredName": "omx-live-rig" + }, + "pinnedCpus": "22", + "reference": { + "rig": "msi", + "image": "omx-ref-2026-10" + } + }, + "offer": { + "id": "omx-clap-host", + "clapExtensions": [ + "clap.audio-ports", + "clap.latency", + "clap.log", + "clap.params", + "clap.state", + "clap.thread-check", + "clap.preset-load/2", + "clap.track-info/1", + "clap.remote-controls/2" + ], + "realtime": "none" + }, + "limits": { + "set": "default", + "applied": [ + { + "limit": "wall-clock", + "value": 600, + "unit": "s", + "by": "supervisor" + }, + { + "limit": "cpu-time", + "value": 540, + "unit": "s", + "by": "rlimit" + }, + { + "limit": "memory", + "value": 2147483648, + "unit": "bytes", + "by": "cgroup" + }, + { + "limit": "tasks", + "value": 64, + "unit": "count", + "by": "cgroup" + }, + { + "limit": "files", + "value": 256, + "unit": "count", + "by": "rlimit" + }, + { + "limit": "file-size", + "value": 67108864, + "unit": "bytes", + "by": "rlimit" + }, + { + "limit": "core-dump", + "value": 0, + "unit": "bytes", + "by": "rlimit" + }, + { + "limit": "rt-priority", + "value": 0, + "by": "rlimit" + }, + { + "limit": "cpus", + "value": "22", + "by": "affinity" + }, + { + "limit": "no-new-privs", + "by": "prctl" + }, + { + "limit": "network", + "value": "none", + "by": "netns" + } + ], + "unavailable": [] + }, + "steps": [ + { + "step": "hash", + "outcome": "probe.ok", + "elapsedMs": 4 + }, + { + "step": "instantiate", + "outcome": "probe.ok", + "elapsedMs": 31 + }, + { + "step": "lifecycle", + "outcome": "probe.ok", + "elapsedMs": 2210 + }, + { + "step": "latency", + "rate": 48000, + "quantum": 128, + "outcome": "probe.ok", + "elapsedMs": 310 + }, + { + "step": "cost", + "rate": 48000, + "quantum": 128, + "outcome": "probe.ok", + "elapsedMs": 30020 + }, + { + "step": "rt-safety", + "rate": 48000, + "quantum": 128, + "outcome": "probe.ok", + "elapsedMs": 4100 + }, + { + "step": "latency", + "rate": 96000, + "quantum": 128, + "outcome": "probe.ok", + "elapsedMs": 305 + }, + { + "step": "cost", + "rate": 96000, + "quantum": 128, + "outcome": "probe.ok", + "elapsedMs": 30011 + }, + { + "step": "rt-safety", + "rate": 96000, + "quantum": 128, + "outcome": "probe.ok", + "elapsedMs": 4080 + }, + { + "step": "note-sweep", + "outcome": "probe.not-attempted", + "elapsedMs": 0, + "detail": "no note input" + } + ], + "grid": [ + { + "rate": 48000, + "quantum": 128, + "outcome": "probe.ok", + "latency": { + "declaredFrames": 0, + "measuredFrames": 0, + "method": "impulse-onset" + }, + "cost": { + "unit": "ns-per-sample", + "median": 41.2, + "percentile": { + "p": 95, + "value": 47.9 + }, + "min": 39.8, + "max": 112.4, + "iqr": 3.1, + "blocks": 11250, + "warmupBlocks": 64, + "warmupMedian": 58.0, + "timingFloorNsPerBlock": 180 + }, + "rtSafety": { + "interposer": true, + "swept": true, + "sweepApplicable": true, + "blocks": 512, + "allocations": 0, + "locks": 0, + "syscalls": 0, + "threadViolations": 0, + "processErrors": 0, + "events": [] + } + }, + { + "rate": 96000, + "quantum": 128, + "outcome": "probe.ok", + "latency": { + "declaredFrames": 0, + "measuredFrames": 0, + "method": "impulse-onset" + }, + "cost": { + "unit": "ns-per-sample", + "median": 40.7, + "percentile": { + "p": 95, + "value": 46.3 + }, + "min": 39.1, + "max": 98.0, + "iqr": 2.8, + "blocks": 22500, + "warmupBlocks": 64, + "timingFloorNsPerBlock": 180, + "samples": { + "name": "cost-96000-128", + "path": "probes/org.example.plate@1.4.2-356902794a4e.cost-96000-128.bin", + "sha256": "5f649130dafd22f1edc70441f9455415d0856f0f7d26cf7274463b82ac2c5813", + "bytes": 180000 + } + }, + "rtSafety": { + "interposer": true, + "swept": true, + "sweepApplicable": true, + "blocks": 512, + "allocations": 0, + "locks": 0, + "syscalls": 0, + "threadViolations": 0, + "processErrors": 0, + "events": [] + } + } + ], + "behaviour": { + "lifecycle": { + "cycles": 30, + "instantiated": 30, + "failed": 0 + }, + "threads": { + "before": 2, + "afterFirstInstantiate": 2, + "afterAllFreed": 2 + }, + "memory": { + "rssStartKb": 9120, + "rssMaxKb": 14988 + }, + "features": { + "extensionsAsked": [ + "clap.audio-ports", + "clap.params", + "clap.state", + "clap.latency" + ] + } + }, + "logs": [ + { + "name": "stderr", + "path": "probes/org.example.plate@1.4.2-356902794a4e.stderr.log", + "sha256": "c46928bc87de23b9a8bd235aa1fe67ccb1542c8389ad3f02bf4ab5cff506e98f", + "bytes": 212 + } + ] +} diff --git a/package.json b/package.json index fbdd044..b9d04e2 100644 --- a/package.json +++ b/package.json @@ -67,6 +67,7 @@ "dist", "src", "!src/**/*.test.ts", + "!src/**/*.support.*", "tools", "!tools/test_*.py", "!**/__pycache__", diff --git a/packaging/plugin-qualify.spec b/packaging/plugin-qualify.spec index 1bcd736..ab116e7 100644 --- a/packaging/plugin-qualify.spec +++ b/packaging/plugin-qualify.spec @@ -28,6 +28,8 @@ BuildRequires: pkgconfig(omxdsp) >= 0.18.0 Requires: nodejs >= 18 Requires: python3 Requires: python3-lilv +# plugin-hostd-probe, the runner every CLAP plugin is measured through; first released in 0.4.0 +Requires: plugin-hostd >= 0.4.0 Recommends: mod-host # omx-clap-scan, which lists the plugins in a .clap before each is qualified Recommends: omx-clap-host diff --git a/src/clap-qualify.test.ts b/src/clap-qualify.test.ts index 34bbefb..1f82c06 100644 --- a/src/clap-qualify.test.ts +++ b/src/clap-qualify.test.ts @@ -22,13 +22,11 @@ import { proberOffer, parseClapQualifyLine, parseClapQualifyOutput, - qualifyClap, - qualifyClapTargets, rtSafetyDocument, rtSafetyDocumentName, rtSafetyVerdict, - type ClapQualifyRun, } from "./clap-qualify.js"; +import { qualifyClap } from "./direct-prober.support.js"; import { HOSTING_CODE } from "./hosting-codes.js"; import { OMX_CLAP_HOST_CONFIGURED_EXTENSIONS, @@ -43,6 +41,7 @@ import { CLAP_SCAN_TOOL_ENV, CLAP_SWEEP_BLOCKS, NATIVE_TOOL_DIRS, + PROBE_RUNNER_ENV, QUALIFY_RATES, RT_INTERPOSE_ENV, RT_INTERPOSE_NAME, @@ -77,6 +76,14 @@ function live(name: string, body: () => void): void { const sha256 = (path: string): string => createHash("sha256").update(readFileSync(path)).digest("hex"); +/** A `plugin-hostd-probe` for the command-line tests: the real prober behind the runner's arguments (src/fake-probe-runner.support.mjs). */ +function standInRunner(): string { + const path = join(scratch, "plugin-hostd-probe"); + writeFileSync(path, `#!/bin/sh\nexec ${JSON.stringify(process.execPath)} ${JSON.stringify(join(PKG, "src", "fake-probe-runner.support.mjs"))} "$@"\n`); + chmodSync(path, 0o755); + return path; +} + /** A prober line as the tool prints it, with `patch` merged over the facts. */ function line( patch: { rtSafety?: object; clap?: object; hostExtensions?: string[]; integrity?: object | null; noteSweep?: object } = {}, @@ -260,13 +267,22 @@ describe("clap-qualify, live — the known RT-safe and RT-unsafe fixture plugins const run = (target: string, out: string) => spawnSync(process.execPath, [bin, "--rates", String(QUALIFY_RATES[0]), "--out", out, target], { encoding: "utf8", - env: { ...process.env, [CLAP_SCAN_TOOL_ENV]: stub }, + env: { ...process.env, [CLAP_SCAN_TOOL_ENV]: stub, [PROBE_RUNNER_ENV]: standInRunner() }, }); const safeOut = join(scratch, "safe"); const safe = run(SAFE, safeOut); expect(safe.status).toBe(0); expect(safe.stdout).toMatch(new RegExp(`urn:clap:${SAFE_ID.replace(/\./g, "\\.")}\\s+suitable`)); - expect(readdirSync(safeOut)).toEqual([`${sha256(SAFE)}.${OPENMIXER_CONSOLE_PROFILE.id}.rt-safety.json`]); + const written = readdirSync(safeOut).sort(); + expect(written).toHaveLength(2); + const [recordName, copyName] = [written.find((f) => f.endsWith(".qualification.json"))!, written.find((f) => f.endsWith(".probe.json"))!]; + expect(recordName.startsWith(`${SAFE_ID}@`)).toBe(true); + expect(recordName).toContain(`-${sha256(SAFE).slice(0, 12)}.`); + const record = JSON.parse(readFileSync(join(safeOut, recordName), "utf8")); + expect(record.probe.file).toBe(copyName); + expect(record.probe.sha256).toBe(sha256(join(safeOut, copyName))); + expect(record.binary.sha256).toBe(sha256(SAFE)); + expect(record.verdicts.map((v: { host: string; rating: string }) => [v.host, v.rating])).toEqual([[OPENMIXER_CONSOLE_PROFILE.id, "suitable"]]); const unsafe = run(UNSAFE, join(scratch, "unsafe")); expect(unsafe.status).toBe(1); expect(unsafe.stdout).toContain(HOSTING_CODE.rtSafetyViolations); @@ -348,7 +364,14 @@ describe("plugin-qualify --all, live — the standard CLAP search path", () => { const r = spawnSync(process.execPath, [bin, "--all", "--rates", String(QUALIFY_RATES[0])], { encoding: "utf8", // the prober is the scanner here too, so the walk reads the real file - env: { ...process.env, HOME: home, CLAP_PATH: onPath, LV2_PATH: join(scratch, "no-lv2"), [CLAP_SCAN_TOOL_ENV]: clapQualifyToolPath() }, + env: { + ...process.env, + HOME: home, + CLAP_PATH: onPath, + LV2_PATH: join(scratch, "no-lv2"), + [CLAP_SCAN_TOOL_ENV]: clapQualifyToolPath(), + [PROBE_RUNNER_ENV]: standInRunner(), + }, }); expect(r.stderr).toContain(`target urn:clap:${SAFE_ID} (${onPath})`); expect(r.stdout).toMatch(new RegExp(`urn:clap:${SAFE_ID.replace(/\./g, "\\.")}\\s+suitable`)); @@ -491,54 +514,9 @@ describe("proberOffer — the run that stands in for a profile", () => { expect(proberOffer({ ...OPENMIXER_CONSOLE_PROFILE, clapExtensions: [...OMX_CLAP_HOST_EXTENSIONS, "clap.gui"] })).toBeUndefined(); expect(proberOffer({ ...OPENMIXER_CONSOLE_PROFILE, clapExtensions: ["clap.log"] })).toBeUndefined(); }); - - it("qualifyClapTargets runs the prober once per offer, and judges each profile from its own run", () => { - const seen: (readonly string[])[] = []; - const run: ClapQualifyRun = (_tool, args) => { - seen.push(args); - const at = args.indexOf("--offer"); - const offered = at < 0 ? [] : args[at + 1]!.split(","); - return { status: 0, signal: null, stdout: line({ hostExtensions: [...OMX_CLAP_HOST_EXTENSIONS, ...offered] }) }; - }; - const env = { [CLAP_QUALIFY_TOOL_ENV]: "/t/clap-qualify", [RT_INTERPOSE_ENV]: "/t/rt.so" }; - const pass = qualifyClapTargets([{ format: "clap", uri: `urn:clap:${SAFE_ID}`, binaryPath: "/p/x.clap" }], { - rates: [48000], - run, - env, - exists: () => true, - profiles: [OPENMIXER_CONSOLE_PROFILE, OMX_CLAP_HOST_PROFILE, shippedHostProfile("jalv")!], - }); - expect(seen.map((a) => a.includes("--offer"))).toEqual([false, true]); - expect(pass.verdicts.map((v) => [v.host, v.rating])).toEqual([ - ["openmixer-console", "suitable"], - ["omx-clap-host", "suitable"], - ["jalv", "unknown"], - ]); - }); }); -describe("qualifyClap — the run, fail closed", () => { - const answering = (stdout: string, extra: { status?: number; signal?: string } = {}): { run: ClapQualifyRun; calls: unknown[] } => { - const calls: unknown[] = []; - return { calls, run: (tool, args, env) => (calls.push({ tool, args, preload: env["LD_PRELOAD"] }), { status: extra.status ?? 0, signal: extra.signal ?? null, stdout }) }; - }; - const exists = () => true; - - it("runs the named prober under the named interposer, at the rates asked", () => { - const { run, calls } = answering(line()); - const env = { [CLAP_QUALIFY_TOOL_ENV]: "/t/clap-qualify", [RT_INTERPOSE_ENV]: "/t/rt.so" }; - qualifyClap("/p/x.clap", SAFE_ID, { rates: [44100, 48000], run, env, exists }); - expect(calls).toEqual([ - { tool: "/t/clap-qualify", args: ["--rates", "44100,48000", "--block", "512", "--blocks", "512", "/p/x.clap", SAFE_ID], preload: "/t/rt.so" }, - ]); - }); - - it("an absent prober or interposer is refused, naming the setting", () => { - const { run } = answering(line()); - expect(() => qualifyClap("/p/x.clap", undefined, { rates: [48000], run, env: {}, exists: (p) => !p.endsWith(".so") })).toThrow(new RegExp(RT_INTERPOSE_ENV)); - expect(() => qualifyClap("/p/x.clap", undefined, { rates: [48000], run, env: {}, exists: () => false })).toThrow(new RegExp(CLAP_QUALIFY_TOOL_ENV)); - }); - +describe("clapQualifyTools — the prober and the interposer a run uses", () => { it("unset, the prober and the interposer are the package's own build, then the installed ones", () => { const own = [clapQualifyToolPath(), rtInterposePath()]; const installed = NATIVE_TOOL_DIRS.map((d) => [join(d, CLAP_QUALIFY_TOOL_NAME), join(d, RT_INTERPOSE_NAME)]); @@ -581,25 +559,4 @@ describe("qualifyClap — the run, fail closed", () => { new RegExp([clapQualifyToolPath(), ...NATIVE_TOOL_DIRS.map((d) => join(d, CLAP_QUALIFY_TOOL_NAME))].join(", ")), ); }); - - it("a crash, a failed load or silence is no measurement", () => { - expect(() => qualifyClap("/p/x.clap", undefined, { rates: [48000], run: answering("", { signal: "SIGSEGV", status: null as never }).run, env: {}, exists })).toThrow(/SIGSEGV/); - expect(() => qualifyClap("/p/x.clap", undefined, { rates: [48000], run: answering(line(), { status: 2 }).run, env: {}, exists })).toThrow(/exited 2/); - expect(() => qualifyClap("/p/x.clap", undefined, { rates: [48000], run: answering("").run, env: {}, exists })).toThrow(/nothing/); - }); - - it("a target that cannot be measured reads unknown, unmeasured, under every profile", () => { - const pass = qualifyClapTargets([{ format: "clap", uri: `urn:clap:${SAFE_ID}` }], { - rates: [48000], - env: {}, - exists, - profiles: [OPENMIXER_CONSOLE_PROFILE, shippedHostProfile("jalv")!], - }); - expect(pass.documents).toEqual([]); - expect(pass.failures).toHaveLength(1); - expect(pass.verdicts.map((v) => [v.host, v.measured, v.rating])).toEqual([ - [OPENMIXER_CONSOLE_PROFILE.id, false, "unknown"], - ["jalv", false, "unknown"], - ]); - }); }); diff --git a/src/clap-qualify.ts b/src/clap-qualify.ts index aee3a4f..c74d222 100644 --- a/src/clap-qualify.ts +++ b/src/clap-qualify.ts @@ -10,16 +10,9 @@ * read, a prober that is absent, crashed or timed out, is a {@link ClapQualifyError}: nothing was * measured, and nothing is rated. */ -import { spawnSync } from "node:child_process"; import { existsSync } from "node:fs"; -import { - CLAP_QUALIFY_TIMEOUT_S, - CLAP_QUALIFY_TOOL_NAME, - CLAP_SWEEP_BLOCKS, - CLAP_SWEEP_FRAMES, - RT_INTERPOSE_NAME, -} from "./qualify-declarations.js"; +import { CLAP_QUALIFY_TOOL_NAME, RT_INTERPOSE_NAME } from "./qualify-declarations.js"; import { HOSTING_CODE, clapRtReasons, @@ -37,7 +30,6 @@ import { OPENMIXER_CONSOLE_PROFILE, type HostProfile, } from "./host-profiles.js"; -import type { PluginTarget } from "./loader.js"; import type { QualifyVerdict } from "./qualify-cli.js"; import { RATING_SEVERITY, worstRating, type SuitabilityRating } from "./rating.js"; import { halfPopulatedRefusal, locateNativeTools } from "./tool-paths.js"; @@ -318,47 +310,7 @@ export function rtSafetyDocumentName(doc: RtSafetyDocument): string { return `${doc.plugin.binary.sha256}.${doc.console}.rt-safety.json`; } -// --- running the prober -------------------------------------------------------------------------- - -/** What a run of the prober gave back; `error` is a spawn failure, `signal` a death. */ -export interface ClapQualifyRunResult { - readonly status: number | null; - readonly signal?: string | null; - readonly stdout: string; - readonly stderr?: string; - readonly error?: Error; -} - -export type ClapQualifyRun = ( - tool: string, - args: readonly string[], - env: Readonly>, - timeoutMs: number, -) => ClapQualifyRunResult; - -const spawnQualify: ClapQualifyRun = (tool, args, env, timeoutMs) => { - const r = spawnSync(tool, [...args], { encoding: "utf8", env: { ...env }, timeout: timeoutMs, maxBuffer: 64 << 20 }); - return { - status: r.status, - signal: r.signal, - stdout: r.stdout ?? "", - stderr: r.stderr ?? "", - ...(r.error === undefined ? {} : { error: r.error }), - }; -}; - -export interface ClapQualifyOptions { - /** The rates the plugin is activated and swept at. */ - readonly rates: readonly number[]; - /** Runs the prober; the default spawns it. */ - readonly run?: ClapQualifyRun; - /** Where the two settings are read from; the default is `process.env`. */ - readonly env?: Readonly>; - /** Whether a file exists; the default asks the filesystem. */ - readonly exists?: (path: string) => boolean; - /** The host extensions the prober offers beyond the console's, each one of {@link OMX_CLAP_HOST_CONFIGURED_EXTENSIONS}. */ - readonly offer?: readonly string[]; -} +// --- the prober and the interposer a run uses --------------------------------------------------- /** * The prober and the interposer this run uses: each its setting, else both from one place, the @@ -381,44 +333,6 @@ export function clapQualifyTools( return { tool: tool!, interposer: interposer! }; } -/** - * Qualify the plugins of one `.clap` (all of them, or the one `id` names) under the interposer. - * Absent tools, a crash, a timeout, a load failure or an unreadable line throw: none of them is a - * measurement. - */ -export function qualifyClap(binaryPath: string, id: string | undefined, options: ClapQualifyOptions): ClapQualifyLine[] { - const env = options.env ?? process.env; - const { tool, interposer } = clapQualifyTools(env, options.exists ?? existsSync); - const args = [ - "--rates", - options.rates.join(","), - "--block", - String(CLAP_SWEEP_FRAMES), - "--blocks", - String(CLAP_SWEEP_BLOCKS), - ...(options.offer === undefined || options.offer.length === 0 ? [] : ["--offer", options.offer.join(",")]), - binaryPath, - ...(id === undefined ? [] : [id]), - ]; - const r = (options.run ?? spawnQualify)(tool, args, { ...env, LD_PRELOAD: interposer }, CLAP_QUALIFY_TIMEOUT_S * 1000); - if (r.error !== undefined) throw new ClapQualifyError(`cannot run ${tool}: ${r.error.message}`); - if (r.signal) throw new ClapQualifyError(`${binaryPath} killed the prober (${r.signal})`); - if (r.status !== 0) throw new ClapQualifyError(`${tool} exited ${String(r.status)} on ${binaryPath}: ${(r.stderr ?? "").trim()}`); - const lines = parseClapQualifyOutput(r.stdout); - if (lines.length === 0) throw new ClapQualifyError(`${tool} printed nothing for ${binaryPath}`); - return lines; -} - -// --- the CLI's CLAP pass ------------------------------------------------------------------------- - -/** What the CLAP pass of one run produced: a document per plugin and console, and a row per verdict. */ -export interface ClapQualifyPass { - readonly documents: readonly RtSafetyDocument[]; - readonly verdicts: readonly QualifyVerdict[]; - /** Each target that could not be measured, with why; its verdict rows read `unknown`, unmeasured. */ - readonly failures: readonly { readonly uri: string; readonly problem: string }[]; -} - /** * What the prober must offer beyond the console's extensions to measure for `profile`: the extra * extensions of its list, when the prober can offer each and the list holds the console's; `undefined` @@ -432,48 +346,3 @@ export function proberOffer(profile: HostProfile): readonly string[] | undefined if (!extra.every((e) => OMX_CLAP_HOST_CONFIGURED_EXTENSIONS.includes(e))) return undefined; return OMX_CLAP_HOST_CONFIGURED_EXTENSIONS.filter((e) => extra.includes(e)); } - -/** - * Qualify every CLAP target, one prober run per target and set of offered extensions (a crash takes - * only that plugin), and judge each line under every host profile, from the run that offered that - * profile's extensions. A target without a binary path (a bare `urn:clap:` id) has nothing to load - * and is a failure, never a pass. - */ -export function qualifyClapTargets( - targets: readonly PluginTarget[], - options: ClapQualifyOptions & { readonly profiles?: readonly HostProfile[] }, -): ClapQualifyPass { - const profiles = options.profiles ?? [OPENMIXER_CONSOLE_PROFILE]; - const documents: RtSafetyDocument[] = []; - const verdicts: QualifyVerdict[] = []; - const failures: { uri: string; problem: string }[] = []; - for (const target of targets) { - const id = target.uri.startsWith("urn:clap:") ? target.uri.slice("urn:clap:".length) : undefined; - // one run per distinct offer, the console's (none) for a profile the prober cannot stand in for - const offers = new Map(); - for (const profile of profiles) { - const offer = proberOffer(profile) ?? []; - offers.set(offer.join(","), offer); - } - const runs = new Map(); - try { - if (target.binaryPath === undefined) throw new ClapQualifyError(`${target.uri}: no .clap to load; name the file or a directory holding it`); - for (const [key, offer] of offers) runs.set(key, qualifyClap(target.binaryPath, id, { ...options, offer })); - } catch (e) { - if (!(e instanceof ClapQualifyError)) throw e; - failures.push({ uri: target.uri, problem: e.message }); - for (const profile of profiles) { - verdicts.push({ uri: target.uri, host: profile.id, measured: false, rating: "unknown", qualified: false, deciding: "not-measured" }); - } - continue; - } - for (const profile of profiles) { - for (const line of runs.get((proberOffer(profile) ?? []).join(","))!) { - const doc = rtSafetyDocument(line, profile); - documents.push(doc); - verdicts.push(rtSafetyVerdict(doc)); - } - } - } - return { documents, verdicts, failures }; -} diff --git a/src/cost-fields.ts b/src/cost-fields.ts index 45100dc..fedfd8d 100644 --- a/src/cost-fields.ts +++ b/src/cost-fields.ts @@ -70,3 +70,42 @@ export function classifierCostFromDocument(reading: CostAtRate, rate: number): C [COST_FIELDS.classifier.coreFraction]: fraction, }; } + +/** + * The key of one point of a cost grid: the rate and the quantum it was measured at. One place spells it, + * so the reader that fills a grid and the gate that looks a point up cannot disagree. + */ +export function costPointKey(rate: number, quantum: number): string { + return `${rate}/${quantum}`; +} + +/** The percentile the classifier's `nsPerSampleP95` is. A figure taken at another percentile is not that field. */ +export const CLASSIFIER_PERCENTILE = 95; + +/** One point of a grid as it is measured: the figures {@link gridCostReadings} turns into the classifier's. */ +export interface CostGridPoint { + readonly rate: number; + readonly quantum: number; + readonly outcome: string; + readonly cost?: { readonly median: number; readonly percentile: { readonly p: number; readonly value: number } }; +} + +/** + * The classifier's cost readings of a grid, by {@link costPointKey}. A point that did not end `probe.ok`, has no + * cost, or whose percentile is not {@link CLASSIFIER_PERCENTILE} has no reading: the gate then reads the pair as + * unmeasured, which is the truth, where a figure under the wrong name would be judged as if it were the P95. + */ +export function gridCostReadings(points: readonly CostGridPoint[], okOutcome: string): Record { + const out: Record = {}; + for (const p of points) { + if (p.outcome !== okOutcome || p.cost === undefined || p.cost.percentile.p !== CLASSIFIER_PERCENTILE) continue; + const fraction = coreFraction(p.cost.percentile.value, p.rate); + if (fraction === undefined) continue; + out[costPointKey(p.rate, p.quantum)] = { + [COST_FIELDS.classifier.median]: p.cost.median, + [COST_FIELDS.classifier.percentile]: p.cost.percentile.value, + [COST_FIELDS.classifier.coreFraction]: fraction, + }; + } + return out; +} diff --git a/src/cost-grid.test.ts b/src/cost-grid.test.ts new file mode 100644 index 0000000..a9f8765 --- /dev/null +++ b/src/cost-grid.test.ts @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +import { describe, expect, it } from "vitest"; + +import { costPointKey, gridCostReadings, type CostGridPoint } from "./cost-fields.js"; +import { DEFAULT_HOSTING_POLICY, HOSTING_CODE, OPENMIXER_CONSOLE_PROFILE, hostingCostReasons } from "./hosting-suitability.js"; + +const OK = "probe.ok"; +const point = (rate: number, quantum: number, p95: number, median = p95 / 2, p = 95, outcome = OK): CostGridPoint => ({ + rate, + quantum, + outcome, + cost: { median, percentile: { p, value: p95 } }, +}); + +const reasonsAt = (cpuCost: Parameters[0]["cpuCost"], rate: number, quantum: number | undefined) => + hostingCostReasons({ cpuCost }, { rate, ...(quantum === undefined ? {} : { quantum }), profile: OPENMIXER_CONSOLE_PROFILE }, DEFAULT_HOSTING_POLICY); + +describe("gridCostReadings", () => { + it("derives the core fraction from the measured primitive, per rate and quantum", () => { + const r = gridCostReadings([point(96000, 128, 600), point(96000, 512, 300, 250)], OK); + expect(Object.keys(r).sort()).toEqual([costPointKey(96000, 128), costPointKey(96000, 512)].sort()); + expect(r["96000/128"]).toEqual({ nsPerSampleMedian: 300, nsPerSampleP95: 600, coreFractionP95: (600 * 96000) / 1e9 }); + expect(r["96000/512"]?.nsPerSampleMedian).toBe(250); + }); + + it("leaves out a point that failed, one with no cost, and one taken at another percentile", () => { + const r = gridCostReadings( + [point(96000, 128, 600, 300, 95, "probe.crashed"), { rate: 96000, quantum: 256, outcome: OK }, point(96000, 512, 300, 150, 99)], + OK, + ); + expect(r).toEqual({}); + }); +}); + +describe("hostingCostReasons: cost at rate AND quantum", () => { + const grid = gridCostReadings([point(96000, 128, 600), point(96000, 1024, 300)], OK); + + it("gives two different reasons for two quanta of different cost", () => { + const small = reasonsAt({ perRateQuantum: grid }, 96000, 128); + const large = reasonsAt({ perRateQuantum: grid }, 96000, 1024); + expect(small.map((r) => [r.code, r.rating])).toEqual([[HOSTING_CODE.costAboveCeiling, "unsuitable"]]); + expect(small[0]?.params).toEqual({ coreFractionP95: (600 * 96000) / 1e9, ceiling: 0.05 }); + expect(large.map((r) => [r.code, r.rating])).toEqual([[HOSTING_CODE.costWithinCeiling, "suitable"]]); + expect(large[0]?.params).toEqual({ coreFractionP95: (300 * 96000) / 1e9, ceiling: 0.05 }); + }); + + it("reads a pair the grid lacks as unmeasured at that rate and quantum, naming both", () => { + const r = reasonsAt({ perRateQuantum: grid }, 96000, 256); + expect(r.map((x) => [x.code, x.rating, x.params])).toEqual([[HOSTING_CODE.costUnmeasured, "unknown", { rate: 96000, quantum: 256 }]]); + }); + + it("does not fall back to the per-rate figure when a grid is there, even for a pair it lacks", () => { + const perRate = { "96000": { coreFractionP95: 0.001, nsPerSampleMedian: 5, nsPerSampleP95: 10 } }; + const r = reasonsAt({ perRate, perRateQuantum: grid }, 96000, 256); + expect(r[0]?.code).toBe(HOSTING_CODE.costUnmeasured); + }); + + it("reads a grid with no quantum asked as unmeasured, quantum 0", () => { + const r = reasonsAt({ perRateQuantum: grid }, 96000, undefined); + expect(r[0]).toMatchObject({ code: HOSTING_CODE.costUnmeasured, params: { rate: 96000, quantum: 0 } }); + }); + + it("reads only the per-rate figure for a document of the old format, whatever the quantum", () => { + const perRate = { "96000": { coreFractionP95: 0.001, nsPerSampleMedian: 5, nsPerSampleP95: 10 } }; + for (const q of [128, 1024, undefined]) { + expect(reasonsAt({ perRate }, 96000, q)[0]?.code).toBe(HOSTING_CODE.costWithinCeiling); + } + }); + + it("keeps the spiky tail reason per point", () => { + const spiky = gridCostReadings([point(96000, 128, 100, 10), point(96000, 1024, 100, 90)], OK); + expect(reasonsAt({ perRateQuantum: spiky }, 96000, 128).map((r) => r.code)).toContain(HOSTING_CODE.costSpiky); + expect(reasonsAt({ perRateQuantum: spiky }, 96000, 1024).map((r) => r.code)).not.toContain(HOSTING_CODE.costSpiky); + }); +}); diff --git a/src/direct-prober.support.ts b/src/direct-prober.support.ts new file mode 100644 index 0000000..c5213b6 --- /dev/null +++ b/src/direct-prober.support.ts @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/** + * The prober run DIRECTLY, as the tests need it: `native/clap-qualify` under the interposer, its line + * read, no runner in between. This is what plugin-hostd's probe runner starts as its child, so the live + * tests of the prober and of the verdict fold use it to see the prober's own line. + * + * Not part of the package: the command line never calls the prober itself any more, it calls the runner + * (`probe-runner.ts`), which keeps the wall clock, the limits and the process tree. The spawn here has a + * timeout only so that a hung test cannot hang the suite. + */ +import { spawnSync } from "node:child_process"; +import { existsSync } from "node:fs"; + +import { + ClapQualifyError, + clapQualifyTools, + parseClapQualifyOutput, + proberOffer, + rtSafetyDocument, + rtSafetyVerdict, + type ClapQualifyLine, + type RtSafetyDocument, +} from "./clap-qualify.js"; +import type { HostProfile } from "./host-profiles.js"; +import { OPENMIXER_CONSOLE_PROFILE } from "./host-profiles.js"; +import type { PluginTarget } from "./loader.js"; +import type { QualifyVerdict } from "./qualify-cli.js"; +import { CLAP_QUALIFY_TIMEOUT_S, CLAP_SWEEP_BLOCKS, CLAP_SWEEP_FRAMES } from "./qualify-declarations.js"; + +// --- running the prober -------------------------------------------------------------------------- + +/** What a run of the prober gave back; `error` is a spawn failure, `signal` a death. */ +export interface ClapQualifyRunResult { + readonly status: number | null; + readonly signal?: string | null; + readonly stdout: string; + readonly stderr?: string; + readonly error?: Error; +} + +export type ClapQualifyRun = ( + tool: string, + args: readonly string[], + env: Readonly>, + timeoutMs: number, +) => ClapQualifyRunResult; + +const spawnQualify: ClapQualifyRun = (tool, args, env, timeoutMs) => { + const r = spawnSync(tool, [...args], { encoding: "utf8", env: { ...env }, timeout: timeoutMs, maxBuffer: 64 << 20 }); + return { + status: r.status, + signal: r.signal, + stdout: r.stdout ?? "", + stderr: r.stderr ?? "", + ...(r.error === undefined ? {} : { error: r.error }), + }; +}; + +export interface ClapQualifyOptions { + /** The rates the plugin is activated and swept at. */ + readonly rates: readonly number[]; + /** Runs the prober; the default spawns it. */ + readonly run?: ClapQualifyRun; + /** Where the two settings are read from; the default is `process.env`. */ + readonly env?: Readonly>; + /** Whether a file exists; the default asks the filesystem. */ + readonly exists?: (path: string) => boolean; + /** The host extensions the prober offers beyond the console's, each one of {@link OMX_CLAP_HOST_CONFIGURED_EXTENSIONS}. */ + readonly offer?: readonly string[]; +} + +/** + * Qualify the plugins of one `.clap` (all of them, or the one `id` names) under the interposer. + * Absent tools, a crash, a timeout, a load failure or an unreadable line throw: none of them is a + * measurement. + */ +export function qualifyClap(binaryPath: string, id: string | undefined, options: ClapQualifyOptions): ClapQualifyLine[] { + const env = options.env ?? process.env; + const { tool, interposer } = clapQualifyTools(env, options.exists ?? existsSync); + const args = [ + "--rates", + options.rates.join(","), + "--block", + String(CLAP_SWEEP_FRAMES), + "--blocks", + String(CLAP_SWEEP_BLOCKS), + ...(options.offer === undefined || options.offer.length === 0 ? [] : ["--offer", options.offer.join(",")]), + binaryPath, + ...(id === undefined ? [] : [id]), + ]; + const r = (options.run ?? spawnQualify)(tool, args, { ...env, LD_PRELOAD: interposer }, CLAP_QUALIFY_TIMEOUT_S * 1000); + if (r.error !== undefined) throw new ClapQualifyError(`cannot run ${tool}: ${r.error.message}`); + if (r.signal) throw new ClapQualifyError(`${binaryPath} killed the prober (${r.signal})`); + if (r.status !== 0) throw new ClapQualifyError(`${tool} exited ${String(r.status)} on ${binaryPath}: ${(r.stderr ?? "").trim()}`); + const lines = parseClapQualifyOutput(r.stdout); + if (lines.length === 0) throw new ClapQualifyError(`${tool} printed nothing for ${binaryPath}`); + return lines; +} + +// --- the CLI's CLAP pass ------------------------------------------------------------------------- + +/** What the CLAP pass of one run produced: a document per plugin and console, and a row per verdict. */ +export interface ClapQualifyPass { + readonly documents: readonly RtSafetyDocument[]; + readonly verdicts: readonly QualifyVerdict[]; + /** Each target that could not be measured, with why; its verdict rows read `unknown`, unmeasured. */ + readonly failures: readonly { readonly uri: string; readonly problem: string }[]; +} + +/** + * Qualify every CLAP target, one prober run per target and set of offered extensions (a crash takes + * only that plugin), and judge each line under every host profile, from the run that offered that + * profile's extensions. A target without a binary path (a bare `urn:clap:` id) has nothing to load + * and is a failure, never a pass. + */ +export function qualifyClapTargets( + targets: readonly PluginTarget[], + options: ClapQualifyOptions & { readonly profiles?: readonly HostProfile[] }, +): ClapQualifyPass { + const profiles = options.profiles ?? [OPENMIXER_CONSOLE_PROFILE]; + const documents: RtSafetyDocument[] = []; + const verdicts: QualifyVerdict[] = []; + const failures: { uri: string; problem: string }[] = []; + for (const target of targets) { + const id = target.uri.startsWith("urn:clap:") ? target.uri.slice("urn:clap:".length) : undefined; + // one run per distinct offer, the console's (none) for a profile the prober cannot stand in for + const offers = new Map(); + for (const profile of profiles) { + const offer = proberOffer(profile) ?? []; + offers.set(offer.join(","), offer); + } + const runs = new Map(); + try { + if (target.binaryPath === undefined) throw new ClapQualifyError(`${target.uri}: no .clap to load; name the file or a directory holding it`); + for (const [key, offer] of offers) runs.set(key, qualifyClap(target.binaryPath, id, { ...options, offer })); + } catch (e) { + if (!(e instanceof ClapQualifyError)) throw e; + failures.push({ uri: target.uri, problem: e.message }); + for (const profile of profiles) { + verdicts.push({ uri: target.uri, host: profile.id, measured: false, rating: "unknown", qualified: false, deciding: "not-measured" }); + } + continue; + } + for (const profile of profiles) { + for (const line of runs.get((proberOffer(profile) ?? []).join(","))!) { + const doc = rtSafetyDocument(line, profile); + documents.push(doc); + verdicts.push(rtSafetyVerdict(doc)); + } + } + } + return { documents, verdicts, failures }; +} diff --git a/src/fake-probe-runner.support.mjs b/src/fake-probe-runner.support.mjs new file mode 100644 index 0000000..ae2a99f --- /dev/null +++ b/src/fake-probe-runner.support.mjs @@ -0,0 +1,44 @@ +#!/usr/bin/env node +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/* + * A stand-in for plugin-hostd's `plugin-hostd-probe run`, for the live tests of the command line: it + * starts the real CLAP prober with the runner's arguments and writes the probe output that carries the + * prober's line (`probeOutputOfLine`). plugin-hostd's runner is not installed where the suite runs, and + * this is its stage-1 translation in the small, so a verdict the command line prints through it is the + * verdict of a real measurement read through the real reader. Not part of the package. + */ +import { spawnSync } from "node:child_process"; +import { renameSync, writeFileSync } from "node:fs"; + +import { probeOutputOfLine } from "../dist/probe-output-fixtures.js"; + +const argv = process.argv.slice(2); +if (argv[0] !== "run") { + process.stderr.write("usage: plugin-hostd-probe run [options] []\n"); + process.exit(2); +} +const opts = {}; +let i = 1; +for (; i < argv.length && argv[i].startsWith("--"); i += 2) opts[argv[i]] = argv[i + 1]; +const [file, id] = argv.slice(i); +if (file === undefined || opts["--out"] === undefined || opts["--child"] === undefined) { + process.stderr.write("a plugin file, --child and --out are needed\n"); + process.exit(2); +} +const args = [ + "--rates", opts["--rates"] ?? "48000", + "--block", opts["--quantum"] ?? "512", + "--blocks", opts["--blocks"] ?? "512", + ...(opts["--offer"] === undefined ? [] : ["--offer", opts["--offer"]]), + file, + ...(id === undefined ? [] : [id]), +]; +const r = spawnSync(opts["--child"], args, { encoding: "utf8", maxBuffer: 64 << 20 }); +if (r.status !== 0) { + process.stderr.write(`the child exited ${r.status}: ${r.stderr}`); + process.exit(r.status === 3 ? 3 : 2); +} +const line = JSON.parse(r.stdout.split("\n").find((l) => l.trim() !== "")); +writeFileSync(`${opts["--out"]}.new`, JSON.stringify(probeOutputOfLine(line))); +renameSync(`${opts["--out"]}.new`, opts["--out"]); diff --git a/src/hosting-suitability.ts b/src/hosting-suitability.ts index 2460f56..e2f9a70 100644 --- a/src/hosting-suitability.ts +++ b/src/hosting-suitability.ts @@ -52,11 +52,19 @@ export interface QualifiedPlugin { readonly scalingClass?: string; readonly paramSweep?: unknown; }; - /** Measured cost per rate (Hz as a string key). */ + /** Measured cost: per rate (Hz as a string key), or per rate AND quantum. */ readonly cpuCost?: { + /** One figure per rate, at the block the prober ran: the old format. Read only when `perRateQuantum` is absent. */ readonly perRate?: Readonly< Record >; + /** + * One figure per rate and quantum, keyed by {@link costPointKey}: a probe output's grid. When this is + * present it is the only cost there is, so a pair it lacks is unmeasured, never read from `perRate`. + */ + readonly perRateQuantum?: Readonly< + Record + >; }; readonly stability?: PluginStabilityMeasurement; readonly rtSafety?: PluginRtSafetyMeasurement; @@ -306,7 +314,7 @@ export type PluginLv2Features = Omit +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { afterAll, describe, expect, it } from "vitest"; + +import { OMX_CLAP_HOST_PROFILE, OPENMIXER_CONSOLE_PROFILE } from "./host-profiles.js"; +import { defaultQualificationDir, judgeExitCode, judgeFiles, parseJudgeArgs } from "./judge-cli.js"; +import { fixtureJson, fixtureText } from "./probe-output-fixtures.js"; + +const PKG = join(dirname(fileURLToPath(import.meta.url)), ".."); +const scratch = mkdtempSync(join(tmpdir(), "judge-cli-")); +afterAll(() => rmSync(scratch, { recursive: true, force: true })); + +const put = (name: string, text: string): string => { + const p = join(scratch, name); + writeFileSync(p, text); + return p; +}; + +describe("parseJudgeArgs", () => { + it("reads files, profiles (repeated or comma-separated, once each), the directory and the format", () => { + const r = parseJudgeArgs(["--host-profile", "omx-clap-host,jalv", "--host-profile", "jalv", "--qualification-dir", "q", "--format", "json", "a.json", "b.json"]); + expect(r).toEqual({ ok: true, args: { files: ["a.json", "b.json"], hostProfiles: ["omx-clap-host", "jalv"], qualificationDir: "q", format: "json", help: false } }); + const d = parseJudgeArgs(["a.json"]); + expect(d.ok && d.args.hostProfiles).toEqual(["openmixer-console"]); + }); + + it("refuses an unknown option, a missing value, a bad format and no file", () => { + expect(parseJudgeArgs(["--nope", "a"])).toEqual({ ok: false, error: "unknown option --nope" }); + expect(parseJudgeArgs(["a", "--qualification-dir"])).toEqual({ ok: false, error: "--qualification-dir needs a value" }); + expect(parseJudgeArgs(["a", "--format", "xml"]).ok).toBe(false); + expect(parseJudgeArgs([]).ok).toBe(false); + expect(parseJudgeArgs(["--help"]).ok).toBe(true); + }); +}); + +describe("defaultQualificationDir", () => { + it("is the XDG state directory, else the home's", () => { + expect(defaultQualificationDir({ XDG_STATE_HOME: "/s" }, "/h")).toBe("/s/omx/qualification"); + expect(defaultQualificationDir({ XDG_STATE_HOME: "" }, "/h")).toBe("/h/.local/state/omx/qualification"); + expect(defaultQualificationDir({}, "/h")).toBe("/h/.local/state/omx/qualification"); + }); +}); + +describe("judgeFiles", () => { + const profiles = [OMX_CLAP_HOST_PROFILE, OPENMIXER_CONSOLE_PROFILE]; + + it("writes a record and a byte copy per file, and the exit code follows the verdicts", () => { + const dir = join(scratch, "q1"); + const ok = put("ok.probe.json", fixtureText("success-msi")); + const r = judgeFiles([ok], { profiles: [OMX_CLAP_HOST_PROFILE], dir }); + expect(r.failures).toEqual([]); + expect(r.verdicts.map((v) => [v.host, v.rating])).toEqual([["omx-clap-host", "suitable"]]); + expect(judgeExitCode(r)).toBe(0); + expect(readdirSync(dir)).toHaveLength(2); + const record = JSON.parse(readFileSync(r.records[0]!.recordPath, "utf8")); + expect(record.probe.sha256).toBe(createHash("sha256").update(readFileSync(ok)).digest("hex")); + const mixed = judgeFiles([ok], { profiles, dir }); + expect(judgeExitCode(mixed)).toBe(1); + }); + + it("a refused file is named with its code and exits 2, and the files after it are still judged", () => { + const bad = fixtureJson("success-msi"); + bad.formatVersion = "2.0"; + const badPath = put("bad.json", JSON.stringify(bad)); + const dir = join(scratch, "q2"); + const r = judgeFiles([badPath, join(scratch, "missing.json"), put("good.json", fixtureText("success-msi"))], { profiles: [OMX_CLAP_HOST_PROFILE], dir }); + expect(r.failures.map((f) => f.problem.split(":")[0])).toEqual(["probe.format.major-unknown", "cannot read"]); + expect(r.verdicts).toHaveLength(1); + expect(judgeExitCode(r)).toBe(2); + }); + + it("a run that crashed exits 2, its record kept", () => { + const r = judgeFiles([put("crash.json", fixtureText("plugin-crash"))], { profiles: [OPENMIXER_CONSOLE_PROFILE], dir: join(scratch, "q3") }); + expect(r.verdicts[0]).toMatchObject({ measured: false, deciding: "probe.crashed" }); + expect(judgeExitCode(r)).toBe(2); + expect(r.records).toHaveLength(1); + }); +}); + +describe("plugin-qualify judge, the command", () => { + const bin = join(PKG, "bin", "plugin-qualify.mjs"); + it("prints the verdict table, writes the pair and exits by the verdicts", (ctx) => { + if (!readdirSync(PKG).includes("dist")) ctx.skip("dist is not built"); + const dir = join(scratch, "cli"); + const r = spawnSync(process.execPath, [bin, "judge", "--host-profile", "omx-clap-host", "--qualification-dir", dir, put("cli.probe.json", fixtureText("success-msi"))], { encoding: "utf8" }); + expect(r.status).toBe(0); + expect(r.stdout).toMatch(/org\.example\.plate\s+suitable\s+-\s+omx-clap-host/); + expect(readdirSync(dir).sort().map((f) => f.replace(/^.*\.omx-clap-host\./, ""))).toEqual(["probe.json", "qualification.json"]); + const bad = spawnSync(process.execPath, [bin, "judge", "--qualification-dir", dir, join(scratch, "none.json")], { encoding: "utf8" }); + expect(bad.status).toBe(2); + expect(bad.stderr).toContain("cannot read"); + }); +}); diff --git a/src/judge-cli.ts b/src/judge-cli.ts new file mode 100644 index 0000000..1ed6393 --- /dev/null +++ b/src/judge-cli.ts @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/** + * `plugin-qualify judge ...`: judge probe outputs already written (by + * `plugin-hostd-probe run`, by plugin-hostd's `judge_cmd`, or by hand) and write the qualification + * record beside a byte copy of each. `bin/plugin-qualify.mjs` only reads argv and prints. + */ +import { readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +import type { HostProfile } from "./host-profiles.js"; +import { judgeProbeOutput, judgmentVerdicts } from "./probe-judge.js"; +import { readProbeOutput } from "./probe-output.js"; +import { writeQualification } from "./qualification-record.js"; +import type { QualifyVerdict } from "./qualify-cli.js"; +import { EXIT_CODE, OUTPUT_FORMATS, type OutputFormat } from "./qualify-options.js"; +import { OPENMIXER_CONSOLE_PROFILE } from "./host-profiles.js"; + +export interface JudgeArgs { + readonly files: readonly string[]; + readonly hostProfiles: readonly string[]; + readonly qualificationDir?: string; + readonly format: OutputFormat; + readonly help: boolean; +} + +export type ParseJudgeResult = { ok: true; args: JudgeArgs } | { ok: false; error: string }; + +const VALUE_FLAGS = ["--host-profile", "--qualification-dir", "--format"]; + +export function parseJudgeArgs(argv: readonly string[]): ParseJudgeResult { + const files: string[] = []; + const specs: string[] = []; + let dir: string | undefined; + let format: string = "text"; + let help = false; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]!; + if (!a.startsWith("--")) { + files.push(a); + continue; + } + if (a === "--help") { + help = true; + continue; + } + if (!VALUE_FLAGS.includes(a)) return { ok: false, error: `unknown option ${a}` }; + const v = argv[++i]; + if (v === undefined || v.startsWith("--")) return { ok: false, error: `${a} needs a value` }; + if (a === "--host-profile") specs.push(...v.split(",").map((t) => t.trim()).filter((t) => t !== "")); + else if (a === "--qualification-dir") dir = v; + else format = v; + } + if (!(OUTPUT_FORMATS as readonly string[]).includes(format)) return { ok: false, error: `--format must be one of ${OUTPUT_FORMATS.join(", ")}` }; + if (!help && files.length === 0) return { ok: false, error: "name at least one probe output" }; + return { + ok: true, + args: { + files, + hostProfiles: [...new Set(specs.length > 0 ? specs : [OPENMIXER_CONSOLE_PROFILE.id])], + ...(dir === undefined ? {} : { qualificationDir: dir }), + format: format as OutputFormat, + help, + }, + }; +} + +export function judgeHelpText(): string { + return [ + "plugin-qualify judge — judge probe outputs and write the qualification records", + "", + " plugin-qualify judge [options] ...", + "", + " --host-profile judge against this host; repeat or comma-separate for one verdict per profile", + ` (default: ${OPENMIXER_CONSOLE_PROFILE.id})`, + " --qualification-dir where the records and the copies of the probe outputs go", + " (default: $XDG_STATE_HOME/omx/qualification, else ~/.local/state/omx/qualification)", + " --format json = the records; text = plugin, verdict, deciding reason (default: text)", + "", + `exit ${EXIT_CODE.allQualified} all qualified / ${EXIT_CODE.someNotQualified} some not / ${EXIT_CODE.couldNotMeasure} something could not be read or judged`, + "", + ].join("\n"); +} + +/** The qualification directory when none is named: the state directory of owner spec section 6.9, per user. */ +export function defaultQualificationDir(env: Readonly>, home: string = homedir()): string { + const state = env["XDG_STATE_HOME"]; + return join(state !== undefined && state !== "" ? state : join(home, ".local", "state"), "omx", "qualification"); +} + +export interface JudgeFilesResult { + readonly verdicts: readonly QualifyVerdict[]; + readonly records: readonly { readonly recordPath: string; readonly probePath: string }[]; + /** Each file that was refused or could not be read, with the code or the reason. */ + readonly failures: readonly { readonly file: string; readonly problem: string }[]; +} + +/** Judge each file under every profile and write its record; a file that does not read is a failure, and the others go on. */ +export function judgeFiles( + files: readonly string[], + options: { profiles: readonly HostProfile[]; dir: string; read?: (path: string) => Buffer }, +): JudgeFilesResult { + const read = options.read ?? ((p: string) => readFileSync(p)); + const verdicts: QualifyVerdict[] = []; + const records: { recordPath: string; probePath: string }[] = []; + const failures: { file: string; problem: string }[] = []; + for (const file of files) { + let bytes: Buffer; + try { + bytes = read(file); + } catch (e) { + failures.push({ file, problem: `cannot read: ${e instanceof Error ? e.message : String(e)}` }); + continue; + } + const r = readProbeOutput(bytes.toString("utf8")); + if (!r.ok) { + failures.push({ file, problem: `${r.code}: ${r.problem}` }); + continue; + } + const judgment = judgeProbeOutput(r.output, options.profiles); + const w = writeQualification(options.dir, bytes, judgment); + records.push({ recordPath: w.recordPath, probePath: w.probePath }); + verdicts.push(...judgmentVerdicts(judgment)); + } + return { verdicts, records, failures }; +} + +/** 2 when a file could not be judged or a plugin was not measured, else 0 when every verdict qualifies, else 1. */ +export function judgeExitCode(result: JudgeFilesResult): number { + if (result.failures.length > 0 || result.verdicts.length === 0 || result.verdicts.some((v) => !v.measured)) return EXIT_CODE.couldNotMeasure; + return result.verdicts.every((v) => v.qualified) ? EXIT_CODE.allQualified : EXIT_CODE.someNotQualified; +} diff --git a/src/pins-file.test.ts b/src/pins-file.test.ts index c13d591..0918c8c 100644 --- a/src/pins-file.test.ts +++ b/src/pins-file.test.ts @@ -16,7 +16,8 @@ import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; import { afterAll, describe, expect, it } from "vitest"; -import { qualifyClapTargets, type RtSafetyDocument } from "./clap-qualify.js"; +import type { RtSafetyDocument } from "./clap-qualify.js"; +import { qualifyClapTargets } from "./direct-prober.support.js"; import { OPENMIXER_CONSOLE_PROFILE } from "./host-profiles.js"; import { clapPinEntries, diff --git a/src/probe-judge.test.ts b/src/probe-judge.test.ts new file mode 100644 index 0000000..e37fe17 --- /dev/null +++ b/src/probe-judge.test.ts @@ -0,0 +1,271 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/** + * The stage-2 proof: a verdict read from a probe output equals the verdict read from today's + * document, line by line and profile by profile, and two quanta of different cost give two + * different cost reasons. + */ +import { describe, expect, it } from "vitest"; + +import { rtSafetyDocument, rtSafetyVerdict, type ClapQualifyLine } from "./clap-qualify.js"; +import { HOSTING_CODE } from "./hosting-codes.js"; +import { + JALV_PROFILE, + OMX_CLAP_HOST_PROFILE, + OMX_CLAP_HOST_EXTENSIONS, + OPENMIXER_CONSOLE_PROFILE, + type HostProfile, +} from "./host-profiles.js"; +import { clapLineFromProbeOutput, judgeProbeOutput } from "./probe-judge.js"; +import { readProbeOutput, type ProbeOutput } from "./probe-output.js"; +import { clapLineFixture, fixtureJson, probeOutputOfLine, readFixture } from "./probe-output-fixtures.js"; + +const withGui = [...OMX_CLAP_HOST_EXTENSIONS, "clap.gui"]; +const OWN_GUI: HostProfile = { ...OPENMIXER_CONSOLE_PROFILE, id: "own-gui", clapExtensions: withGui }; + +/** Every shape of line the existing tests of the CLAP arm exercise, each with the profiles it is judged under. */ +const CASES: { name: string; line: ClapQualifyLine; profiles: HostProfile[]; expected: string }[] = [ + { name: "a clean effect", line: clapLineFixture(), profiles: [OPENMIXER_CONSOLE_PROFILE, OMX_CLAP_HOST_PROFILE, JALV_PROFILE, OWN_GUI], expected: "suitable" }, + { name: "an allocation", line: clapLineFixture({ rtSafety: { allocationsInRun: 3 } }), profiles: [OPENMIXER_CONSOLE_PROFILE], expected: "unsuitable" }, + { name: "a lock", line: clapLineFixture({ rtSafety: { locksInRun: 1 } }), profiles: [OPENMIXER_CONSOLE_PROFILE], expected: "unsuitable" }, + { name: "a syscall", line: clapLineFixture({ rtSafety: { syscallsInRun: 3 } }), profiles: [OPENMIXER_CONSOLE_PROFILE], expected: "unsuitable" }, + { name: "a thread violation", line: clapLineFixture({ clap: { threadViolations: 1 } }), profiles: [OPENMIXER_CONSOLE_PROFILE], expected: "unsuitable" }, + { name: "a process error", line: clapLineFixture({ clap: { processErrors: 2 } }), profiles: [OPENMIXER_CONSOLE_PROFILE], expected: "unsuitable" }, + { name: "a sweep that ran nothing", line: clapLineFixture({ clap: { processCalls: 0 } }), profiles: [OPENMIXER_CONSOLE_PROFILE], expected: "unknown" }, + { name: "no interposer", line: clapLineFixture({ rtSafety: { interposer: false } }), profiles: [OPENMIXER_CONSOLE_PROFILE], expected: "unknown" }, + { name: "controls not swept", line: clapLineFixture({ rtSafety: { swept: false } }), profiles: [OPENMIXER_CONSOLE_PROFILE], expected: "unknown" }, + { name: "a plugin with no parameter", line: clapLineFixture({ rtSafety: { swept: false, sweepApplicable: false } }), profiles: [OPENMIXER_CONSOLE_PROFILE], expected: "suitable" }, + { + name: "other host extensions than the profile's", + line: clapLineFixture({ hostExtensions: withGui }), + profiles: [OPENMIXER_CONSOLE_PROFILE, OWN_GUI], + expected: "unknown", + }, + { + name: "a played instrument, clean", + line: clapLineFixture({ + clap: { noteInputs: 1 }, + noteSweep: { applicable: true, dialect: "clap", blocks: 512, blockFrames: 512, noteEvents: 40, processErrors: 0, nonFinite: false, interposer: true, allocationsInRun: 0, locksInRun: 0, syscallsInRun: 0, events: [] }, + }), + profiles: [OPENMIXER_CONSOLE_PROFILE], + expected: "suitable", + }, + { + name: "a played instrument that allocates, in the midi dialect", + line: clapLineFixture({ + clap: { noteInputs: 1 }, + noteSweep: { applicable: true, dialect: "midi", blocks: 512, blockFrames: 512, noteEvents: 40, processErrors: 0, nonFinite: false, interposer: true, allocationsInRun: 5, locksInRun: 0, syscallsInRun: 0, events: [] }, + }), + profiles: [OPENMIXER_CONSOLE_PROFILE], + expected: "unsuitable", + }, + { + name: "an instrument played with a syscall", + line: clapLineFixture({ + clap: { noteInputs: 1 }, + noteSweep: { applicable: true, dialect: "clap", blocks: 512, blockFrames: 512, noteEvents: 40, processErrors: 0, nonFinite: false, interposer: true, allocationsInRun: 0, locksInRun: 0, syscallsInRun: 2, events: [] }, + }), + profiles: [OPENMIXER_CONSOLE_PROFILE], + expected: "unsuitable", + }, + { + name: "an instrument that could not be played", + line: clapLineFixture({ clap: { noteInputs: 1 }, noteSweep: { applicable: false } }), + profiles: [OPENMIXER_CONSOLE_PROFILE], + expected: "unknown", + }, + { + name: "an instrument played with no interposer", + line: clapLineFixture({ + clap: { noteInputs: 1 }, + rtSafety: { interposer: false }, + noteSweep: { applicable: true, dialect: "clap", blocks: 512, blockFrames: 512, noteEvents: 40, processErrors: 0, nonFinite: false, interposer: false }, + }), + profiles: [OPENMIXER_CONSOLE_PROFILE], + expected: "unknown", + }, +]; + +function judged(line: ClapQualifyLine, profile: HostProfile) { + const read = readProbeOutput(JSON.stringify(probeOutputOfLine(line))); + if (!read.ok) throw new Error(`${read.code}: ${read.problem}`); + return judgeProbeOutput(read.output, [profile], { noteInputs: line.clap.noteInputs }); +} + +describe("verdicts from a probe output equal verdicts from today's documents", () => { + for (const c of CASES) { + for (const profile of c.profiles) { + it(`${c.name}, judged for ${profile.id}`, () => { + const today = rtSafetyDocument(c.line, profile); + const j = judged(c.line, profile); + expect(j.hosts).toHaveLength(1); + expect(j.hosts[0]?.verdict).toEqual(rtSafetyVerdict(today)); + expect(j.hosts[0]?.reasons.map((r) => [r.dimension, r.rating, r.code])).toEqual( + today.verdict.reasons.map((r) => [r.dimension, r.rating, r.code]), + ); + expect(j.documents[0]?.verdict.rating).toBe(today.verdict.rating); + expect(j.documents[0]?.measured.integrity).toEqual(c.line.integrity); + }); + } + } + + it("the cases span every rating, so the equality is not three copies of one answer", () => { + const ratings = new Set(CASES.flatMap((c) => c.profiles.map((p) => rtSafetyDocument(c.line, p).verdict.rating))); + expect([...ratings].sort()).toEqual(["suitable", "unknown", "unsuitable"]); + for (const c of CASES) expect(rtSafetyDocument(c.line, c.profiles[0]!).verdict.rating).toBe(c.expected); + }); + + it("a probe output with the counts changed gives the changed verdict (the proof can fail)", () => { + const line = clapLineFixture(); + const doc = probeOutputOfLine(line); + doc["grid"][0]["rtSafety"]["allocations"] = 1; + const read = readProbeOutput(JSON.stringify(doc)); + if (!read.ok) throw new Error(read.problem); + const j = judgeProbeOutput(read.output, [OPENMIXER_CONSOLE_PROFILE]); + expect(j.hosts[0]?.verdict).toMatchObject({ rating: "unsuitable", qualified: false, deciding: HOSTING_CODE.rtSafetyViolations }); + expect(j.hosts[0]?.verdict).not.toEqual(rtSafetyVerdict(rtSafetyDocument(line))); + }); +}); + +describe("the line a probe output stands for", () => { + it("sums the interposer's counts over the points that carry them and reads a missing count as unmeasured", () => { + const doc = probeOutputOfLine(clapLineFixture()); + doc["grid"][1]["rtSafety"] = { ...doc["grid"][0]["rtSafety"], allocations: 2, blocks: 100, threadViolations: 1 }; + const read = readProbeOutput(JSON.stringify(doc)); + if (!read.ok) throw new Error(read.problem); + const line = clapLineFromProbeOutput(read.output); + expect(line.rtSafety).toMatchObject({ allocationsInRun: 2, blocks: 612, interposer: true }); + expect(line.clap.threadViolations).toBe(1); + doc["grid"][1]["rtSafety"]["threadViolations"] = 0; + delete doc["grid"][1]["rtSafety"]["locks"]; + const again = readProbeOutput(JSON.stringify(doc)); + if (!again.ok) throw new Error(again.problem); + expect(clapLineFromProbeOutput(again.output).rtSafety.locksInRun).toBeUndefined(); + expect(judgeProbeOutput(again.output, [OPENMIXER_CONSOLE_PROFILE]).hosts[0]?.verdict).toMatchObject({ rating: "unknown", deciding: HOSTING_CODE.rtSafetyUnmeasured }); + }); + + it("reads the interposer as absent when any counting point ran without it", () => { + const doc = probeOutputOfLine(clapLineFixture()); + doc["grid"][1]["rtSafety"] = { interposer: false }; + const read = readProbeOutput(JSON.stringify(doc)); + if (!read.ok) throw new Error(read.problem); + expect(clapLineFromProbeOutput(read.output).rtSafety).toEqual({ interposer: false }); + }); + + it("takes note inputs from the descriptor's fact when given, else from the note sweep", () => { + const withNotes = probeOutputOfLine(clapLineFixture({ clap: { noteInputs: 1 }, noteSweep: { applicable: true, dialect: "clap", interposer: true, allocationsInRun: 0, locksInRun: 0, syscallsInRun: 0 } })); + const read = readProbeOutput(JSON.stringify(withNotes)); + if (!read.ok) throw new Error(read.problem); + expect(clapLineFromProbeOutput(read.output).clap.noteInputs).toBe(1); + expect(clapLineFromProbeOutput(read.output, { noteInputs: 3 }).clap.noteInputs).toBe(3); + const none = readFixture("success-msi"); + expect(clapLineFromProbeOutput(none).clap.noteInputs).toBe(0); + }); +}); + +describe("the fixtures of plugin-hostd", () => { + it("the success on the reference rig is suitable for omx-clap-host, and unknown for a console it did not offer to", () => { + const o = readFixture("success-msi"); + const j = judgeProbeOutput(o, [OMX_CLAP_HOST_PROFILE, OPENMIXER_CONSOLE_PROFILE]); + expect(j.hosts.map((h) => [h.verdict.host, h.verdict.rating, h.hosting])).toEqual([ + ["omx-clap-host", "suitable", "isolated"], + ["openmixer-console", "unknown", "isolated"], + ]); + expect(j.hosts[1]?.verdict.deciding).toBe(HOSTING_CODE.clapExtensionsNotCovered); + expect(j.hosts[0]?.reasons.map((r) => r.code)).toEqual([HOSTING_CODE.rtSafetyClean]); + expect(j.documents[0]?.measured.integrity.layout?.sha256).toBe("0a3e01399262321514e05af0575ebb0b207a35567ac262a060fde655ee069805"); + }); + + it("a run that crashed is unknown under every profile, the run's outcome the reason, and unmeasured for the exit code", () => { + const o = readFixture("plugin-crash"); + const j = judgeProbeOutput(o, [OPENMIXER_CONSOLE_PROFILE, OMX_CLAP_HOST_PROFILE]); + expect(j.hosts.map((h) => h.verdict)).toEqual([ + { uri: "urn:example:zero-path", host: "openmixer-console", measured: false, rating: "unknown", qualified: false, deciding: "probe.crashed" }, + { uri: "urn:example:zero-path", host: "omx-clap-host", measured: false, rating: "unknown", qualified: false, deciding: "probe.crashed" }, + ]); + expect(j.documents).toEqual([]); + const refused = judgeProbeOutput(readFixture("memory-limit-refused"), [OPENMIXER_CONSOLE_PROFILE]); + expect(refused.hosts[0]?.verdict.deciding).toBe("probe.memory-limit"); + }); + + it("a grid point that failed under a run that says ok is the failure it records, never a pass that skipped the point", () => { + const doc = fixtureJson("success-msi"); + doc["grid"][1]["outcome"] = "probe.timeout"; + const read = readProbeOutput(JSON.stringify(doc)); + if (!read.ok) throw new Error(read.problem); + const j = judgeProbeOutput(read.output, [OMX_CLAP_HOST_PROFILE]); + expect(j.hosts[0]?.verdict).toMatchObject({ measured: false, rating: "unknown", deciding: "probe.timeout" }); + doc["grid"][1]["outcome"] = "probe.not-attempted"; + doc["grid"][1]["rtSafety"]["allocations"] = 5; + const skipped = readProbeOutput(JSON.stringify(doc)); + if (!skipped.ok) throw new Error(skipped.problem); + expect(judgeProbeOutput(skipped.output, [OMX_CLAP_HOST_PROFILE]).hosts[0]?.verdict).toMatchObject({ measured: true, rating: "suitable" }); + }); + + it("an LV2 plugin is judged on the interposer's counts, under its URI", () => { + const doc = fixtureJson("success-msi"); + doc["plugin"] = { format: "lv2", id: "urn:example:plate" }; + delete doc["offer"]["clapExtensions"]; + const read = readProbeOutput(JSON.stringify(doc)); + if (!read.ok) throw new Error(read.problem); + const clean = judgeProbeOutput(read.output, [OPENMIXER_CONSOLE_PROFILE]); + expect(clean.hosts[0]?.verdict).toEqual({ uri: "urn:example:plate", host: "openmixer-console", measured: true, rating: "suitable", qualified: true }); + expect(clean.hosts[0]?.hosting).toBe("in-process"); + expect(clean.documents).toEqual([]); + doc["grid"][1]["rtSafety"]["syscalls"] = 7; + const dirty = readProbeOutput(JSON.stringify(doc)); + if (!dirty.ok) throw new Error(dirty.problem); + const j = judgeProbeOutput(dirty.output, [OPENMIXER_CONSOLE_PROFILE]); + expect(j.hosts[0]?.verdict).toMatchObject({ rating: "unsuitable", deciding: HOSTING_CODE.rtSafetyViolations }); + expect(j.hosts[0]?.reasons[0]?.params).toEqual({ allocations: 0, syscalls: 7, locks: 0 }); + }); +}); + +describe("cost at rate and quantum, from the grid of a probe output", () => { + /** The success fixture with its grid replaced by cost points at one rate and the given quanta. */ + function withCost(points: { quantum: number; p95: number; median: number }[]): ProbeOutput { + const doc = fixtureJson("success-msi"); + const costOf = (p: { p95: number; median: number }) => ({ + unit: "ns-per-sample", + median: p.median, + percentile: { p: 95, value: p.p95 }, + blocks: 1000, + warmupBlocks: 64, + }); + doc["grid"] = points.map((p) => ({ rate: 96000, quantum: p.quantum, outcome: "probe.ok", cost: costOf(p), rtSafety: doc["grid"][0]["rtSafety"] })); + const read = readProbeOutput(JSON.stringify(doc)); + if (!read.ok) throw new Error(read.problem); + return read.output; + } + const small = { ...OPENMIXER_CONSOLE_PROFILE, id: "small-quantum", cost: { ...OPENMIXER_CONSOLE_PROFILE.cost, quantum: 128 } }; + const large = OPENMIXER_CONSOLE_PROFILE; + + it("two quanta of different cost give two different cost reasons", () => { + const o = withCost([ + { quantum: 128, p95: 800, median: 700 }, + { quantum: large.cost.quantum, p95: 200, median: 180 }, + ]); + const [a, b] = judgeProbeOutput(o, [small, large]).hosts; + expect(a?.cost.quantum).toBe(128); + expect(b?.cost.quantum).toBe(large.cost.quantum); + expect(a?.cost.reasons.map((r) => [r.code, r.rating, r.params])).toEqual([ + [HOSTING_CODE.costAboveCeiling, "unsuitable", { coreFractionP95: (800 * 96000) / 1e9, ceiling: 0.05 }], + ]); + expect(b?.cost.reasons.map((r) => [r.code, r.rating, r.params])).toEqual([ + [HOSTING_CODE.costWithinCeiling, "suitable", { coreFractionP95: (200 * 96000) / 1e9, ceiling: 0.05 }], + ]); + }); + + it("a quantum the grid does not hold reads unmeasured at that rate and quantum, never the nearest", () => { + const o = withCost([{ quantum: 128, p95: 800, median: 700 }]); + const [, b] = judgeProbeOutput(o, [small, large]).hosts; + expect(b?.cost.reasons.map((r) => [r.code, r.params])).toEqual([[HOSTING_CODE.costUnmeasured, { rate: 96000, quantum: large.cost.quantum }]]); + }); + + it("cost sits beside the verdict and does not change it", () => { + const o = withCost([{ quantum: 128, p95: 800, median: 700 }]); + const [a] = judgeProbeOutput(o, [small]).hosts; + expect(a?.cost.reasons[0]?.rating).toBe("unsuitable"); + expect(a?.verdict.rating).not.toBe("unsuitable"); + }); +}); diff --git a/src/probe-judge.ts b/src/probe-judge.ts new file mode 100644 index 0000000..2895f90 --- /dev/null +++ b/src/probe-judge.ts @@ -0,0 +1,264 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/** + * Judging from a probe output. The probe output holds facts (`probe-output.ts`); this module turns + * them into the facts the one judge already reads and lets that judge speak. There is no second + * judge: a CLAP plugin's verdict is {@link rtSafetyDocument}'s fold of the same reasons it folds + * for a prober line, an LV2 plugin's is {@link rtSafetyReasons}, and cost is read by + * {@link hostingCostReasons} at the host's rate and quantum. + * + * What a probe output does not carry, this module does not invent: + * + * - `noteInputs` is the descriptor's fact. Without one the run's note sweep stands in: a plugin + * whose sweep was applicable has at least one note input; one whose sweep was not applicable is + * read as having none, so an instrument that could not be played reads as an effect. The caller + * that holds the descriptor passes the real count. + * - The note sweep's block count is the parameter sweep's: both run in one prober process at one + * block count, and the schema holds the note sweep's counts but not its blocks. + */ +import { DEFAULT_HOSTING_POLICY } from "./qualify-declarations.js"; +import { + rtSafetyDocument, + type ClapNoteSweepReading, + type ClapQualifyLine, + type ClapRtSafetyReading, + type RtSafetyDocument, +} from "./clap-qualify.js"; +import { gridCostReadings } from "./cost-fields.js"; +import { + hostingCostReasons, + rtSafetyReasons, + contributedHostingRating, + type HostingReason, +} from "./hosting-suitability.js"; +import type { HostProfile } from "./host-profiles.js"; +import { PROBE_NOT_ATTEMPTED, PROBE_OK, type ProbeGridPoint, type ProbeOutput } from "./probe-output.js"; +import type { QualifyVerdict } from "./qualify-cli.js"; +import { RATING_SEVERITY, worstRating, type SuitabilityRating } from "./rating.js"; +import type { IntegrityMeasurement } from "./measurement/format.js"; + +/** Facts of the descriptor, which the probe output repeats none of. */ +export interface JudgeFacts { + /** Note input ports the plugin declares. */ + readonly noteInputs?: number; +} + +/** A sum of one count over the points, or `undefined` when a point lacks it: a missing count is not zero. */ +function total(points: readonly ProbeGridPoint[], count: (p: ProbeGridPoint) => number | undefined): number | undefined { + let sum = 0; + for (const p of points) { + const n = count(p); + if (n === undefined) return undefined; + sum += n; + } + return sum; +} + +/** The points whose run ended well: a point that crashed measured nothing. */ +function okPoints(output: ProbeOutput): ProbeGridPoint[] { + return output.grid.filter((p) => p.outcome === PROBE_OK); +} + +/** + * The interposer's counts over every grid point that carries them. The interposer must have been + * loaded at every such point; the sweep counts as done when every point swept, and as applicable + * when any did. A count a point lacks leaves the total absent, and the judge reads that as + * unmeasured. + */ +export function rtSafetyOfProbeOutput(output: ProbeOutput): ClapRtSafetyReading { + const points = okPoints(output).filter((p) => p.rtSafety !== undefined); + if (points.length === 0 || points.some((p) => p.rtSafety?.interposer !== true)) return { interposer: false }; + const reading = (p: ProbeGridPoint) => p.rtSafety; + const allocations = total(points, (p) => reading(p)?.allocations); + const locks = total(points, (p) => reading(p)?.locks); + const syscalls = total(points, (p) => reading(p)?.syscalls); + return { + interposer: true, + swept: points.every((p) => reading(p)?.swept === true), + sweepApplicable: points.some((p) => reading(p)?.sweepApplicable === true), + blocks: total(points, (p) => reading(p)?.blocks ?? 0) ?? 0, + blockFrames: points[0]?.quantum ?? 0, + ...(allocations === undefined ? {} : { allocationsInRun: allocations }), + ...(locks === undefined ? {} : { locksInRun: locks }), + ...(syscalls === undefined ? {} : { syscallsInRun: syscalls }), + events: points.flatMap((p) => reading(p)?.events ?? []), + }; +} + +function noteSweepOfProbeOutput(output: ProbeOutput, rt: ClapRtSafetyReading): ClapNoteSweepReading { + const played = okPoints(output).find((p) => p.noteSweep?.applicable === true)?.noteSweep; + if (played === undefined) return { applicable: false }; + const dialect: { dialect?: "clap" | "midi" } = {}; + if (played.dialect === "clap" || played.dialect === "midi") dialect.dialect = played.dialect; + const base = { applicable: true, ...dialect, blocks: rt.blocks ?? 0, blockFrames: rt.blockFrames ?? 0, noteEvents: 0, processErrors: 0, nonFinite: false }; + if (!rt.interposer) return { ...base, interposer: false }; + return { + ...base, + interposer: true, + ...(played.allocations === undefined ? {} : { allocationsInRun: played.allocations }), + ...(played.locks === undefined ? {} : { locksInRun: played.locks }), + ...(played.syscalls === undefined ? {} : { syscallsInRun: played.syscalls }), + }; +} + +function integrityOfProbeOutput(output: ProbeOutput): IntegrityMeasurement { + const { files, layout, unpinnable } = output.binary; + if (layout === undefined || files.length === 0) { + return { unmeasuredReason: unpinnable === undefined ? "probe-error" : "unpinnable", unmeasuredDetail: unpinnable ?? "the probe output holds no layout pin" }; + } + return { + ...(output.plugin.version === undefined ? {} : { version: output.plugin.version }), + files: files.slice().sort((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0)), + layout, + }; +} + +/** + * A CLAP probe output as the prober's line it stands for, so that {@link rtSafetyDocument} judges it. + * The facts the line holds and a probe output does not (the port layout, the features) are not read + * by the rtSafety fold and are left empty here. + */ +export function clapLineFromProbeOutput(output: ProbeOutput, facts: JudgeFacts = {}): ClapQualifyLine { + const rtSafety = rtSafetyOfProbeOutput(output); + const noteSweep = noteSweepOfProbeOutput(output, rtSafety); + const points = okPoints(output); + const latency: Record = {}; + for (const p of points) { + if (p.latency !== undefined && latency[String(p.rate)] === undefined) latency[String(p.rate)] = p.latency; + } + const withRt = points.filter((p) => p.rtSafety !== undefined); + return { + uri: `urn:clap:${output.plugin.id}`, + id: output.plugin.id, + binary: { path: output.binary.path, sha256: output.binary.sha256 }, + hostExtensions: output.offer.clapExtensions ?? [], + clap: { + mainInputChannels: 0, + mainOutputChannels: 0, + sidechainInputs: 0, + features: [], + headlessOk: output.steps.some((s) => s.step === "instantiate" && s.outcome === PROBE_OK), + noteInputs: facts.noteInputs ?? (noteSweep.applicable ? 1 : 0), + latency, + threadViolations: total(withRt, (p) => p.rtSafety?.threadViolations ?? 0) ?? 0, + processErrors: total(withRt, (p) => p.rtSafety?.processErrors ?? 0) ?? 0, + // the sweep ran as many blocks as the points that counted say; a run with none measured nothing + processCalls: rtSafety.interposer ? (rtSafety.blocks ?? 0) : points.length, + nonFinite: output.steps.some((s) => s.outcome === "probe.non-finite-output"), + }, + rtSafety, + noteSweep, + integrity: integrityOfProbeOutput(output), + }; +} + +/** One reason as the record writes it: the code is a string since a failed run's reason is the run's outcome code. */ +export interface RecordReason { + readonly dimension: string; + readonly rating: SuitabilityRating; + readonly code: string; + readonly params: Readonly>; +} + +const asRecordReason = (r: HostingReason): RecordReason => ({ dimension: r.dimension, rating: contributedHostingRating(r), code: r.code, params: r.params }); + +/** What one host profile makes of one probe output. */ +export interface HostJudgment { + readonly verdict: QualifyVerdict; + /** In-process only for a suitable plugin on a host that runs its plugins in its own process, else isolated. */ + readonly hosting: "in-process" | "isolated"; + readonly reasons: readonly RecordReason[]; + /** The cost reasons at this host's rate and quantum, kept beside the verdict, not folded into it. */ + readonly cost: { readonly rate: number; readonly quantum: number; readonly reasons: readonly RecordReason[] }; +} + +export interface Judgment { + readonly output: ProbeOutput; + readonly hosts: readonly HostJudgment[]; + /** For a CLAP plugin, the document of each host that today's judge writes; the pins are taken from these. */ + readonly documents: readonly RtSafetyDocument[]; +} + +function decidingOf(reasons: readonly RecordReason[], rating: SuitabilityRating): RecordReason | undefined { + if (rating === "suitable") return undefined; + return [...reasons].sort((a, b) => RATING_SEVERITY[b.rating] - RATING_SEVERITY[a.rating]).find((r) => r.rating === rating); +} + +function costOf(output: ProbeOutput, profile: HostProfile): HostJudgment["cost"] { + const { rate, quantum } = profile.cost; + const reasons = hostingCostReasons( + { cpuCost: { perRateQuantum: gridCostReadings(output.grid, PROBE_OK) } }, + { rate, quantum, profile }, + DEFAULT_HOSTING_POLICY, + ); + return { rate, quantum, reasons: reasons.map(asRecordReason) }; +} + +/** + * The outcome that ended the run badly, or `undefined` when it ended well. The run's own outcome is the + * first word; a grid point that failed under a run that says `probe.ok` is a document that contradicts + * itself, and is read as the failure it records rather than as a pass that skipped the point. + */ +function failedOutcome(output: ProbeOutput): string | undefined { + if (output.run.outcome !== PROBE_OK) return output.run.outcome; + return output.grid.find((p) => p.outcome !== PROBE_OK && p.outcome !== PROBE_NOT_ATTEMPTED)?.outcome; +} + +/** + * Every profile's judgment of one probe output. + * + * A run that did not end `probe.ok` is `unknown` under every profile, with the run's outcome code as + * the reason: the measurement did not finish, so there is nothing to rate, and it is not a plugin + * `unsuitable` on evidence the run never gathered. A step that was not attempted is part of a run that + * ended well (a plugin with no note input has no note sweep). + */ +export function judgeProbeOutput(output: ProbeOutput, profiles: readonly HostProfile[], facts: JudgeFacts = {}): Judgment { + const uri = output.plugin.format === "clap" ? `urn:clap:${output.plugin.id}` : output.plugin.id; + const failure = failedOutcome(output); + if (failure !== undefined) { + const reason: RecordReason = { dimension: "rtSafety", rating: "unknown", code: failure, params: {} }; + return { + output, + documents: [], + hosts: profiles.map((profile) => ({ + verdict: { uri, host: profile.id, measured: false, rating: "unknown", qualified: false, deciding: failure }, + hosting: "isolated", + reasons: [reason], + cost: costOf(output, profile), + })), + }; + } + const line = output.plugin.format === "clap" ? clapLineFromProbeOutput(output, facts) : undefined; + const documents: RtSafetyDocument[] = []; + const hosts = profiles.map((profile): HostJudgment => { + let reasons: RecordReason[]; + if (line !== undefined) { + const doc = rtSafetyDocument(line, profile); + documents.push(doc); + reasons = doc.verdict.reasons.map(asRecordReason); + } else { + reasons = rtSafetyReasons(rtSafetyOfProbeOutput(output)).map(asRecordReason); + } + const rating = worstRating(reasons.map((r) => r.rating)); + const deciding = decidingOf(reasons, rating); + return { + verdict: { + uri, + host: profile.id, + measured: true, + rating, + qualified: rating === "suitable", + ...(deciding === undefined ? {} : { deciding: deciding.code }), + }, + hosting: rating === "suitable" && profile.isolation === "in-process" ? "in-process" : "isolated", + reasons, + cost: costOf(output, profile), + }; + }); + return { output, hosts, documents }; +} + +/** The verdict rows of a judgment, as the CLI prints them. */ +export function judgmentVerdicts(judgment: Judgment): QualifyVerdict[] { + return judgment.hosts.map((h) => h.verdict); +} diff --git a/src/probe-output-fixtures.ts b/src/probe-output-fixtures.ts new file mode 100644 index 0000000..29a15ee --- /dev/null +++ b/src/probe-output-fixtures.ts @@ -0,0 +1,146 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/** + * The probe-output documents the tests build on, read from `fixtures/probe-output/`: plugin-hostd's + * three example documents (a success on the reference rig, a crash, a refusal for memory). Kept + * out of the test files so every test reads the same documents; a `.test.ts` imported for a helper + * runs its whole suite in each importer. + */ +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import type { ClapQualifyLine } from "./clap-qualify.js"; +import { OMX_CLAP_HOST_EXTENSIONS } from "./host-profiles.js"; +import { readProbeOutput, type ProbeOutput } from "./probe-output.js"; + +const FIXTURES = join(dirname(fileURLToPath(import.meta.url)), "..", "fixtures", "probe-output"); + +export function fixtureText(name: string): string { + return readFileSync(join(FIXTURES, `${name}.json`), "utf8"); +} + +/** A fixture as a mutable JSON value, for the tests that break one field of it. */ +export function fixtureJson(name: string): Record { + return JSON.parse(fixtureText(name)); +} + +/** A fixture through the reader; throws when the reader refuses it. */ +export function readFixture(name: string): ProbeOutput { + const r = readProbeOutput(fixtureText(name)); + if (!r.ok) throw new Error(`${name}: ${r.code}: ${r.problem}`); + return r.output; +} + +/** + * The probe output that carries what a prober line carries, for the proofs that a verdict read from a + * probe output equals the verdict read from today's document. Built on the success fixture, so its machine, + * limits and run are a real document's. + * + * What the line holds and a probe output cannot is not encoded: the port layout and the features (no + * verdict reads them), the note sweep's process errors and non-finite flag, and a block count apart + * from the sweep's (the probe output has one, the blocks the sweep processed, which is `processCalls`). + */ +export function probeOutputOfLine(line: ClapQualifyLine): Record { + const doc = fixtureJson("success-msi"); + const rt = line.rtSafety; + const note = line.noteSweep; + const pin = line.integrity; + const rates = Object.keys(line.clap.latency); + const quantum = rt.blockFrames ?? 512; + doc["plugin"] = { format: "clap", id: line.id, ...(pin.version === undefined ? {} : { version: pin.version }) }; + doc["binary"] = { + path: line.binary.path, + sha256: line.binary.sha256, + files: pin.files ?? [], + ...(pin.layout === undefined ? {} : { layout: pin.layout }), + ...(pin.unmeasuredReason === undefined ? {} : { unpinnable: pin.unmeasuredDetail ?? pin.unmeasuredReason }), + }; + delete doc["descriptor"]; + doc["offer"] = { id: "default", clapExtensions: [...line.hostExtensions], realtime: "none" }; + doc["steps"] = [{ step: "hash", outcome: "probe.ok", elapsedMs: 1 }, { step: "instantiate", outcome: "probe.ok", elapsedMs: 1 }]; + const first: Record = {}; + if (rt.interposer) { + Object.assign(first, { + rtSafety: { + interposer: true, + swept: rt.swept, + sweepApplicable: rt.sweepApplicable, + blocks: line.clap.processCalls, + allocations: rt.allocationsInRun, + locks: rt.locksInRun, + syscalls: rt.syscallsInRun, + threadViolations: line.clap.threadViolations, + processErrors: line.clap.processErrors, + events: rt.events ?? [], + }, + }); + } else first["rtSafety"] = { interposer: false }; + if (note.applicable) { + first["noteSweep"] = { + applicable: true, + ...(note.dialect === undefined ? {} : { dialect: note.dialect }), + ...(note.interposer === true ? { allocations: note.allocationsInRun, locks: note.locksInRun, syscalls: note.syscallsInRun } : {}), + }; + } + const at = rates.length > 0 ? rates : ["48000"]; + doc["grid"] = at.map((rate, i) => ({ + rate: Number(rate), + quantum, + outcome: "probe.ok", + ...(line.clap.latency[rate] === undefined ? {} : { latency: { ...line.clap.latency[rate], method: "impulse-onset" } }), + ...(i === 0 ? first : {}), + })); + return doc; +} + +const PIN = { + version: "1", + files: [{ path: "rt-safe.clap", sha256: "a".repeat(64) }], + layout: { scheme: "omx-layout/1", sha256: "b".repeat(64) }, +}; + +/** A prober line for a clean stereo effect, with `patch` merged over it. */ +export function clapLineFixture(patch: { + rtSafety?: object; + clap?: object; + hostExtensions?: string[]; + noteSweep?: object; + integrity?: object; +} = {}): ClapQualifyLine { + return { + uri: "urn:clap:org.example.fx", + id: "org.example.fx", + binary: { path: "/x/rt-safe.clap", sha256: "a".repeat(64) }, + hostExtensions: patch.hostExtensions ?? [...OMX_CLAP_HOST_EXTENSIONS], + clap: { + mainInputChannels: 2, + mainOutputChannels: 2, + sidechainInputs: 0, + features: ["audio-effect"], + headlessOk: true, + noteInputs: 0, + latency: { "48000": { declaredFrames: 0, measuredFrames: 0 }, "96000": { declaredFrames: 4, measuredFrames: 4 } }, + threadViolations: 0, + processErrors: 0, + processCalls: 512, + nonFinite: false, + ...patch.clap, + }, + rtSafety: { + interposer: true, + swept: true, + sweepApplicable: true, + blocks: 512, + blockFrames: 512, + allocationsInRun: 0, + locksInRun: 0, + syscallsInRun: 0, + events: [], + ...patch.rtSafety, + }, + noteSweep: { applicable: false, ...patch.noteSweep }, + integrity: { ...PIN, ...patch.integrity }, + }; +} + diff --git a/src/probe-output.test.ts b/src/probe-output.test.ts new file mode 100644 index 0000000..894a01d --- /dev/null +++ b/src/probe-output.test.ts @@ -0,0 +1,186 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +import { describe, expect, it } from "vitest"; + +import { PROBE_FORMAT_CODE, readProbeOutput } from "./probe-output.js"; +import { fixtureJson, readFixture } from "./probe-output-fixtures.js"; + +function refused(doc: unknown): { code: string; problem: string } { + const r = readProbeOutput(JSON.stringify(doc)); + if (r.ok) throw new Error("the document was read; the reader should have refused it"); + return { code: r.code, problem: r.problem }; +} + +describe("readProbeOutput: the three documents of plugin-hostd", () => { + it("reads the success on the reference rig, figure by figure", () => { + const o = readFixture("success-msi"); + expect(o.run.outcome).toBe("probe.ok"); + expect(o.run.protocol).toEqual({ name: "plugin-hostd.probe-protocol", version: 1 }); + expect(o.plugin).toMatchObject({ format: "clap", id: "org.example.plate", version: "1.4.2" }); + expect(o.binary.sha256).toBe("356902794a4eec2122989e072271ef995968131145db918d9a3084ce582f244b"); + expect(o.binary.layout?.scheme).toBe("omx-layout/1"); + expect(o.grid.map((p) => [p.rate, p.quantum])).toEqual([ + [48000, 128], + [96000, 128], + ]); + expect(o.grid[0]?.cost).toEqual({ median: 41.2, percentile: { p: 95, value: 47.9 }, blocks: 11250, warmupBlocks: 64 }); + expect(o.grid[1]?.rtSafety).toMatchObject({ interposer: true, allocations: 0, locks: 0, syscalls: 0 }); + expect(o.machine.profile.meetsProtocol).toBe(true); + expect(o.machine.class["cstatesLimited"]).toBe(true); + expect(o.offer.clapExtensions).toHaveLength(9); + }); + + it("reads the crash and the refusal, which have steps and no figures to speak of", () => { + const crash = readFixture("plugin-crash"); + expect(crash.run.complete).toBe(false); + expect(crash.run.outcome).toBe("probe.crashed"); + expect(crash.steps.at(-1)).toMatchObject({ step: "rt-safety", outcome: "probe.crashed" }); + expect(crash.machine.profile.unmet).toEqual(["probe.profile.governor", "probe.profile.deep-idle-entered"]); + const refusal = readFixture("memory-limit-refused"); + expect(refusal.run.outcome).toBe("probe.memory-limit"); + expect(refusal.grid).toEqual([]); + }); +}); + +describe("readProbeOutput: the major", () => { + it("refuses a major it does not know, before it reads anything else", () => { + const r = refused({ format: "plugin-hostd.probe-output", formatVersion: "2.0" }); + expect(r.code).toBe("probe.format.major-unknown"); + expect(PROBE_FORMAT_CODE.majorUnknown).toBe("probe.format.major-unknown"); + }); + + it("refuses major 2 of a full document, and major 0", () => { + for (const v of ["2.0", "3.4", "0.9"]) { + const doc = fixtureJson("success-msi"); + doc["formatVersion"] = v; + expect(refused(doc).code).toBe("probe.format.major-unknown"); + } + }); + + it("reads a newer minor and ignores the fields it adds", () => { + const doc = fixtureJson("success-msi"); + doc["formatVersion"] = "1.7"; + doc["newThing"] = { anything: 1 }; + doc["grid"][0]["cost"]["futureStatistic"] = 3; + const r = readProbeOutput(JSON.stringify(doc)); + expect(r.ok).toBe(true); + if (r.ok) expect(r.output.formatVersion).toBe("1.7"); + }); + + it("refuses a version that is not major.minor", () => { + for (const v of ["1", "1.0.0", "one", 1.0]) { + const doc = fixtureJson("success-msi"); + doc["formatVersion"] = v; + expect(refused(doc).code).toBe("probe.format.invalid"); + } + }); +}); + +describe("readProbeOutput: derived fields are recomputed, never trusted", () => { + it("refuses cstatesLimited true when cpuidle holds a deep state that is enabled", () => { + const doc = fixtureJson("success-msi"); + doc["machine"]["profile"]["cpuidle"][2]["disabled"] = false; + const r = refused(doc); + expect(r.code).toBe("probe.format.derived-mismatch"); + expect(r.problem).toContain("cstatesLimited is true but cpuidle[] gives false"); + }); + + it("refuses cstatesLimited true when a deep state was entered while the run measured", () => { + const doc = fixtureJson("success-msi"); + doc["machine"]["profile"]["cpuidle"][2]["usageDelta"] = 4; + expect(refused(doc).code).toBe("probe.format.derived-mismatch"); + }); + + it("refuses cstatesLimited false when no deep state is enabled", () => { + const doc = fixtureJson("success-msi"); + doc["machine"]["class"]["cstatesLimited"] = false; + expect(refused(doc).code).toBe("probe.format.derived-mismatch"); + }); + + it("refuses cstatesLimited true when the machine reads no idle state at all", () => { + const doc = fixtureJson("success-msi"); + doc["machine"]["profile"]["cpuidle"] = []; + expect(refused(doc).code).toBe("probe.format.derived-mismatch"); + }); + + it("refuses meetsProtocol true under a governor that is not performance", () => { + const doc = fixtureJson("success-msi"); + doc["machine"]["profile"]["governor"] = "powersave"; + const r = refused(doc); + expect(r.code).toBe("probe.format.derived-mismatch"); + expect(r.problem).toContain("probe.profile.governor"); + }); + + it("refuses an unmet list that leaves out a deep state the run entered", () => { + const doc = fixtureJson("plugin-crash"); + doc["machine"]["profile"]["unmet"] = ["probe.profile.governor"]; + expect(refused(doc).code).toBe("probe.format.derived-mismatch"); + }); + + it("refuses meetsProtocol that disagrees with unmet in either direction", () => { + const a = fixtureJson("success-msi"); + a["machine"]["profile"]["unmet"] = ["probe.profile.not-pinned"]; + expect(refused(a).code).toBe("probe.format.derived-mismatch"); + const b = fixtureJson("plugin-crash"); + b["machine"]["profile"]["meetsProtocol"] = true; + expect(refused(b).code).toBe("probe.format.derived-mismatch"); + }); + + it("takes a code it cannot recompute as the runner wrote it", () => { + const doc = fixtureJson("success-msi"); + doc["machine"]["profile"]["meetsProtocol"] = false; + doc["machine"]["profile"]["unmet"] = ["probe.profile.not-pinned"]; + const r = readProbeOutput(JSON.stringify(doc)); + expect(r.ok).toBe(true); + }); +}); + +describe("readProbeOutput: a document that is not a probe output", () => { + it("refuses a judgment inside it, whichever key carries it", () => { + for (const key of ["verdict", "verdicts", "rating", "hosting"]) { + const doc = fixtureJson("success-msi"); + doc[key] = key === "verdicts" ? [] : "suitable"; + const r = refused(doc); + expect(r.code).toBe("probe.format.invalid"); + expect(r.problem).toContain(key); + } + }); + + it("refuses text that is not JSON, another format, and a missing block", () => { + expect(readProbeOutput("{not json")).toMatchObject({ ok: false, code: "probe.format.invalid", problem: "not JSON" }); + expect(refused({ format: "lv2-plugin-measurements", formatVersion: "1.3" }).code).toBe("probe.format.invalid"); + const doc = fixtureJson("success-msi"); + delete doc["binary"]; + expect(refused(doc).problem).toContain("binary"); + }); + + it("refuses a digest that is not 64 lowercase hex digits", () => { + const doc = fixtureJson("success-msi"); + doc["binary"]["sha256"] = doc["binary"]["sha256"].toUpperCase(); + expect(refused(doc).code).toBe("probe.format.invalid"); + }); + + it("refuses a grid that names one rate and quantum twice, since a cost read there would be ambiguous", () => { + const doc = fixtureJson("success-msi"); + doc["grid"][1]["rate"] = 48000; + expect(refused(doc).problem).toContain("rate 48000 at quantum 128 twice"); + }); + + it("refuses a cost percentile below the median's side, a negative figure and an outcome that is no probe code", () => { + const a = fixtureJson("success-msi"); + a["grid"][0]["cost"]["percentile"]["p"] = 50; + expect(refused(a).code).toBe("probe.format.invalid"); + const b = fixtureJson("success-msi"); + b["grid"][0]["cost"]["median"] = -1; + expect(refused(b).code).toBe("probe.format.invalid"); + const c = fixtureJson("success-msi"); + c["steps"][0]["outcome"] = "exploded"; + expect(refused(c).code).toBe("probe.format.invalid"); + }); + + it("accepts an outcome code of a later minor, which only adds codes", () => { + const doc = fixtureJson("success-msi"); + doc["steps"][0]["outcome"] = "probe.something-new"; + expect(readProbeOutput(JSON.stringify(doc)).ok).toBe(true); + }); +}); diff --git a/src/probe-output.ts b/src/probe-output.ts new file mode 100644 index 0000000..5bb3f63 --- /dev/null +++ b/src/probe-output.ts @@ -0,0 +1,505 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/** + * The reader of plugin-hostd's probe output (`plugin-hostd.probe-output`): the document + * `plugin-hostd-probe run` writes about one plugin binary and one offer. It holds facts only, so + * this module reads and checks; it never judges (`probe-judge.ts` does). + * + * The reader is structural, as `measurement/probe-json.ts` is: it validates what plugin-qualify + * needs and ignores the rest, so a newer minor of the format, which only adds fields, still reads. + * It refuses three things, each with a code that is never reworded: + * + * - `probe.format.major-unknown`: a major version this reader does not know. The check comes + * before the content is looked at, since a new major may change what the content means. + * - `probe.format.derived-mismatch`: `machine.class.cstatesLimited` or `machine.profile.meetsProtocol` + * disagrees with the fields they are computed from. Both are kept in the document because the + * machine class is shared; neither is trusted, both are recomputed here. + * - `probe.format.invalid`: anything else that is not the document: not JSON, a verdict inside it + * (the format has no field for one), a missing required block, a malformed figure. + */ +import { createHash } from "node:crypto"; + +import { finiteNumber, isRecord, nonEmptyString, optional } from "./measurement/guards.js"; + +export const PROBE_OUTPUT_FORMAT = "plugin-hostd.probe-output"; +/** The one major this reader knows. A minor of it only adds fields. */ +export const PROBE_OUTPUT_MAJOR = 1; + +export const PROBE_FORMAT_CODE = { + majorUnknown: "probe.format.major-unknown", + derivedMismatch: "probe.format.derived-mismatch", + invalid: "probe.format.invalid", +} as const; +export type ProbeFormatCode = (typeof PROBE_FORMAT_CODE)[keyof typeof PROBE_FORMAT_CODE]; + +/** The outcome that means a step ran to its end. */ +export const PROBE_OK = "probe.ok"; +export const PROBE_NOT_ATTEMPTED = "probe.not-attempted"; + +/** + * An exit-latency above this many microseconds makes an idle state deep. The runner writes the + * document by this rule, so the reader recomputes by it; the protocol document will state it + * (plugin-hostd#78) and this number follows that document. + */ +export const DEEP_IDLE_LATENCY_US = 20; + +/** The keys the format has no room for: a document holding one is a judgment, not a fact. */ +const JUDGMENT_KEYS = ["verdict", "verdicts", "rating", "hosting"] as const; + +const SHA256 = /^[0-9a-f]{64}$/; +const OUTCOME = /^probe\.[a-z][a-z0-9.-]*$/; + +export interface ProbeFile { + readonly path: string; + readonly sha256: string; +} + +export interface ProbeCost { + readonly median: number; + /** The percentile the figure is taken at, and its value, in nanoseconds per sample. */ + readonly percentile: { readonly p: number; readonly value: number }; + readonly blocks: number; + readonly warmupBlocks: number; +} + +export interface ProbeRtSafety { + readonly interposer: boolean; + readonly swept?: boolean; + readonly sweepApplicable?: boolean; + readonly blocks?: number; + readonly allocations?: number; + readonly locks?: number; + readonly syscalls?: number; + readonly threadViolations?: number; + readonly processErrors?: number; + readonly events?: readonly string[]; +} + +export interface ProbeNoteSweep { + readonly applicable: boolean; + readonly dialect?: string; + readonly allocations?: number; + readonly locks?: number; + readonly syscalls?: number; +} + +/** One point of the protocol's grid: the figures at one rate and one quantum. */ +export interface ProbeGridPoint { + readonly rate: number; + readonly quantum: number; + readonly outcome: string; + readonly latency?: { readonly declaredFrames: number; readonly measuredFrames: number }; + readonly cost?: ProbeCost; + readonly rtSafety?: ProbeRtSafety; + readonly noteSweep?: ProbeNoteSweep; +} + +export interface ProbeStep { + readonly step: string; + readonly outcome: string; + readonly rate?: number; + readonly quantum?: number; + readonly detail?: string; +} + +export interface ProbeIdleState { + readonly cpu: number; + readonly state: string; + readonly latencyUs: number; + readonly disabled: boolean; + readonly usageDelta: number; +} + +export interface ProbeOutput { + readonly formatVersion: string; + readonly run: { + readonly id: string; + readonly startedAt: string; + readonly elapsedMs: number; + readonly complete: boolean; + readonly outcome: string; + readonly reason?: string; + readonly protocol: { readonly name: string; readonly version: number }; + readonly tools: Readonly>; + readonly via: string; + }; + readonly plugin: { + readonly format: "clap" | "lv2"; + readonly id: string; + readonly name?: string; + readonly version?: string; + }; + readonly binary: { + readonly path: string; + readonly sha256: string; + readonly files: readonly ProbeFile[]; + readonly layout?: { readonly scheme: string; readonly sha256: string }; + /** Why the files cannot be pinned, when they cannot. */ + readonly unpinnable?: string; + }; + readonly machine: { + readonly class: Readonly>; + readonly profile: { + readonly governor: string | null; + readonly cpuidle: readonly ProbeIdleState[]; + readonly meetsProtocol: boolean; + readonly unmet: readonly string[]; + }; + readonly pinnedCpus: string; + }; + readonly offer: { readonly id: string; readonly clapExtensions?: readonly string[] }; + readonly steps: readonly ProbeStep[]; + readonly grid: readonly ProbeGridPoint[]; + /** What the document says about how the plugin behaves, as it was written. */ + readonly behaviour?: Readonly>; +} + +export type ReadProbeOutputResult = + | { readonly ok: true; readonly output: ProbeOutput } + | { readonly ok: false; readonly code: ProbeFormatCode; readonly problem: string }; + +class Refusal extends Error { + constructor( + readonly code: ProbeFormatCode, + message: string, + ) { + super(message); + } +} + +function invalid(what: string): never { + throw new Refusal(PROBE_FORMAT_CODE.invalid, what); +} + +function record(v: unknown, where: string): Record { + return isRecord(v) ? v : invalid(`${where} is not an object`); +} + +function str(o: Record, key: string, where: string): string { + return nonEmptyString(o[key]) ?? invalid(`${where} has no ${key}`); +} + +function bool(o: Record, key: string, where: string): boolean { + const v = o[key]; + return typeof v === "boolean" ? v : invalid(`${where} has no boolean ${key}`); +} + +function num(o: Record, key: string, where: string): number { + return finiteNumber(o[key]) ?? invalid(`${where} has no number ${key}`); +} + +function posInt(o: Record, key: string, where: string): number { + const v = o[key]; + return typeof v === "number" && Number.isSafeInteger(v) && v > 0 ? v : invalid(`${where} has no positive integer ${key}`); +} + +function count(o: Record, key: string, where: string): number { + const v = o[key]; + return typeof v === "number" && Number.isSafeInteger(v) && v >= 0 ? v : invalid(`${where} has no count ${key}`); +} + +function optCount(o: Record, key: string, where: string): number | undefined { + return o[key] === undefined ? undefined : count(o, key, where); +} + +function optBool(o: Record, key: string, where: string): boolean | undefined { + return o[key] === undefined ? undefined : bool(o, key, where); +} + +function outcome(o: Record, where: string): string { + const v = str(o, "outcome", where); + return OUTCOME.test(v) ? v : invalid(`${where} outcome ${v} is not a probe code`); +} + +function list(o: Record, key: string, where: string): unknown[] { + const v = o[key]; + return Array.isArray(v) ? v : invalid(`${where} has no list ${key}`); +} + +function sha(o: Record, key: string, where: string): string { + const v = str(o, key, where); + return SHA256.test(v) ? v : invalid(`${where} ${key} is not 64 lowercase hex digits`); +} + +/** The strings of a list that holds nothing else, or `undefined`. */ +function stringList(v: unknown): string[] | undefined { + if (!Array.isArray(v)) return undefined; + const out: string[] = []; + for (const item of v) { + if (typeof item !== "string") return undefined; + out.push(item); + } + return out; +} + +function readCost(v: unknown, where: string): ProbeCost { + const o = record(v, where); + const pct = record(o["percentile"], `${where} percentile`); + const p = num(pct, "p", `${where} percentile`); + if (!(p > 50 && p <= 100)) invalid(`${where} percentile p ${p} is outside (50, 100]`); + const median = num(o, "median", where); + const value = num(pct, "value", `${where} percentile`); + if (median < 0 || value < 0) invalid(`${where} holds a negative cost`); + return { + median, + percentile: { p, value }, + blocks: posInt(o, "blocks", where), + warmupBlocks: count(o, "warmupBlocks", where), + }; +} + +function readRtSafety(v: unknown, where: string): ProbeRtSafety { + const o = record(v, where); + const events = o["events"] === undefined ? undefined : (stringList(o["events"]) ?? invalid(`${where} events is not a list of strings`)); + return { + interposer: bool(o, "interposer", where), + ...optional("swept", optBool(o, "swept", where)), + ...optional("sweepApplicable", optBool(o, "sweepApplicable", where)), + ...optional("blocks", optCount(o, "blocks", where)), + ...optional("allocations", optCount(o, "allocations", where)), + ...optional("locks", optCount(o, "locks", where)), + ...optional("syscalls", optCount(o, "syscalls", where)), + ...optional("threadViolations", optCount(o, "threadViolations", where)), + ...optional("processErrors", optCount(o, "processErrors", where)), + ...optional("events", events), + }; +} + +function readNoteSweep(v: unknown, where: string): ProbeNoteSweep { + const o = record(v, where); + return { + applicable: bool(o, "applicable", where), + ...optional("dialect", nonEmptyString(o["dialect"])), + ...optional("allocations", optCount(o, "allocations", where)), + ...optional("locks", optCount(o, "locks", where)), + ...optional("syscalls", optCount(o, "syscalls", where)), + }; +} + +function readPoint(v: unknown, i: number): ProbeGridPoint { + const where = `grid[${i}]`; + const o = record(v, where); + const latencyIn = o["latency"]; + let latency: ProbeGridPoint["latency"]; + if (latencyIn !== undefined) { + const l = record(latencyIn, `${where} latency`); + latency = { declaredFrames: count(l, "declaredFrames", `${where} latency`), measuredFrames: count(l, "measuredFrames", `${where} latency`) }; + } + return { + rate: posInt(o, "rate", where), + quantum: posInt(o, "quantum", where), + outcome: outcome(o, where), + ...optional("latency", latency), + ...(o["cost"] === undefined ? {} : { cost: readCost(o["cost"], `${where} cost`) }), + ...(o["rtSafety"] === undefined ? {} : { rtSafety: readRtSafety(o["rtSafety"], `${where} rtSafety`) }), + ...(o["noteSweep"] === undefined ? {} : { noteSweep: readNoteSweep(o["noteSweep"], `${where} noteSweep`) }), + }; +} + +function readStep(v: unknown, i: number): ProbeStep { + const where = `steps[${i}]`; + const o = record(v, where); + return { + step: str(o, "step", where), + outcome: outcome(o, where), + ...optional("rate", o["rate"] === undefined ? undefined : posInt(o, "rate", where)), + ...optional("quantum", o["quantum"] === undefined ? undefined : posInt(o, "quantum", where)), + ...optional("detail", nonEmptyString(o["detail"])), + }; +} + +function readIdle(v: unknown, i: number): ProbeIdleState { + const where = `machine.profile.cpuidle[${i}]`; + const o = record(v, where); + return { + cpu: count(o, "cpu", where), + state: str(o, "state", where), + latencyUs: count(o, "latencyUs", where), + disabled: bool(o, "disabled", where), + usageDelta: count(o, "usageDelta", where), + }; +} + +/** Whether any idle state deeper than {@link DEEP_IDLE_LATENCY_US} is enabled: then the machine does not limit its C-states. */ +export function deepIdleEnabled(cpuidle: readonly ProbeIdleState[]): boolean { + return cpuidle.some((s) => s.latencyUs > DEEP_IDLE_LATENCY_US && !s.disabled); +} + +/** Whether a deep idle state was entered while the run measured. */ +export function deepIdleEntered(cpuidle: readonly ProbeIdleState[]): boolean { + return cpuidle.some((s) => s.latencyUs > DEEP_IDLE_LATENCY_US && s.usageDelta > 0); +} + +const PROFILE_GOVERNOR = "probe.profile.governor"; +const PROFILE_DEEP_IDLE = "probe.profile.deep-idle-entered"; + +/** + * The two derived fields, recomputed. `cstatesLimited` is "idle states are read and none deeper than + * the limit is enabled", and it cannot hold when a deep state was entered. `meetsProtocol` is the + * performance governor, no deep state entered and nothing else unmet: the codes the reader can + * recompute (`probe.profile.governor`, `probe.profile.deep-idle-entered`) must be in `unmet`, and the + * flag is true exactly when `unmet` is empty. A code the reader cannot recompute (the pinning, the + * turbo and SMT readings) is taken as the runner wrote it. + */ +function checkDerived(machine: ProbeOutput["machine"]): void { + const idle = machine.profile.cpuidle; + const limited = idle.length > 0 && !deepIdleEnabled(idle); + const claimed = machine.class["cstatesLimited"]; + if (claimed !== limited) { + throw new Refusal( + PROBE_FORMAT_CODE.derivedMismatch, + `machine.class.cstatesLimited is ${String(claimed)} but cpuidle[] gives ${String(limited)}`, + ); + } + const { governor, unmet, meetsProtocol } = machine.profile; + const required = [...(governor === "performance" ? [] : [PROFILE_GOVERNOR]), ...(deepIdleEntered(idle) ? [PROFILE_DEEP_IDLE] : [])]; + const missing = required.filter((code) => !unmet.includes(code)); + if (missing.length > 0) { + throw new Refusal(PROBE_FORMAT_CODE.derivedMismatch, `machine.profile.unmet lacks ${missing.join(", ")}, which governor and cpuidle[] give`); + } + if (meetsProtocol !== (unmet.length === 0)) { + throw new Refusal( + PROBE_FORMAT_CODE.derivedMismatch, + `machine.profile.meetsProtocol is ${String(meetsProtocol)} but unmet[] holds ${unmet.length} code${unmet.length === 1 ? "" : "s"}`, + ); + } +} + +function readMachine(v: unknown): ProbeOutput["machine"] { + const o = record(v, "machine"); + const cls = record(o["class"], "machine.class"); + const klass: Record = {}; + for (const [k, value] of Object.entries(cls)) if (typeof value === "string" || typeof value === "boolean") klass[k] = value; + if (typeof klass["cstatesLimited"] !== "boolean") invalid("machine.class has no boolean cstatesLimited"); + const p = record(o["profile"], "machine.profile"); + const governor = p["governor"]; + if (!(governor === null || typeof governor === "string")) invalid("machine.profile.governor is not a string or null"); + const unmet = p["unmet"] === undefined ? [] : (stringList(p["unmet"]) ?? invalid("machine.profile.unmet is not a list of strings")); + return { + class: klass, + profile: { + governor, + cpuidle: list(p, "cpuidle", "machine.profile").map(readIdle), + meetsProtocol: bool(p, "meetsProtocol", "machine.profile"), + unmet, + }, + pinnedCpus: str(o, "pinnedCpus", "machine"), + }; +} + +function readBinary(v: unknown): ProbeOutput["binary"] { + const o = record(v, "binary"); + const files = list(o, "files", "binary").map((f, i) => { + const fo = record(f, `binary.files[${i}]`); + return { path: str(fo, "path", `binary.files[${i}]`), sha256: sha(fo, "sha256", `binary.files[${i}]`) }; + }); + const layoutIn = o["layout"]; + let layout: { scheme: string; sha256: string } | undefined; + if (layoutIn !== undefined) { + const lo = record(layoutIn, "binary.layout"); + layout = { scheme: str(lo, "scheme", "binary.layout"), sha256: sha(lo, "sha256", "binary.layout") }; + } + return { + path: str(o, "path", "binary"), + sha256: sha(o, "sha256", "binary"), + files, + ...optional("layout", layout), + ...optional("unpinnable", nonEmptyString(o["unpinnable"])), + }; +} + +function readRun(v: unknown): ProbeOutput["run"] { + const o = record(v, "run"); + const protocol = record(o["protocol"], "run.protocol"); + const toolsIn = record(o["tools"], "run.tools"); + const tools: Record = {}; + for (const [k, value] of Object.entries(toolsIn)) if (typeof value === "string") tools[k] = value; + return { + id: str(o, "id", "run"), + startedAt: str(o, "startedAt", "run"), + elapsedMs: count(o, "elapsedMs", "run"), + complete: bool(o, "complete", "run"), + outcome: outcome(o, "run"), + ...optional("reason", nonEmptyString(o["reason"])), + protocol: { name: str(protocol, "name", "run.protocol"), version: posInt(protocol, "version", "run.protocol") }, + tools, + via: str(o, "via", "run"), + }; +} + +function readPlugin(v: unknown): ProbeOutput["plugin"] { + const o = record(v, "plugin"); + const format = o["format"]; + if (format !== "clap" && format !== "lv2") invalid(`plugin.format ${JSON.stringify(format)} is neither clap nor lv2`); + return { + format, + id: str(o, "id", "plugin"), + ...optional("name", nonEmptyString(o["name"])), + ...optional("version", nonEmptyString(o["version"])), + }; +} + +function readOffer(v: unknown): ProbeOutput["offer"] { + const o = record(v, "offer"); + const ext = o["clapExtensions"] === undefined ? undefined : (stringList(o["clapExtensions"]) ?? invalid("offer.clapExtensions is not a list of strings")); + return { id: str(o, "id", "offer"), ...optional("clapExtensions", ext) }; +} + +/** The version `"."`, or `undefined` when it is not one. */ +function parseVersion(v: unknown): { major: number; minor: number } | undefined { + const m = typeof v === "string" ? /^(\d+)\.(\d+)$/.exec(v) : null; + return m === null ? undefined : { major: Number(m[1]), minor: Number(m[2]) }; +} + +/** + * A probe output from its text, or the code that refuses it. The order of the checks is the order of + * what can be trusted: that it is JSON and the format, then the major (before any content is read), + * then that it holds no judgment, then the content, then the derived fields against it. + */ +export function readProbeOutput(text: string): ReadProbeOutputResult { + try { + let json: unknown; + try { + json = JSON.parse(text); + } catch { + return invalid("not JSON"); + } + const doc = record(json, "the document"); + if (doc["format"] !== PROBE_OUTPUT_FORMAT) invalid(`format is ${JSON.stringify(doc["format"])}, not ${PROBE_OUTPUT_FORMAT}`); + const version = parseVersion(doc["formatVersion"]); + if (version === undefined) invalid(`formatVersion ${JSON.stringify(doc["formatVersion"])} is not "."`); + if (version.major !== PROBE_OUTPUT_MAJOR) { + throw new Refusal(PROBE_FORMAT_CODE.majorUnknown, `formatVersion ${String(doc["formatVersion"])}: this reader knows major ${PROBE_OUTPUT_MAJOR}`); + } + for (const key of JUDGMENT_KEYS) if (key in doc) invalid(`the document carries ${key}: a probe output holds facts, never a judgment`); + const grid = doc["grid"] === undefined ? [] : list(doc, "grid", "the document").map(readPoint); + const seen = new Set(); + for (const p of grid) { + const key = `${p.rate}/${p.quantum}`; + if (seen.has(key)) invalid(`grid holds rate ${p.rate} at quantum ${p.quantum} twice`); + seen.add(key); + } + const behaviour = doc["behaviour"]; + const output: ProbeOutput = { + formatVersion: String(doc["formatVersion"]), + run: readRun(doc["run"]), + plugin: readPlugin(doc["plugin"]), + binary: readBinary(doc["binary"]), + machine: readMachine(doc["machine"]), + offer: readOffer(doc["offer"]), + steps: list(doc, "steps", "the document").map(readStep), + grid, + ...(isRecord(behaviour) ? { behaviour } : {}), + }; + checkDerived(output.machine); + return { ok: true, output }; + } catch (e) { + if (e instanceof Refusal) return { ok: false, code: e.code, problem: e.message }; + throw e; + } +} + +/** The SHA-256 of some bytes, as the record names a probe output's copy. */ +export function sha256Hex(bytes: Uint8Array): string { + return createHash("sha256").update(bytes).digest("hex"); +} diff --git a/src/probe-runner.test.ts b/src/probe-runner.test.ts new file mode 100644 index 0000000..5628271 --- /dev/null +++ b/src/probe-runner.test.ts @@ -0,0 +1,318 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/** + * The runner as plugin-qualify drives it. The runner of plugin-hostd is not installed where the suite + * runs, so a stand-in answers: either a function that writes a document to the `--out` it is given, or a + * real script spawned the way the real runner is, to see the argument list, the environment and the + * exit statuses cross a process boundary. + */ +import { chmodSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { afterAll, describe, expect, it } from "vitest"; + +import { HOSTING_CODE } from "./hosting-codes.js"; +import { JALV_PROFILE, OMX_CLAP_HOST_CONFIGURED_EXTENSIONS, OMX_CLAP_HOST_PROFILE, OPENMIXER_CONSOLE_PROFILE } from "./host-profiles.js"; +import { ProbeRunnerError, clapRunnerArgs, lv2RunnerArgs, probeRunnerTool, probeTarget, qualifyTargets, type ProbeRun } from "./probe-runner.js"; +import { fixtureJson, fixtureText, clapLineFixture, probeOutputOfLine } from "./probe-output-fixtures.js"; +import { + CLAP_QUALIFY_TOOL_ENV, + CLAP_QUALIFY_TIMEOUT_S, + CLAP_SWEEP_BLOCKS, + LV2_HOSTING_TIMEOUT_S, + PROBE_RUNNER_ENV, + RT_INTERPOSE_ENV, +} from "./qualify-declarations.js"; + +const PKG = join(dirname(fileURLToPath(import.meta.url)), ".."); +const scratch = mkdtempSync(join(tmpdir(), "probe-runner-")); +afterAll(() => rmSync(scratch, { recursive: true, force: true })); + +const TOOLS = { [CLAP_QUALIFY_TOOL_ENV]: "/t/clap-qualify", [RT_INTERPOSE_ENV]: "/t/rt.so" }; +const exists = () => true; +const CLAP = { format: "clap", uri: "urn:clap:org.example.plate", binaryPath: "/p/example-plate.clap" } as const; +const LV2 = { format: "lv2", uri: "urn:example:zero-path" } as const; + +/** What a stand-in runner was called with. */ +interface Call { + readonly tool: string; + readonly args: readonly string[]; + readonly preload: string | undefined; +} + +/** A stand-in runner: writes `document(args)` to its `--out` and exits `status`. */ +function runner(document: ((args: readonly string[]) => string) | undefined, status = 0, extra: Partial> = {}): { run: ProbeRun; calls: Call[] } { + const calls: Call[] = []; + const run: ProbeRun = (tool, args, env) => { + calls.push({ tool, args, preload: env["LD_PRELOAD"] }); + if (document !== undefined) writeFileSync(args[args.indexOf("--out") + 1]!, document(args)); + return { status, signal: null, ...extra }; + }; + return { run, calls }; +} + +const successText = fixtureText("success-msi"); + +describe("the runner's arguments", () => { + it("a CLAP file is run by the prober the package locates, under the declared sweep and deadline", () => { + const { run, calls } = runner(() => successText); + probeTarget(CLAP, { rates: [44100, 48000], run, env: TOOLS, exists }); + expect(calls).toHaveLength(1); + const c = calls[0]!; + expect(c.tool).toBe("plugin-hostd-probe"); + expect(c.preload).toBe("/t/rt.so"); + const out = c.args[c.args.indexOf("--out") + 1]!; + const log = c.args[c.args.indexOf("--log") + 1]!; + expect(c.args).toEqual([ + "run", "--child", "/t/clap-qualify", "--rates", "44100,48000", "--quantum", "512", "--blocks", String(CLAP_SWEEP_BLOCKS), + "--wall-s", String(CLAP_QUALIFY_TIMEOUT_S), "--out", out, "--log", log, "/p/example-plate.clap", "org.example.plate", + ]); + expect(out).not.toBe(log); + }); + + it("the offered extensions travel as --offer, and only when there are some", () => { + const base = { rates: [48000], out: "o", log: "l", child: "c" }; + expect(clapRunnerArgs({ ...base, offer: ["clap.a", "clap.b"] }, "/x.clap", "id")).toContain("--offer"); + expect(clapRunnerArgs({ ...base, offer: ["clap.a", "clap.b"] }, "/x.clap", "id")[clapRunnerArgs({ ...base, offer: ["x"] }, "/x.clap", "id").indexOf("--offer") + 1]).toBe("clap.a,clap.b"); + expect(clapRunnerArgs({ ...base, offer: [] }, "/x.clap", "id")).not.toContain("--offer"); + expect(clapRunnerArgs(base, "/x.clap", undefined).at(-1)).toBe("/x.clap"); + }); + + it("an LV2 plugin is run as lv2: under its own deadline", () => { + const args = lv2RunnerArgs({ rates: [48000], out: "o", log: "l" }, "urn:example:zero-path"); + expect(args.at(-1)).toBe("lv2:urn:example:zero-path"); + expect(args[args.indexOf("--wall-s") + 1]).toBe(String(LV2_HOSTING_TIMEOUT_S)); + expect(args).not.toContain("--child"); + }); + + it("the runner is OMX_PROBE_RUNNER when set, else plugin-hostd-probe on the PATH", () => { + expect(probeRunnerTool({})).toBe("plugin-hostd-probe"); + expect(probeRunnerTool({ [PROBE_RUNNER_ENV]: "" })).toBe("plugin-hostd-probe"); + expect(probeRunnerTool({ [PROBE_RUNNER_ENV]: "/opt/phd/plugin-hostd-probe" })).toBe("/opt/phd/plugin-hostd-probe"); + const { run, calls } = runner(() => successText); + probeTarget(CLAP, { rates: [48000], run, env: { ...TOOLS, [PROBE_RUNNER_ENV]: "/opt/phd/plugin-hostd-probe" }, exists }); + expect(calls[0]?.tool).toBe("/opt/phd/plugin-hostd-probe"); + }); +}); + +describe("the runner's exit decides whether there is a document", () => { + const options = { rates: [48000], env: TOOLS, exists }; + + it("exit 0 and exit 1 both leave a document to read; the bytes are kept as written", () => { + for (const status of [0, 1]) { + const { run } = runner(() => successText, status); + const p = probeTarget(CLAP, { ...options, run }); + expect(p.bytes.toString("utf8")).toBe(successText); + expect(p.output.plugin.id).toBe("org.example.plate"); + expect(p.output.grid[0]?.cost?.median).toBe(41.2); + } + }); + + it("exit 2 is no document and says what the runner said", () => { + const { run } = runner(undefined, 2, { stderr: "plugin-hostd-probe: a required limit is unavailable: memory\n" }); + expect(() => probeTarget(CLAP, { ...options, run })).toThrow( + /exited 2 measuring urn:clap:org.example.plate, and wrote no document: plugin-hostd-probe: a required limit is unavailable: memory/, + ); + }); + + it("exit 3 is no such plugin", () => { + const { run } = runner(undefined, 3); + expect(() => probeTarget(CLAP, { ...options, run })).toThrow(/no such plugin urn:clap:org.example.plate/); + }); + + it("a runner that exits 0 and wrote nothing, or was killed, or is absent, is no measurement", () => { + expect(() => probeTarget(CLAP, { ...options, run: runner(undefined, 0).run })).toThrow(/wrote no document at the path/); + expect(() => probeTarget(CLAP, { ...options, run: runner(undefined, null as never, { signal: "SIGKILL" }).run })).toThrow(/killed \(SIGKILL\)/); + const missing: ProbeRun = () => ({ status: null, error: new Error("spawn plugin-hostd-probe ENOENT") }); + expect(() => probeTarget(CLAP, { ...options, run: missing })).toThrow(new RegExp(`install plugin-hostd or set ${PROBE_RUNNER_ENV}`)); + }); + + it("a document the reader refuses is reported with its code, never judged", () => { + const major2 = JSON.stringify({ ...fixtureJson("success-msi"), formatVersion: "2.0" }); + const { run } = runner(() => major2); + expect(() => probeTarget(CLAP, { ...options, run })).toThrow(/refused \(probe\.format\.major-unknown\)/); + const mismatch = fixtureJson("success-msi"); + mismatch.machine.class.cstatesLimited = false; + expect(() => probeTarget(CLAP, { ...options, run: runner(() => JSON.stringify(mismatch)).run })).toThrow(/probe\.format\.derived-mismatch/); + }); + + it("a document about another plugin than the one asked for is not judged as it", () => { + const { run } = runner(() => successText); + expect(() => probeTarget({ ...CLAP, uri: "urn:clap:org.example.other" }, { ...options, run })).toThrow( + /measured org.example.plate when asked for org.example.other/, + ); + }); + + it("a CLAP target with no file to load, and a prober that is not installed, are refused before the runner is started", () => { + const { run, calls } = runner(() => successText); + expect(() => probeTarget({ format: "clap", uri: "urn:clap:x" }, { ...options, run })).toThrow(ProbeRunnerError); + expect(() => probeTarget(CLAP, { rates: [48000], env: {}, exists: () => false, run })).toThrow(/no CLAP prober at/); + expect(calls).toEqual([]); + }); +}); + +describe("qualifyTargets: a pass over the targets", () => { + /** A runner that answers each run with a clean CLAP document taken with the extensions it was asked to offer. */ + function perOffer() { + return runner((args) => { + const at = args.indexOf("--offer"); + const offered = at < 0 ? [] : args[at + 1]!.split(","); + const base = clapLineFixture().hostExtensions; + return JSON.stringify(probeOutputOfLine(clapLineFixture({ hostExtensions: [...base, ...offered] }))); + }); + } + const target = { format: "clap", uri: "urn:clap:org.example.fx", binaryPath: "/p/fx.clap" } as const; + + it("runs the runner once per offer and judges each profile from its own run", () => { + const { run, calls } = perOffer(); + const pass = qualifyTargets([target], { + rates: [48000], + run, + env: TOOLS, + exists, + profiles: [OPENMIXER_CONSOLE_PROFILE, OMX_CLAP_HOST_PROFILE, JALV_PROFILE], + }); + expect(calls.map((c) => c.args.includes("--offer"))).toEqual([false, true]); + expect(calls[1]?.args[calls[1].args.indexOf("--offer") + 1]).toBe(OMX_CLAP_HOST_CONFIGURED_EXTENSIONS.join(",")); + expect(pass.verdicts.map((v) => [v.uri, v.host, v.rating, v.measured])).toEqual([ + ["urn:clap:org.example.fx", "openmixer-console", "suitable", true], + ["urn:clap:org.example.fx", "omx-clap-host", "suitable", true], + ["urn:clap:org.example.fx", "jalv", "unknown", true], + ]); + expect(pass.verdicts[2]?.deciding).toBe(HOSTING_CODE.clapFormatNotHosted); + expect(pass.failures).toEqual([]); + expect(pass.runs.map((r) => r.judgment.hosts.map((h) => h.verdict.host))).toEqual([["openmixer-console", "jalv"], ["omx-clap-host"]]); + expect(pass.documents.map((d) => d.console).sort()).toEqual(["jalv", "omx-clap-host", "openmixer-console"]); + }); + + it("a target the runner cannot measure reads unknown, unmeasured, under every profile, and the others go on", () => { + let n = 0; + const run: ProbeRun = (_tool, args) => { + if (n++ === 0) return { status: 2, stderr: "no limits" }; + writeFileSync(args[args.indexOf("--out") + 1]!, JSON.stringify(probeOutputOfLine(clapLineFixture()))); + return { status: 0 }; + }; + const pass = qualifyTargets([target, { ...target, uri: "urn:clap:org.example.fx", binaryPath: "/p/b.clap" }], { + rates: [48000], + run, + env: TOOLS, + exists, + profiles: [OPENMIXER_CONSOLE_PROFILE, JALV_PROFILE], + }); + expect(pass.failures).toEqual([{ uri: "urn:clap:org.example.fx", problem: expect.stringContaining("exited 2") }]); + expect(pass.verdicts.map((v) => [v.host, v.measured, v.rating, v.deciding])).toEqual([ + ["openmixer-console", false, "unknown", "not-measured"], + ["jalv", false, "unknown", "not-measured"], + ["openmixer-console", true, "suitable", undefined], + ["jalv", true, "unknown", HOSTING_CODE.clapFormatNotHosted], + ]); + }); + + it("a target whose second offer's run fails leaves no run and no document of its first one", () => { + const perOfferRun = perOffer().run; + let n = 0; + const run: ProbeRun = (tool, args, env) => (n++ === 1 ? { status: 2, stderr: "no limits" } : perOfferRun(tool, args, env)); + // the same plugin id in two files: the first one's second run fails, the second file measures whole + const pass = qualifyTargets([target, { ...target, binaryPath: "/p/b.clap" }], { + rates: [48000], + run, + env: TOOLS, + exists, + profiles: [OPENMIXER_CONSOLE_PROFILE, OMX_CLAP_HOST_PROFILE], + }); + expect(n).toBe(4); + expect(pass.failures).toEqual([{ uri: "urn:clap:org.example.fx", problem: expect.stringContaining("exited 2") }]); + expect(pass.verdicts.map((v) => [v.host, v.measured])).toEqual([ + ["openmixer-console", false], + ["omx-clap-host", false], + ["openmixer-console", true], + ["omx-clap-host", true], + ]); + // only the second file's two runs and two documents: the failed target's first run counts for nothing + expect(pass.runs.map((r) => r.judgment.hosts.map((h) => h.verdict.host))).toEqual([["openmixer-console"], ["omx-clap-host"]]); + expect(pass.documents.map((d) => d.console)).toEqual(["openmixer-console", "omx-clap-host"]); + }); + + it("an LV2 plugin is run once for every profile, through the runner, and judged on its counts", () => { + const doc = fixtureJson("success-msi"); + doc.plugin = { format: "lv2", id: LV2.uri }; + doc.grid[0].rtSafety.locks = 4; + const { run, calls } = runner(() => JSON.stringify(doc)); + const pass = qualifyTargets([LV2], { rates: [48000], run, env: {}, profiles: [OPENMIXER_CONSOLE_PROFILE, JALV_PROFILE] }); + expect(calls).toHaveLength(1); + expect(calls[0]?.args.at(-1)).toBe("lv2:urn:example:zero-path"); + expect(calls[0]?.preload).toBeUndefined(); + expect(pass.verdicts.map((v) => [v.uri, v.host, v.rating, v.deciding])).toEqual([ + [LV2.uri, "openmixer-console", "unsuitable", HOSTING_CODE.rtSafetyViolations], + [LV2.uri, "jalv", "unsuitable", HOSTING_CODE.rtSafetyViolations], + ]); + expect(pass.documents).toEqual([]); + }); + + it("a run that crashed is a verdict of unknown, unmeasured, with the run's outcome, and its document is kept", () => { + const { run } = runner(() => fixtureText("plugin-crash"), 1); + const pass = qualifyTargets([LV2], { rates: [48000], run, env: {}, profiles: [OPENMIXER_CONSOLE_PROFILE] }); + expect(pass.failures).toEqual([]); + expect(pass.verdicts).toEqual([{ uri: LV2.uri, host: "openmixer-console", measured: false, rating: "unknown", qualified: false, deciding: "probe.crashed" }]); + expect(pass.runs).toHaveLength(1); + }); +}); + +describe("a real process stands in for the runner", () => { + const script = join(scratch, "plugin-hostd-probe"); + writeFileSync( + script, + `#!/bin/sh +# records its arguments, writes $FIXTURE to the --out it is given, exits $EXITING +printf '%s\\n' "$@" > "$ARGS_FILE" +printf '%s' "$LD_PRELOAD" > "$ARGS_FILE.preload" +while [ $# -gt 0 ]; do + if [ "$1" = "--out" ]; then out="$2"; fi + shift +done +[ -n "$FIXTURE" ] && cp "$FIXTURE" "$out" +echo "stand-in said: $EXITING" >&2 +exit "\${EXITING:-0}" +`, + ); + chmodSync(script, 0o755); + const fixture = join(scratch, "doc.json"); + writeFileSync(fixture, successText); + const env = (extra: Record = {}) => ({ ...TOOLS, PATH: process.env["PATH"], [PROBE_RUNNER_ENV]: script, FIXTURE: fixture, ARGS_FILE: join(scratch, "args"), ...extra }); + + it("is started with the arguments, the interposer in LD_PRELOAD and no deadline of ours, and its document is read", () => { + const p = probeTarget(CLAP, { rates: [48000], env: env(), exists }); + expect(p.output.run.outcome).toBe("probe.ok"); + const args = readFileSync(join(scratch, "args"), "utf8").split("\n"); + expect(args.slice(0, 2)).toEqual(["run", "--child"]); + expect(args).toContain("org.example.plate"); + expect(readFileSync(join(scratch, "args.preload"), "utf8")).toBe("/t/rt.so"); + // the work directory is gone once the document is read + expect(existsSync(dirname(args[args.indexOf("--out") + 1]!))).toBe(false); + }); + + it("its exit status crosses the process boundary: 3 is no such plugin, 2 is no document, and 2 says why", () => { + expect(() => probeTarget(CLAP, { rates: [48000], env: env({ EXITING: "3", FIXTURE: "" }), exists })).toThrow(/no such plugin/); + expect(() => probeTarget(CLAP, { rates: [48000], env: env({ EXITING: "2", FIXTURE: "" }), exists })).toThrow(/exited 2[\s\S]*stand-in said: 2/); + expect(probeTarget(CLAP, { rates: [48000], env: env({ EXITING: "1" }), exists }).output.plugin.id).toBe("org.example.plate"); + }); + + it("an absent runner is the spawn failure, named by its setting", () => { + expect(() => probeTarget(CLAP, { rates: [48000], env: env({ [PROBE_RUNNER_ENV]: join(scratch, "nowhere") }), exists })).toThrow( + new RegExp(`set ${PROBE_RUNNER_ENV}`), + ); + }); +}); + +describe("no timeout of ours sits around a prober", () => { + const read = (rel: string) => readFileSync(join(PKG, "src", rel), "utf8"); + const TIMEOUT_OPTION = /timeout\s*:/; + + it("the scanner's spawn has one (positive control), the runner's and the CLAP arm's have none", () => { + expect(TIMEOUT_OPTION.test(read("clap-loader.ts"))).toBe(true); + expect(TIMEOUT_OPTION.test(read("probe-runner.ts"))).toBe(false); + expect(read("clap-qualify.ts")).not.toMatch(/spawnSync/); + expect(TIMEOUT_OPTION.test(read("direct-prober.support.ts"))).toBe(true); + }); +}); diff --git a/src/probe-runner.ts b/src/probe-runner.ts new file mode 100644 index 0000000..b94562b --- /dev/null +++ b/src/probe-runner.ts @@ -0,0 +1,244 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/** + * Measuring through plugin-hostd's probe runner (`plugin-hostd-probe run`). The runner starts the + * prober under limits (a wall clock that kills the whole process tree, CPU time, memory, files), + * and writes the probe output; plugin-qualify waits for the runner and reads the document. No + * timeout of ours sits around it: the runner owns the deadline, and a node timer that signalled only + * the direct child is exactly what the runner replaces. + * + * The runner's exit says whether there is a document: 0 and 1 write one (1 is a run whose steps did + * not all end well, which the document records), 2 is no document (usage, a required limit + * unavailable, an output that could not be written), 3 is no such plugin. + */ +import { spawnSync } from "node:child_process"; +import { existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { ClapQualifyError, clapQualifyTools, proberOffer, type RtSafetyDocument } from "./clap-qualify.js"; +import type { HostProfile } from "./host-profiles.js"; +import { OPENMIXER_CONSOLE_PROFILE } from "./host-profiles.js"; +import type { PluginTarget } from "./loader.js"; +import { judgeProbeOutput, judgmentVerdicts, type Judgment } from "./probe-judge.js"; +import { readProbeOutput, type ProbeOutput } from "./probe-output.js"; +import type { QualifyVerdict } from "./qualify-cli.js"; +import { + CLAP_QUALIFY_TIMEOUT_S, + CLAP_SWEEP_BLOCKS, + CLAP_SWEEP_FRAMES, + LV2_HOSTING_BLOCK_FRAMES, + LV2_HOSTING_TIMEOUT_S, + PROBE_RUNNER_DEFAULT, + PROBE_RUNNER_ENV, +} from "./qualify-declarations.js"; + +export class ProbeRunnerError extends Error { + constructor(message: string) { + super(message); + this.name = "ProbeRunnerError"; + } +} + +/** The runner: the declared setting when set, else the declared default (on `PATH`). */ +export function probeRunnerTool(env: Readonly>): string { + const named = env[PROBE_RUNNER_ENV]; + return named === undefined || named === "" ? PROBE_RUNNER_DEFAULT : named; +} + +/** What one run of the runner gave back; `error` is a spawn failure, `signal` a death. */ +export interface ProbeRunResult { + readonly status: number | null; + readonly signal?: string | null; + readonly stderr?: string; + readonly error?: Error; +} + +/** Runs the runner. It is given no time limit of its own, since the runner keeps the deadline. */ +export type ProbeRun = (tool: string, args: readonly string[], env: Readonly>) => ProbeRunResult; + +const spawnRunner: ProbeRun = (tool, args, env) => { + const r = spawnSync(tool, [...args], { encoding: "utf8", env: { ...env }, maxBuffer: 64 << 20 }); + return { status: r.status, signal: r.signal, stderr: r.stderr ?? "", ...(r.error === undefined ? {} : { error: r.error }) }; +}; + +export interface ProbeTargetOptions { + /** The rates the plugin is measured at. */ + readonly rates: readonly number[]; + /** Host extensions to offer beyond the prober's own (a CLAP run). */ + readonly offer?: readonly string[]; + readonly run?: ProbeRun; + readonly env?: Readonly>; + readonly exists?: (path: string) => boolean; +} + +/** The runner's arguments for a CLAP file; `child` is the prober the runner starts. */ +export function clapRunnerArgs( + options: { rates: readonly number[]; offer?: readonly string[]; out: string; log: string; child: string }, + binaryPath: string, + id: string | undefined, +): string[] { + return [ + "run", + "--child", + options.child, + "--rates", + options.rates.join(","), + "--quantum", + String(CLAP_SWEEP_FRAMES), + "--blocks", + String(CLAP_SWEEP_BLOCKS), + ...(options.offer === undefined || options.offer.length === 0 ? [] : ["--offer", options.offer.join(",")]), + "--wall-s", + String(CLAP_QUALIFY_TIMEOUT_S), + "--out", + options.out, + "--log", + options.log, + binaryPath, + ...(id === undefined ? [] : [id]), + ]; +} + +/** + * The runner's arguments for an LV2 plugin, named as `lv2:` and measured by the runner's own LV2 + * child. The runner of plugin-hostd 0.4.0 starts a CLAP prober only (the LV2 child, plugin-hostd#76, is in no release yet), so + * against it this run ends with the runner's own refusal, which {@link runOne} reports. + */ +export function lv2RunnerArgs(options: { rates: readonly number[]; out: string; log: string }, uri: string): string[] { + return [ + "run", + "--rates", + options.rates.join(","), + "--quantum", + String(LV2_HOSTING_BLOCK_FRAMES), + "--wall-s", + String(LV2_HOSTING_TIMEOUT_S), + "--out", + options.out, + "--log", + options.log, + `lv2:${uri}`, + ]; +} + +/** One probe output and the bytes it was read from (the bytes are what gets copied beside the record). */ +export interface ProbedTarget { + readonly bytes: Buffer; + readonly output: ProbeOutput; +} + +function runOne(target: PluginTarget, options: ProbeTargetOptions): ProbedTarget { + const env = options.env ?? process.env; + const tool = probeRunnerTool(env); + const work = mkdtempSync(join(tmpdir(), "plugin-qualify-probe-")); + const out = join(work, "probe.json"); + const log = join(work, "child.stderr.log"); + try { + let args: string[]; + let runEnv: Record = { ...env }; + let expectedId: string; + if (target.format === "clap") { + if (target.binaryPath === undefined) throw new ProbeRunnerError(`${target.uri}: no .clap to load; name the file or a directory holding it`); + expectedId = target.uri.startsWith("urn:clap:") ? target.uri.slice("urn:clap:".length) : target.uri; + const { tool: child, interposer } = clapQualifyTools(env, options.exists ?? existsSync); + args = clapRunnerArgs({ rates: options.rates, ...(options.offer === undefined ? {} : { offer: options.offer }), out, log, child }, target.binaryPath, expectedId); + runEnv = { ...env, LD_PRELOAD: interposer }; + } else { + expectedId = target.uri; + args = lv2RunnerArgs({ rates: options.rates, out, log }, target.uri); + } + const r = (options.run ?? spawnRunner)(tool, args, runEnv); + if (r.error !== undefined) { + throw new ProbeRunnerError(`cannot run the probe runner "${tool}" (${r.error.message}): install plugin-hostd or set ${PROBE_RUNNER_ENV} to plugin-hostd-probe`); + } + if (r.signal) throw new ProbeRunnerError(`${tool} was killed (${r.signal}) measuring ${target.uri}`); + const said = (r.stderr ?? "").trim(); + if (r.status === 3) throw new ProbeRunnerError(`${tool}: no such plugin ${target.uri}${said === "" ? "" : `: ${said}`}`); + if (r.status !== 0 && r.status !== 1) { + throw new ProbeRunnerError(`${tool} exited ${String(r.status)} measuring ${target.uri}, and wrote no document${said === "" ? "" : `: ${said}`}`); + } + let bytes: Buffer; + try { + bytes = readFileSync(out); + } catch { + throw new ProbeRunnerError(`${tool} exited ${String(r.status)} but wrote no document at the path it was given`); + } + const read = readProbeOutput(bytes.toString("utf8")); + if (!read.ok) throw new ProbeRunnerError(`${tool} wrote a document that is refused (${read.code}): ${read.problem}`); + if (read.output.plugin.id !== expectedId) { + throw new ProbeRunnerError(`${tool} measured ${read.output.plugin.id} when asked for ${expectedId}: not judged as the plugin asked for`); + } + return { bytes, output: read.output }; + } finally { + rmSync(work, { recursive: true, force: true }); + } +} + +/** The probe output of one target, through the runner. Throws {@link ProbeRunnerError} when there is none to judge. */ +export function probeTarget(target: PluginTarget, options: ProbeTargetOptions): ProbedTarget { + return runOne(target, options); +} + +/** What the pass over a run's targets produced. */ +export interface ProbePass { + /** Every probe output that was judged, with its bytes, for the qualification directory. */ + readonly runs: readonly { readonly bytes: Buffer; readonly judgment: Judgment }[]; + readonly verdicts: readonly QualifyVerdict[]; + /** The CLAP documents of today's judge, which the pins file is taken from. */ + readonly documents: readonly RtSafetyDocument[]; + /** Each target that could not be measured, with why; its verdict rows read `unknown`, unmeasured. */ + readonly failures: readonly { readonly uri: string; readonly problem: string }[]; +} + +/** + * Measure every target through the runner, one run per target and set of offered extensions (a crash + * takes only that plugin), and judge each probe output under the profiles whose offer it was taken + * with. A CLAP profile the prober cannot stand in for is judged from the run with no offer and reads + * `unknown` there, as before. An LV2 plugin has one run for every profile. + */ +export function qualifyTargets( + targets: readonly PluginTarget[], + options: ProbeTargetOptions & { readonly profiles?: readonly HostProfile[] }, +): ProbePass { + const profiles = options.profiles ?? [OPENMIXER_CONSOLE_PROFILE]; + const runs: { bytes: Buffer; judgment: Judgment }[] = []; + const verdicts: QualifyVerdict[] = []; + const documents: RtSafetyDocument[] = []; + const failures: { uri: string; problem: string }[] = []; + for (const target of targets) { + const byOffer = new Map(); + for (const profile of profiles) { + const offer = target.format === "clap" ? (proberOffer(profile) ?? []) : []; + const key = offer.join(","); + const entry = byOffer.get(key) ?? { offer, profiles: [] }; + entry.profiles.push(profile); + byOffer.set(key, entry); + } + const judged = new Map(); + // a target's runs and documents count only when every one of its runs gave a probe output: one that + // failed leaves the target not measured, and then nothing of it is recorded or pinned + const targetRuns: { bytes: Buffer; judgment: Judgment }[] = []; + try { + for (const { offer, profiles: forOffer } of byOffer.values()) { + const probed = probeTarget(target, { ...options, offer }); + const judgment = judgeProbeOutput(probed.output, forOffer); + targetRuns.push({ bytes: probed.bytes, judgment }); + for (const v of judgmentVerdicts(judgment)) judged.set(v.host, v); + } + } catch (e) { + if (!(e instanceof ProbeRunnerError) && !(e instanceof ClapQualifyError)) throw e; + failures.push({ uri: target.uri, problem: e.message }); + for (const profile of profiles) { + verdicts.push({ uri: target.uri, host: profile.id, measured: false, rating: "unknown", qualified: false, deciding: "not-measured" }); + } + continue; + } + for (const r of targetRuns) { + runs.push(r); + documents.push(...r.judgment.documents); + } + for (const profile of profiles) verdicts.push(judged.get(profile.id)!); + } + return { runs, verdicts, documents, failures }; +} diff --git a/src/qualification-record.test.ts b/src/qualification-record.test.ts new file mode 100644 index 0000000..dcd90ea --- /dev/null +++ b/src/qualification-record.test.ts @@ -0,0 +1,120 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +import { createHash } from "node:crypto"; +import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterAll, describe, expect, it } from "vitest"; + +import { OMX_CLAP_HOST_PROFILE, OPENMIXER_CONSOLE_PROFILE } from "./host-profiles.js"; +import { judgeProbeOutput } from "./probe-judge.js"; +import { readProbeOutput } from "./probe-output.js"; +import { fixtureText } from "./probe-output-fixtures.js"; +import { QUALIFICATION_RECORD_SCHEMA, probeCopyFileName, qualificationBaseName, recordFileName, writeQualification } from "./qualification-record.js"; + +const scratch = mkdtempSync(join(tmpdir(), "qualification-record-")); +afterAll(() => rmSync(scratch, { recursive: true, force: true })); + +const sha = (bytes: Buffer | string): string => createHash("sha256").update(bytes).digest("hex"); + +function judge(text: string) { + const r = readProbeOutput(text); + if (!r.ok) throw new Error(r.problem); + return judgeProbeOutput(r.output, [OMX_CLAP_HOST_PROFILE, OPENMIXER_CONSOLE_PROFILE]); +} + +describe("the qualification record and the probe output it judged", () => { + it("names the copy it wrote and the SHA-256 of that copy's bytes", () => { + const dir = join(scratch, "a"); + const text = fixtureText("success-msi"); + const w = writeQualification(dir, Buffer.from(text), judge(text)); + expect(readdirSync(dir).sort()).toEqual([ + "org.example.plate@1.4.2-356902794a4e.omx-clap-host.probe.json", + "org.example.plate@1.4.2-356902794a4e.omx-clap-host.qualification.json", + ]); + const onDisk = readFileSync(w.probePath); + expect(onDisk.toString("utf8")).toBe(text); + expect(w.record.probe.file).toBe("org.example.plate@1.4.2-356902794a4e.omx-clap-host.probe.json"); + expect(w.record.probe.sha256).toBe(sha(onDisk)); + expect(w.record.probe.sha256).toBe(sha(text)); + expect(w.record.probe.bytes).toBe(Buffer.byteLength(text)); + const read = JSON.parse(readFileSync(w.recordPath, "utf8")); + expect(read).toEqual(JSON.parse(JSON.stringify(w.record))); + }); + + it("holds the verdict per host, the cost beside it and the figures as measured", () => { + const dir = join(scratch, "c"); + const text = fixtureText("success-msi"); + const { record } = writeQualification(dir, Buffer.from(text), judge(text)); + expect(record.schema).toBe(QUALIFICATION_RECORD_SCHEMA); + expect(record.plugin).toEqual({ format: "clap", id: "org.example.plate", name: "Plate" }); + expect(record.testedVersion).toBe("1.4.2"); + expect(record.verdicts.map((v) => [v.host, v.rating, v.qualified, v.hosting, v.deciding])).toEqual([ + ["omx-clap-host", "suitable", true, "isolated", undefined], + ["openmixer-console", "unknown", false, "isolated", "hosting.clap.extensions-not-covered"], + ]); + expect(record.cost.map((c) => [c.host, c.rate, c.quantum, c.reasons.map((r) => r.code)])).toEqual([ + ["omx-clap-host", 48000, 128, ["hosting.cost.within-ceiling"]], + ["openmixer-console", 96000, 1024, ["hosting.cost.unmeasured-at-rate-and-quantum"]], + ]); + expect(record.measurements.grid.map((p) => [p.rate, p.quantum, p.cost?.median])).toEqual([ + [48000, 128, 41.2], + [96000, 128, 40.7], + ]); + expect(record.measuredAt).toBe("2026-10-17T22:15:00+02:00"); + expect(record.tool["runner"]).toBe("plugin-hostd-probe 0.1.0"); + expect(record.probe).toMatchObject({ + protocol: { name: "plugin-hostd.probe-protocol", version: 1 }, + reason: "reference", + outcome: "probe.ok", + complete: true, + meetsProtocol: true, + }); + expect(record.probe.machineClass).toMatchObject({ arch: "x86_64", cstatesLimited: true }); + expect(record.binary.sha256).toBe("356902794a4eec2122989e072271ef995968131145db918d9a3084ce582f244b"); + }); + + it("keeps a run that crashed: unknown, the outcome as the reason, and the copy that shows why", () => { + const dir = join(scratch, "d"); + const text = fixtureText("plugin-crash"); + const w = writeQualification(dir, Buffer.from(text), judge(text)); + expect(w.record.verdicts[0]).toMatchObject({ measured: false, rating: "unknown", qualified: false, deciding: "probe.crashed" }); + expect(w.record.verdicts[0]?.reasons).toEqual([{ dimension: "rtSafety", rating: "unknown", code: "probe.crashed", params: {} }]); + expect(w.record.probe).toMatchObject({ outcome: "probe.crashed", complete: false, meetsProtocol: false }); + expect(readFileSync(w.probePath, "utf8")).toBe(text); + }); + + it("replaces the pair on a rescan of the same binary and offer, leaving no half-written file", () => { + const dir = join(scratch, "e"); + const text = fixtureText("success-msi"); + writeQualification(dir, Buffer.from(text), judge(text)); + const second = JSON.parse(text); + second.run.id = "msi-again"; + const again = JSON.stringify(second); + const w = writeQualification(dir, Buffer.from(again), judge(again)); + expect(readFileSync(w.probePath, "utf8")).toBe(again); + expect(w.record.probe.sha256).toBe(sha(again)); + expect(readdirSync(dir)).toHaveLength(2); + expect(existsSync(`${w.recordPath}.new`)).toBe(false); + }); + + it("names files by id, version, binary hash and offer, with nothing a path could use", () => { + const text = fixtureText("success-msi"); + const r = readProbeOutput(text); + if (!r.ok) throw new Error(r.problem); + expect(qualificationBaseName(r.output)).toBe("org.example.plate@1.4.2-356902794a4e.omx-clap-host"); + const lv2 = JSON.parse(fixtureText("plugin-crash")); + const o = readProbeOutput(JSON.stringify(lv2)); + if (!o.ok) throw new Error(o.problem); + expect(recordFileName(o.output)).toBe(`urn_example_zero-path@0.0-${o.output.binary.sha256.slice(0, 12)}.omx-console-lv2.qualification.json`); + expect(probeCopyFileName(o.output)).toMatch(/^urn_example_zero-path@0\.0-[0-9a-f]{12}\.omx-console-lv2\.probe\.json$/); + delete lv2.plugin.version; + const bare = readProbeOutput(JSON.stringify(lv2)); + if (!bare.ok) throw new Error(bare.problem); + expect(qualificationBaseName(bare.output)).toContain("@unversioned-"); + lv2.plugin.id = "../../etc/x"; + const evil = readProbeOutput(JSON.stringify(lv2)); + if (!evil.ok) throw new Error(evil.problem); + expect(qualificationBaseName(evil.output)).not.toContain("/"); + }); +}); diff --git a/src/qualification-record.ts b/src/qualification-record.ts new file mode 100644 index 0000000..22b5de5 --- /dev/null +++ b/src/qualification-record.ts @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Copyright (C) 2026 Pau Aliagas +/** + * The qualification record: what plugin-qualify wrote down about one plugin binary and one offer + * after judging its probe output, kept in `qualification_dir`. Evidence, not a constraint: no + * reader refuses a descriptor because of it. + * + * Two files, always together. The record is the judgment; beside it is a byte copy of the probe + * output it judged, because the probe directory is runtime and may be wiped. The record names the + * copy and the SHA-256 of the copy's bytes, so the facts outlive the run without hostd keeping a + * second store, and a reader can tell that the figures it holds are the ones the verdict came from. + * + * Not yet in the record: the fingerprint of the descriptor's controls, since the descriptor is + * written by the scan (stage 5) and plugin-qualify does not read one yet. + */ +import { mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +import type { Judgment, RecordReason } from "./probe-judge.js"; +import { sha256Hex, type ProbeGridPoint, type ProbeOutput } from "./probe-output.js"; + +export const QUALIFICATION_RECORD_SCHEMA = "plugin-qualify.qualification-record/1"; + +/** One host's verdict in the record. */ +export interface RecordVerdict { + readonly host: string; + /** False when the run did not finish: the rating is `unknown` and the reason is the run's outcome. */ + readonly measured: boolean; + readonly rating: string; + readonly qualified: boolean; + readonly hosting: "in-process" | "isolated"; + /** The code that decided it; absent for `suitable`. */ + readonly deciding?: string; + readonly reasons: readonly RecordReason[]; +} + +export interface QualificationRecord { + readonly schema: typeof QUALIFICATION_RECORD_SCHEMA; + readonly plugin: { readonly format: string; readonly id: string; readonly name?: string }; + /** The plugin's version when it was tested; a newer installed one is shown as "tested on X, installed Y". */ + readonly testedVersion?: string; + readonly binary: ProbeOutput["binary"]; + readonly verdicts: readonly RecordVerdict[]; + /** The cost reasons at each host's rate and quantum, beside the verdicts. */ + readonly cost: readonly { readonly host: string; readonly rate: number; readonly quantum: number; readonly reasons: readonly RecordReason[] }[]; + /** The figures as measured, per rate and quantum. */ + readonly measurements: { readonly grid: readonly ProbeGridPoint[]; readonly behaviour?: Readonly> }; + /** Who measured and when. */ + readonly tool: Readonly>; + readonly measuredAt: string; + /** The probe output this record judged. */ + readonly probe: { + readonly file: string; + readonly sha256: string; + readonly bytes: number; + readonly protocol: { readonly name: string; readonly version: number }; + readonly reason?: string; + readonly outcome: string; + readonly complete: boolean; + readonly machineClass: Readonly>; + /** Whether the machine met the protocol's optimised profile: a figure that did not is not a reference figure. */ + readonly meetsProtocol: boolean; + }; +} + +const safe = (text: string): string => text.replace(/[^A-Za-z0-9._-]/g, "_"); + +/** `@-.`: one binary and one offer, whatever the run. */ +export function qualificationBaseName(output: ProbeOutput): string { + return `${safe(output.plugin.id)}@${safe(output.plugin.version ?? "unversioned")}-${output.binary.sha256.slice(0, 12)}.${safe(output.offer.id)}`; +} + +export const recordFileName = (output: ProbeOutput): string => `${qualificationBaseName(output)}.qualification.json`; +export const probeCopyFileName = (output: ProbeOutput): string => `${qualificationBaseName(output)}.probe.json`; + +export function buildQualificationRecord(judgment: Judgment, copy: { file: string; sha256: string; bytes: number }): QualificationRecord { + const o = judgment.output; + return { + schema: QUALIFICATION_RECORD_SCHEMA, + plugin: { format: o.plugin.format, id: o.plugin.id, ...(o.plugin.name === undefined ? {} : { name: o.plugin.name }) }, + ...(o.plugin.version === undefined ? {} : { testedVersion: o.plugin.version }), + binary: o.binary, + verdicts: judgment.hosts.map((h) => ({ + host: h.verdict.host, + measured: h.verdict.measured, + rating: h.verdict.rating, + qualified: h.verdict.qualified, + hosting: h.hosting, + ...(h.verdict.deciding === undefined ? {} : { deciding: h.verdict.deciding }), + reasons: h.reasons, + })), + cost: judgment.hosts.map((h) => ({ host: h.verdict.host, ...h.cost })), + measurements: { grid: o.grid, ...(o.behaviour === undefined ? {} : { behaviour: o.behaviour }) }, + tool: o.run.tools, + measuredAt: o.run.startedAt, + probe: { + file: copy.file, + sha256: copy.sha256, + bytes: copy.bytes, + protocol: o.run.protocol, + ...(o.run.reason === undefined ? {} : { reason: o.run.reason }), + outcome: o.run.outcome, + complete: o.run.complete, + machineClass: o.machine.class, + meetsProtocol: o.machine.profile.meetsProtocol, + }, + }; +} + +/** The record of a judgment, with the copy fields taken from the bytes that were judged. */ +export function qualificationRecordOf(probeBytes: Uint8Array, judgment: Judgment): QualificationRecord { + return buildQualificationRecord(judgment, { file: probeCopyFileName(judgment.output), sha256: sha256Hex(probeBytes), bytes: probeBytes.length }); +} + +/** A file whole or not at all: written beside its name and renamed over it. */ +function writeWhole(path: string, bytes: string | Uint8Array): void { + const pending = `${path}.new`; + writeFileSync(pending, bytes); + renameSync(pending, path); +} + +export interface WrittenQualification { + readonly recordPath: string; + readonly probePath: string; + readonly record: QualificationRecord; +} + +/** + * Write the byte copy of the probe output, then the record that names it. The copy goes first, and the + * record names the SHA-256 of the copy as it is on disk, read back, so a record never names bytes that + * are not there. A rescan of the same binary and offer replaces both. + */ +export function writeQualification(dir: string, probeBytes: Uint8Array, judgment: Judgment): WrittenQualification { + mkdirSync(dir, { recursive: true }); + const probePath = join(dir, probeCopyFileName(judgment.output)); + writeWhole(probePath, probeBytes); + const copy = readFileSync(probePath); + const record = buildQualificationRecord(judgment, { file: probeCopyFileName(judgment.output), sha256: sha256Hex(copy), bytes: copy.length }); + const recordPath = join(dir, recordFileName(judgment.output)); + writeWhole(recordPath, `${JSON.stringify(record, null, 2)}\n`); + return { recordPath, probePath, record }; +} diff --git a/src/qualify-cli.ts b/src/qualify-cli.ts index c56fcc5..a06fa8f 100644 --- a/src/qualify-cli.ts +++ b/src/qualify-cli.ts @@ -118,7 +118,7 @@ export function helpText(): string { " plugin-qualify ... exactly the named plugins; a .lv2 bundle = every plugin in it", " plugin-qualify --all every CLAP plugin on CLAP_PATH, then ~/.clap, then /usr/lib/clap;", " refuses (exit 2) when LV2 bundles are on the LV2 path (LV2_PATH),", - " which this entry point cannot measure", + " whose plugins --all does not enumerate: name them instead", "", ...rows, "", diff --git a/src/qualify-declarations.ts b/src/qualify-declarations.ts index 311a272..6781324 100644 --- a/src/qualify-declarations.ts +++ b/src/qualify-declarations.ts @@ -80,6 +80,14 @@ export const CLAP_PATH_ENV = "CLAP_PATH"; export const CLAP_USER_DIR = ".clap"; export const CLAP_SYSTEM_DIR = "/usr/lib/clap"; +/** + * The ONE setting that locates plugin-hostd's probe runner: the environment variable that names the + * `plugin-hostd-probe` binary, and what is run when it is unset (looked up on `PATH`). The runner is + * plugin-hostd's, not a tool of this package, so it is not one of `NATIVE_TOOLS`. + */ +export const PROBE_RUNNER_ENV = "OMX_PROBE_RUNNER"; +export const PROBE_RUNNER_DEFAULT = "plugin-hostd-probe"; + /** Seconds one run of the CLAP scanner may take over everything it was pointed at. */ export const CLAP_SCAN_TIMEOUT_S = 300; diff --git a/src/rate-quantum-literal-scan.test.ts b/src/rate-quantum-literal-scan.test.ts index c9602d2..ed65ecf 100644 --- a/src/rate-quantum-literal-scan.test.ts +++ b/src/rate-quantum-literal-scan.test.ts @@ -31,6 +31,7 @@ const ALLOWED: Record = { "tools/benchmark.py": "every rate, quantum and timeout imports from the generated tools/_declarations.py; the literals left are docstring worked examples (20e-9 * 96000) and the --rate usage example", "src/measurement/fixtures.ts": "round-trip document fixtures for the measurement-format tests: sample data, not a rate or quantum claim", + "src/probe-output-fixtures.ts": "the probe-output documents the tests build on: sample data in plugin-hostd's example documents, not a rate or quantum claim", "src/measurement/format.ts": "a docstring example of the per-rate key shape (\"96000\"), not a rate claim", "tools/lv2-measure.mjs": "a --rates usage example in help text, not a declaration", "tools/_declarations.py": "generated by tools/gen-python-declarations.mjs from src/qualify-declarations.ts and checked byte-identical", diff --git a/tsconfig.json b/tsconfig.json index a3399b0..5451436 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -15,5 +15,5 @@ "outDir": "dist" }, "include": ["src"], - "exclude": ["**/*.test.ts", "**/__fixtures__/**"] + "exclude": ["**/*.test.ts", "**/*.support.ts", "**/__fixtures__/**"] }