diff --git a/README.md b/README.md index bc59628..d95fd49 100644 --- a/README.md +++ b/README.md @@ -272,3 +272,18 @@ Deletion/pruning preview by default and require `--apply` to unlink records. No automatic history, browser opening, rescanning, deletion, telemetry or Cloud sync. [The history guide](docs/SCANNER_HISTORY.md) documents storage, bounds, strict parsing, privacy, original evidence, age/keep-last selection and filesystem/interruption risks. + +### Local comparison — unpublished Phase C candidate + +On this development branch, compare two explicitly saved local scan IDs without +rescanning or modifying records: + +```sh +node dist/index.js history compare BEFORE_UUID AFTER_UUID --directory ./my-history --html ./new-comparison.html +``` + +Terminal and `--json` comparison are also available. Source movement, incomplete +coverage and Scanner/catalog differences remain explicit uncertainties. Disappearing +findings do not prove a fix; source changes are manual or external. See the +[matching and safety guide](docs/SCANNER_HISTORY.md#local-comparison--unpublished-phase-c-candidate). +Phase C is a local review candidate, not advertised as merged or released. diff --git a/docs/SCANNER_HISTORY.md b/docs/SCANNER_HISTORY.md index 8b15452..cf92917 100644 --- a/docs/SCANNER_HISTORY.md +++ b/docs/SCANNER_HISTORY.md @@ -147,3 +147,62 @@ Local history is a free manual evidence feature. It adds no continuous monitorin refreshed provider intelligence, advanced impact analysis, repair, customer-patch verification, GitHub repair PR, hosted/team sync, account or billing capability. Cloud remains planned and unavailable. Phase C comparison requires separate work. + +## Local comparison — unpublished Phase C candidate + +This development branch adds comparison for review; Phase C is not advertised as +merged or publicly available. Compare two explicit saved UUIDs from the same +initialized history directory: + +```sh +node dist/index.js history compare BEFORE_UUID AFTER_UUID --directory ./my-history +node dist/index.js history compare BEFORE_UUID AFTER_UUID --directory ./my-history --json +node dist/index.js history compare BEFORE_UUID AFTER_UUID --directory ./my-history --html ./new-comparison.html +``` + +Both records must pass the existing strict schema-1 validator and belong to the +same project UUID. No legacy raw JSON or HTML import, cross-project override, +implicit latest selection or target lookup is supported. Argument position defines +before/after even if clocks are equal or reversed; warnings preserve that distinction. +The existing private, bounded, identity-checked repository reader is reused. Neither +record is modified, and comparison does not acquire a mutation lock, rescan source, +read today's catalog, execute code, open a browser or access the network. + +Independent comparison schema 1 contains original Scanner/catalog metadata, +project/scan IDs, scope, coverage, findings and privacy-filtered saved references. +Integrations match on provider/package. Assessments match on provider/package, +change ID, scope/category and a unique saved client reference corroborated by its +original definition file/line and manifest. Explicit integration-level fallback +matches only when no client is recorded. Multiple candidates are inconclusive; +moved/renamed/withheld client scope is never guessed. Historical record fingerprints +must agree for like-for-like status transitions. Finding references remain evidence, +not persistent source identities or proof of authenticity. + +Categories describe newly detected, no longer detected and unchanged integrations; +newly recorded, no longer recorded, changed-status, unchanged-status and inconclusive +assessments. Increased/reduced uncertainty describes transitions into/out of Unknown +only. Unchanged integration identity can still have different declarations or source +references. Evidence comparison retains declaration row associations and sorts block +representations/notes; order changes within a block may conservatively count as an +evidence difference. No overall health score or runtime outcome is computed. + +Scanner version/build, catalog revision/fingerprint/provider coverage, effective +budgets and supported coverage are compared separately. Missing metadata, truncation, +partial scope, incompatible context and ambiguous/unmatched assessments are explicit +limitations. Observed statuses remain visible without recalculation even when causal +attribution is inconclusive. Hidden raw API declaration values and other omitted +privacy details cannot be reconstructed: equal displayed context is not proof of +identical runtime inputs. A missing finding never means fixed, resolved or repaired. +Any source changes are manual or external; comparison performs no repair. + +Terminal output provides summaries; JSON and collapsed dashboard evidence retain the +full compared privacy-filtered references. HTML uses the Phase A exclusive new-file +writer, escaping and static hashed-CSS CSP; there are no scripts or external assets. +Output is capped at 32 MiB and no truncation silently discards results. Two input +records retain the existing 32 MiB/100,000-node/depth-64 limits each. Matching uses +bounded maps and sorted keys, not pairwise fuzzy matching. Validation/parsing is not +a hard CPU/heap sandbox. The two reads are sequential, not an atomic directory +snapshot; retain immutable records in a stable private directory. Portable filesystem +race limits, privacy-sensitive relative labels and unauthenticated records remain. +Invalid/missing/unsafe input and output errors return 1; valid inconclusive comparisons +return 0 with warnings, not a compatibility verdict. No paid Cloud features are added. diff --git a/scripts/verify-pilot-package.mjs b/scripts/verify-pilot-package.mjs index aa7fb98..7c9d9f3 100644 --- a/scripts/verify-pilot-package.mjs +++ b/scripts/verify-pilot-package.mjs @@ -95,6 +95,21 @@ registerHooks({ resolve(specifier, context, nextResolve) { const saved = JSON.parse(run(['history', 'list', '--directory', history, '--json', '--html', resolve(work, 'history-list.html')])); assert.equal(saved.records.length, 2); const first = saved.records[0].scanId; + const beforeBytes = await readFile(resolve(history, `${saved.records[1].scanId}.json`)); + const afterBytes = await readFile(resolve(history, `${first}.json`)); + const compareArgs = ['history', 'compare', saved.records[1].scanId, first, '--directory', history, '--json', '--html', resolve(work, 'comparison.html')]; + const comparison = JSON.parse(run(compareArgs)); + assert.equal(comparison.schemaVersion, 1); + assert.ok(comparison.assessments.every((item) => item.category === 'unchanged')); + assert.ok(!JSON.stringify(comparison).includes(fixture)); + const comparisonHtml = await readFile(resolve(work, 'comparison.html'), 'utf8'); + assert.match(comparisonHtml, /Scan Comparison/); + assert.ok(!/<(?:script|img|iframe|form|link)\b/i.test(comparisonHtml)); + assert.equal(spawnSync(process.execPath, [...args, cli, ...compareArgs], settings).status, 1); + assert.equal(await readFile(resolve(work, 'comparison.html'), 'utf8'), comparisonHtml); + assert.deepEqual(await readFile(resolve(history, `${saved.records[1].scanId}.json`)), beforeBytes); + assert.deepEqual(await readFile(resolve(history, `${first}.json`)), afterBytes); + const reopened = JSON.parse(run(['history', 'show', first, '--directory', history, '--json', '--html', resolve(work, 'historical.html')])); assert.ok(!JSON.stringify(reopened).includes(fixture)); assert.ok(!JSON.stringify(reopened).includes('SYNTHETIC_PRIVATE_SENTINEL')); @@ -110,6 +125,6 @@ registerHooks({ resolve(specifier, context, nextResolve) { assert.equal(changes.catalogCoverage.liveMonitoring, false); const invalid = spawnSync(process.execPath, [...args, cli, 'scan', '../outside', '--json'], settings); assert.equal(invalid.status, 1); - console.log(JSON.stringify({ archive, sha256: digest, installation: 'passed: empty prefix/cache, offline, scripts disabled', runtime: 'self-contained; parent dependencies blocked', network: 'common Node transports denied; help/terminal/JSON/HTML/catalog/history passed', html: 'privacy/CSP/JSON compatibility/no-overwrite passed', history: 'init/two saves/list/reopen/delete preview/apply/prune preview passed', executable: 'passed', targetUnchanged: true, ignoredEnv: true, traversalRejected: true, files: expected, node: process.version }, null, 2)); + console.log(JSON.stringify({ archive, sha256: digest, installation: 'passed: empty prefix/cache, offline, scripts disabled', runtime: 'self-contained; parent dependencies blocked', network: 'common Node transports denied; help/terminal/JSON/HTML/catalog/history passed', html: 'privacy/CSP/JSON compatibility/no-overwrite passed', history: 'init/two saves/list/reopen/compare with immutable inputs and no-overwrite/delete preview/apply/prune preview passed', executable: 'passed', targetUnchanged: true, ignoredEnv: true, traversalRejected: true, files: expected, node: process.version }, null, 2)); } finally { await rm(work, { recursive: true, force: true }); } assert.equal(root, process.cwd()); diff --git a/src/cli/history.ts b/src/cli/history.ts index 46f722d..a8b0146 100644 --- a/src/cli/history.ts +++ b/src/cli/history.ts @@ -1,3 +1,5 @@ +import { compareSavedScans } from '../comparison/index.ts'; +import { formatComparisonTerminal, formatComparisonJson, formatComparisonHtml } from '../reports/comparison.ts'; import type { CliOutput } from './run.ts'; import { LocalScanHistoryRepository } from '../history/repository.ts'; import { formatHistoryTerminal, formatHistoryHtml, formatSavedTerminal, historySummary } from '../reports/history.ts'; @@ -11,6 +13,7 @@ history init --directory DIR [--json] scan TARGET --history-dir DIR --project-id UUID [--json] [--html NEW.html] history list --directory DIR [--project-id UUID] [--json] [--html NEW.html] history show SCAN_UUID --directory DIR [--json] [--html NEW.html] +history compare BEFORE_UUID AFTER_UUID --directory DIR [--json] [--html NEW.html] history delete SCAN_UUID --directory DIR [--apply] [--json] history prune --directory DIR --project-id UUID [--keep-last N] [--older-than-days N] [--apply] [--json] @@ -22,8 +25,8 @@ Records and reports can retain sensitive root-relative labels; inspect privately export async function runHistory(args: string[], output: CliOutput): Promise { const command = args[1]; - if (!command || !['init', 'list', 'show', 'delete', 'prune'].includes(command)) throw new Error('Unknown history command. Use patchwrk history --help.'); - const values = new Map(); let scanId: string | undefined; let json = false; let apply = false; + if (!command || !['init', 'list', 'show', 'delete', 'prune', 'compare'].includes(command)) throw new Error('Unknown history command. Use patchwrk history --help.'); + const values = new Map(); let scanId: string | undefined; let afterId: string | undefined; let json = false; let apply = false; const paths = new Set(['--directory', '--project-id', '--html', '--keep-last', '--older-than-days']); for (let index = 2; index < args.length; index++) { const arg = args[index]!; @@ -31,16 +34,25 @@ export async function runHistory(args: string[], output: CliOutput): Promise = { init: ['--directory'], list: ['--directory', '--project-id', '--html'], show: ['--directory', '--html'], delete: ['--directory'], prune: ['--directory', '--project-id', '--keep-last', '--older-than-days'] }; + const allowed: Record = { compare: ['--directory', '--html'], init: ['--directory'], list: ['--directory', '--project-id', '--html'], show: ['--directory', '--html'], delete: ['--directory'], prune: ['--directory', '--project-id', '--keep-last', '--older-than-days'] }; if ([...values.keys()].some((key) => !allowed[command]!.includes(key)) || (apply && !['delete', 'prune'].includes(command))) throw new Error('Option not supported for this history command.'); const dir = values.get('--directory'); if (!dir) throw new Error('History requires an explicit --directory.'); if (command === 'init') { const project = await LocalScanHistoryRepository.init(dir); output.stdout(json ? JSON.stringify(project, null, 2) : `History initialized. Project UUID: ${project.projectId}`); return 0; } + if (command === 'compare' && (!scanId || !afterId)) throw new Error('History compare requires explicit before and after scan UUIDs.'); const repo = await LocalScanHistoryRepository.open(dir, values.get('--project-id')); + if (command === 'compare') { + const comparison = compareSavedScans(await repo.get(scanId!), await repo.get(afterId!)); + const text = json ? formatComparisonJson(comparison) : formatComparisonTerminal(comparison); + const html = values.get('--html'); + if (html) await writeNewArtifact(html, formatComparisonHtml(comparison)); + output.stdout(text); return 0; + } if (command === 'list') { const listing = await repo.list(); const html = values.get('--html'); if (html) await writeNewArtifact(html, formatHistoryHtml(listing)); diff --git a/src/comparison/index.ts b/src/comparison/index.ts new file mode 100644 index 0000000..0652e97 --- /dev/null +++ b/src/comparison/index.ts @@ -0,0 +1,102 @@ +import { createHash } from 'node:crypto'; +import { validateSavedScan } from '../history/validation.ts'; +import { canonicalJson } from '../reports/context.ts'; +import { maxHtmlBytes } from '../artifacts/filesystem.ts'; +import type { SavedScan } from '../types/history.ts'; +import type { ReportBlock } from '../types/reporting.ts'; +import type { ScanComparison, ComparisonContext, AssessmentEvidence, AssessmentComparison } from '../types/comparison.ts'; + +const digest = (value: unknown) => createHash('sha256').update(canonicalJson(value)).digest('hex'); +const rows = (block: ReportBlock, label: string) => block.rows.filter((row) => row.label === label).map((row) => row.value); +const ordered = (values: unknown[]) => values.map(canonicalJson).sort(); +const section = (record: SavedScan, kind: 'scope' | 'integrations' | 'findings' | 'limitations') => record.snapshot.sections.find((s) => s.kind === kind)!.blocks; +const budget = (record: SavedScan) => ordered(section(record, 'scope').flatMap((b) => b.rows.filter((r) => /^(?:Scan|Module|Assessment) budget:/u.test(r.label)))); +function context(record: SavedScan): ComparisonContext { + return { scanId: record.scanId, projectId: record.projectId, savedAt: record.savedAt, envelopeSchema: record.schemaVersion, + projectionSchema: record.snapshot.schemaVersion, metadata: record.snapshot.metadata, complete: record.snapshot.complete, + coverage: record.snapshot.coverage, scope: section(record, 'scope'), notices: record.snapshot.notices, + limitations: section(record, 'limitations'), integrations: record.identities.integrations.length, assessments: record.identities.findings.length }; +} +function evidenceFingerprint(blocks: ReportBlock[]): string { + return digest(ordered(blocks.map((b) => ({ rows: b.rows.filter((r) => !['Finding reference', 'Existing machine status'].includes(r.label)), + notes: [...b.notes].sort(), finding: b.finding ? { ...b.finding, actions: [...b.finding.actions].sort(), references: [...b.finding.references].sort(), moreReferences: b.finding.moreReferences, uncertainty: [...b.finding.uncertainty].sort() } : null })))); +} +function assessments(record: SavedScan): Map { + const blocks = new Map(); + for (const block of section(record, 'findings').filter((b) => b.finding)) { + const references = rows(block, 'Finding reference'); + if (references.length !== 1 || blocks.has(references[0]!)) throw new Error('Comparison requires unambiguous saved finding references.'); + blocks.set(references[0]!, block); + } + const groups = new Map(); + for (const identity of record.identities.findings) { + const evidence = blocks.get(identity.reference)!; + const definition = rows(evidence, 'Original client definition'); const manifest = rows(evidence, 'Definition manifest'); + // Positional hashes are not persistent identities. Require corroborating unique saved scope. + const clientScope = identity.clientReference === null && definition.length === 0 ? ['integration-fallback'] + : identity.clientReference !== null && definition.length === 1 && manifest.length === 1 && ![...definition, ...manifest].some((v) => /withheld|unknown/iu.test(v)) + ? [identity.clientReference, definition[0], manifest[0]] : ['unmatchable', record.scanId, identity.reference]; + const key = canonicalJson([identity.provider, identity.packageName, identity.changeId, identity.scope, identity.category, clientScope]); + const group = groups.get(key) ?? []; group.push({ identity, evidence }); groups.set(key, group); + } + for (const group of groups.values()) group.sort((a, b) => a.identity.reference < b.identity.reference ? -1 : a.identity.reference > b.identity.reference ? 1 : 0); + return groups; +} + +/** Pure bounded comparison of validated schema-1 records. Never reads targets or current catalog. */ +export function compareSavedScans(beforeInput: SavedScan, afterInput: SavedScan): ScanComparison { + const before = validateSavedScan(beforeInput); const after = validateSavedScan(afterInput); + if (before.projectId !== after.projectId) throw new Error('Comparison requires the same explicit project UUID; cross-project comparison is unsupported.'); + const warnings: string[] = []; + const bm = before.snapshot.metadata; const am = after.snapshot.metadata; + if (before.scanId === after.scanId) warnings.push('The same saved scan was selected twice; this is not evidence of a separate manual scan.'); + if (Date.parse(bm.startedAt) >= Date.parse(am.startedAt)) warnings.push('Original timestamps are equal or reversed; command position defines before/after and arguments were not reordered.'); + if (Date.parse(before.savedAt) >= Date.parse(after.savedAt)) warnings.push('Save timestamps are equal or reversed; wall clocks do not prove scan chronology.'); + if (!bm.scannerVersion || !am.scannerVersion || !bm.buildFingerprint || !am.buildFingerprint) warnings.push('Scanner version/build identity is unknown; detection context comparability is inconclusive.'); + if (bm.scannerVersion !== am.scannerVersion || bm.buildFingerprint !== am.buildFingerprint) warnings.push('Scanner version or build fingerprint differs; detection changes cannot be isolated from source evidence changes.'); + if (bm.catalogVersion !== am.catalogVersion || bm.catalogFingerprint !== am.catalogFingerprint || bm.catalogRecordCount !== am.catalogRecordCount || digest(bm.providers) !== digest(am.providers)) warnings.push('Catalog revision, fingerprint or provider coverage differs; recorded assessments are not recalculated and source-only attribution is inconclusive.'); + if (!before.snapshot.complete || !after.snapshot.complete) warnings.push('At least one scan has incomplete coverage; absence of evidence cannot establish removal or resolution.'); + const requiredCoverage = ['Discovery coverage', 'Module coverage', 'Assessment coverage', 'Assessments truncated', 'Evidence display truncated']; + const completeCoverage: Record = { 'Discovery coverage': 'Eligible scope analyzed', 'Module coverage': 'Indexed scope analyzed', 'Assessment coverage': 'Supported candidates displayed', 'Assessments truncated': 'false', 'Evidence display truncated': 'false' }; + for (const [name, record] of [['Before', before], ['After', after]] as const) { + if (requiredCoverage.some((label) => record.snapshot.coverage.filter((r) => r.label === label).length !== 1)) warnings.push(`${name} coverage metadata is incomplete or ambiguous; comparison is inconclusive.`); + if (record.snapshot.coverage.some((r) => Object.hasOwn(completeCoverage, r.label) && completeCoverage[r.label] !== r.value)) warnings.push(`${name} coverage records partial, truncated or unsupported scope; comparison is inconclusive.`); + if (record.identities.integrations.some((i) => section(record, 'integrations').filter((b) => b.title.endsWith(` · ${i.packageName}`)).length !== 1)) warnings.push(`${name} integration evidence projection is absent or ambiguous.`); + if (record.snapshot.coverage.some((r) => /truncated/iu.test(r.label) && r.value !== 'false') || section(record, 'findings').some((b) => (b.finding !== undefined && (rows(b, 'Evidence display complete').length !== 1 || rows(b, 'Evidence display complete')[0] !== 'true')) || (b.finding?.moreReferences ?? 0) > 0)) warnings.push(`${name} saved evidence is truncated or its display completeness is unknown.`); + } + if (digest(ordered(before.snapshot.coverage)) !== digest(ordered(after.snapshot.coverage))) warnings.push('Recorded coverage differs; missing findings may reflect different supported or displayed scope.'); + if (!budget(before).length || !budget(after).length || digest(budget(before)) !== digest(budget(after))) warnings.push('Effective resource budgets differ or are absent; scan scope comparability is inconclusive.'); + const integrations: ScanComparison['integrations'] = []; + const integrationMap = (record: SavedScan) => new Map(record.identities.integrations.map((i) => [canonicalJson([i.provider, i.packageName]), i])); + const bi = integrationMap(before); const ai = integrationMap(after); + for (const key of [...new Set([...bi.keys(), ...ai.keys()])].sort()) { + const i = bi.get(key) ?? ai.get(key)!; + const blocks = (r: SavedScan) => section(r, 'integrations').filter((b) => b.title.endsWith(` · ${i.packageName}`)); + const b = bi.has(key) ? blocks(before) : []; const a = ai.has(key) ? blocks(after) : []; + integrations.push({ ...i, category: !bi.has(key) ? 'new' : !ai.has(key) ? 'missing' : 'unchanged', evidenceChanged: evidenceFingerprint(b) !== evidenceFingerprint(a), before: b, after: a }); + } + const bg = assessments(before); const ag = assessments(after); const result: AssessmentComparison[] = []; + for (const key of [...new Set([...bg.keys(), ...ag.keys()])].sort()) { + const b = bg.get(key) ?? []; const a = ag.get(key) ?? []; const left = b[0]; const right = a[0]; + let category: AssessmentComparison['category']; let reason: string; + if (b.length > 1 || a.length > 1 || key.includes('"unmatchable"')) { category = 'inconclusive'; reason = 'Client scope is missing or ambiguous; no transition is asserted.'; } + else if (!left) { category = 'new'; reason = 'Newly recorded assessment; not a newly announced provider change.'; } + else if (!right) { category = 'missing'; reason = 'Assessment no longer recorded; this does not prove a fix or runtime removal.'; } + else if (left.identity.recordFingerprint !== right.identity.recordFingerprint) { category = 'inconclusive'; reason = 'The embedded historical record fingerprint changed; statuses are shown without asserting a like-for-like transition.'; } + else { category = left.identity.status === right.identity.status ? 'unchanged' : 'changed'; reason = 'Matched unique saved client/integration scope and historical record fingerprint; observed statuses only, not runtime verification.'; } + const uncertainty = category === 'inconclusive' || !left || !right ? 'inconclusive' : left.identity.status !== 'unknown' && right.identity.status === 'unknown' ? 'increased' : left.identity.status === 'unknown' && right.identity.status !== 'unknown' ? 'reduced' : 'unchanged'; + result.push({ key: digest(key), category, reason, uncertainty, evidenceChanged: evidenceFingerprint(b.map((x) => x.evidence)) !== evidenceFingerprint(a.map((x) => x.evidence)), before: b, after: a }); + } + if (result.some((r) => r.category === 'new' || r.category === 'missing')) warnings.push('Unmatched assessments remain separate; source movement or changed client scope cannot be distinguished from new/missing evidence.'); + if (result.some((r) => r.category === 'inconclusive')) warnings.push('Some assessment identities or historical records cannot be matched conclusively.'); + const comparison: ScanComparison = { schemaVersion: 1, before: context(before), after: context(after), inconclusive: warnings.length > 0, warnings, + limitations: ['Saved snapshots are historical, privacy-filtered and not authenticated. No project was rescanned or code executed.', + 'Matching positional source references cannot prove persistent client identity, runtime behavior, a repair or application health.', + 'Absence can reflect unsupported discovery, withheld/truncated evidence, source movement, options, Scanner build or catalog changes; it never proves a fix.', + 'Uncertainty increased/reduced describes only transitions into/out of Unknown, not compatibility improvement. Partial coverage is reported separately.', + 'Command position defines older/newer roles. Any source edits were manual or external; PatchWRK performed no repair.', + 'Saved evidence omits raw arguments, API declaration values and source text; equal displayed scope cannot establish identical runtime inputs.'], + integrations, assessments: result }; + if (Buffer.byteLength(JSON.stringify(comparison), 'utf8') > maxHtmlBytes) throw new Error('Comparison exceeds the 32 MiB output limit.'); + return comparison; +} diff --git a/src/reports/comparison.ts b/src/reports/comparison.ts new file mode 100644 index 0000000..874ba0b --- /dev/null +++ b/src/reports/comparison.ts @@ -0,0 +1,32 @@ +import { createHash } from 'node:crypto'; +import type { ScanComparison, AssessmentComparison } from '../types/comparison.ts'; +import { escapeHtml } from './html.ts'; +import { maxHtmlBytes } from '../artifacts/filesystem.ts'; + +const css = `*{box-sizing:border-box}body{font:16px/1.6 system-ui,sans-serif;background:#f6f5f1;color:#191a1c;margin:0}main{max-width:1120px;margin:auto;padding:40px 24px}.brand{font-size:18px;font-weight:750}h1{font-size:clamp(30px,4.2vw,44px);line-height:1.15}h2{font-size:24px;margin-top:32px}h3{font-size:18px;margin:0}.muted,dt{color:#55585d}.notice{border-left:3px solid #191a1c;background:#fff;padding:12px 18px;margin:24px 0}.stats{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:12px}.stat,article,.technical{background:#fff;border:1px solid #deded7;border-radius:12px;padding:20px;margin:12px 0}.stat strong{display:block;font-size:28px}.stat span{font-size:13px}.badge{font-size:12px;font-weight:650;border-radius:20px;border:1px solid #d7dade;background:#f1f2f3;color:#4a5058;padding:4px 10px;display:inline-block}.review{color:#805000;background:#fff6e6;border-color:#ead4a9}dl{display:grid;grid-template-columns:1fr 2fr;gap:8px 20px}dd{margin:0}p,dd,li,h3,pre{overflow-wrap:anywhere}pre{font:13px/1.6 ui-monospace,monospace;white-space:pre-wrap}details{border-top:1px solid #deded7;margin-top:18px;padding-top:14px}summary{cursor:pointer;font-weight:650}summary:focus-visible{outline:3px solid #151515;outline-offset:4px}footer{font-size:13px;color:#55585d;margin-top:28px}@media(max-width:700px){main{padding:24px 16px}.stats{grid-template-columns:repeat(2,minmax(0,1fr))}dl{grid-template-columns:1fr}dd{margin-bottom:10px}article,.technical{padding:16px}}`; +const labels = { new: 'Newly recorded assessment', missing: 'Assessment no longer recorded', changed: 'Changed assessment status', unchanged: 'Unchanged assessment status', inconclusive: 'Inconclusive comparison' }; +const statuses = { affected: 'Affected', 'potentially-affected': 'Potential risk', unknown: 'Unknown', 'not-affected-by-this-change': 'Not affected by this change' }; +const transition = (r: AssessmentComparison) => `${r.before.map((v) => statuses[v.identity.status]).join(', ') || 'Not recorded'} → ${r.after.map((v) => statuses[v.identity.status]).join(', ') || 'Not recorded'}`; +function bounded(text: string): string { + if (Buffer.byteLength(text, 'utf8') > maxHtmlBytes) throw new Error('Comparison exceeds the 32 MiB output limit.'); + return text; +} +export function formatComparisonJson(c: ScanComparison): string { return bounded(JSON.stringify(c, null, 2)); } +export function formatComparisonTerminal(c: ScanComparison): string { + return bounded(['PatchWRK Scanner — Local Scan Comparison (historical snapshots)', + `Before: ${c.before.scanId} | ${c.before.metadata.startedAt} | integrations ${c.before.integrations} | assessments ${c.before.assessments}`, + `After: ${c.after.scanId} | ${c.after.metadata.startedAt} | integrations ${c.after.integrations} | assessments ${c.after.assessments}`, + ...c.warnings.map((w) => `Warning: ${w}`), + ...c.integrations.map((i) => `${i.category === 'new' ? 'Newly detected' : i.category === 'missing' ? 'No longer detected' : 'Unchanged'} integration: ${JSON.stringify([i.provider, i.packageName])}; saved evidence ${i.evidenceChanged ? 'differs' : 'unchanged'}`), + ...c.assessments.map((r) => `${labels[r.category]}: ${JSON.stringify((r.before[0] ?? r.after[0])!.identity.changeId)} | ${transition(r)} | uncertainty ${r.uncertainty} | evidence ${r.evidenceChanged ? 'differs' : 'unchanged'}\n ${r.reason}`), + ...c.limitations, 'Use --json for complete saved evidence references and comparison context, or --html NEW.html for a private dashboard.'].join('\n')); +} +export function formatComparisonHtml(c: ScanComparison): string { + const e = escapeHtml; const hash = createHash('sha256').update(css).digest('base64'); + const details = (title: string, value: unknown) => `
${e(title)}
${e(JSON.stringify(value, null, 2))}
`; + const integration = (i: ScanComparison['integrations'][number]) => `

${e(i.provider)} · ${e(i.packageName)}

${i.category === 'new' ? 'Newly detected integration' : i.category === 'missing' ? 'Integration no longer detected' : 'Unchanged integration identity'}

Saved evidence ${i.evidenceChanged ? 'differs' : 'unchanged'}. Detection is limited to recorded supported scope; absence is not proof of removal.

${details('Source evidence — before and after', { before: i.before, after: i.after })}
`; + const assessment = (r: AssessmentComparison) => `

${e((r.before[0] ?? r.after[0])!.identity.changeId)}

${labels[r.category]}

${e(transition(r))}

Uncertainty: ${r.uncertainty}. Saved evidence ${r.evidenceChanged ? 'differs' : 'unchanged'}.

${e(r.reason)}

${details('Historical identifiers and source evidence', { before: r.before, after: r.after })}
`; + const stat = (label: string, value: number) => `
${value}${label}
`; + const changed = c.assessments.filter((r) => r.category !== 'unchanged'); const unchanged = c.assessments.filter((r) => r.category === 'unchanged'); + return bounded(`PatchWRK — Scan Comparison

PatchWRK Scanner

Scan Comparison

Review recorded differences between two manual local scans.

Comparison overview

Older role: original scan (UTC)
${e(c.before.metadata.startedAt)}
Newer role: original scan (UTC)
${e(c.after.metadata.startedAt)}
Supported integrations
${c.before.integrations} → ${c.after.integrations}
Recorded assessments
${c.before.assessments} → ${c.after.assessments}
${stat('Changed assessment statuses', c.assessments.filter((r) => r.category === 'changed').length)}${stat('Unchanged assessment statuses', unchanged.length)}${stat('Increased uncertainty', c.assessments.filter((r) => r.uncertainty === 'increased').length)}${stat('Reduced uncertainty', c.assessments.filter((r) => r.uncertainty === 'reduced').length)}

What changed

${c.integrations.filter((i) => i.category !== 'unchanged').map(integration).join('')}${changed.map(assessment).join('') || '

No changed, new, missing or inconclusive assessment identities were recorded.

'}

What stayed the same

${c.integrations.filter((i) => i.category === 'unchanged').map(integration).join('')}${unchanged.map(assessment).join('') || '

No unchanged matched assessments.

'}

Evidence

Expand each card for original source references, declarations, configuration evidence, historical finding references and record fingerprints. Line movement or changed references do not establish changed runtime behavior.

Technical details

${details('Original scan identities, Scanner/catalog fingerprints, schemas, scope and coverage', { schemaVersion: c.schemaVersion, before: c.before, after: c.after })}
No health score, monitoring, automatic repair, telemetry or Cloud synchronization. Schema validation does not authenticate local records. Review privately before sharing.
`); +} diff --git a/src/types/comparison.ts b/src/types/comparison.ts new file mode 100644 index 0000000..ab58b39 --- /dev/null +++ b/src/types/comparison.ts @@ -0,0 +1,28 @@ +import type { SavedScan } from './history.ts'; +import type { ReportBlock, ReportView } from './reporting.ts'; + +export interface ComparisonContext { + scanId: string; projectId: string; savedAt: string; envelopeSchema: 1; projectionSchema: 1; + metadata: ReportView['metadata']; complete: boolean; coverage: ReportView['coverage']; + scope: ReportBlock[]; notices: string[]; limitations: ReportBlock[]; + integrations: number; assessments: number; +} +export interface IntegrationComparison { + provider: string; packageName: string; category: 'new' | 'missing' | 'unchanged'; + evidenceChanged: boolean; before: ReportBlock[]; after: ReportBlock[]; +} +export interface AssessmentEvidence { + identity: SavedScan['identities']['findings'][number]; evidence: ReportBlock; +} +export interface AssessmentComparison { + key: string; category: 'new' | 'missing' | 'changed' | 'unchanged' | 'inconclusive'; + uncertainty: 'increased' | 'reduced' | 'unchanged' | 'inconclusive'; + evidenceChanged: boolean; reason: string; + before: AssessmentEvidence[]; after: AssessmentEvidence[]; +} +/** Independent schema; never a current scan or a mutable saved record. */ +export interface ScanComparison { + schemaVersion: 1; before: ComparisonContext; after: ComparisonContext; + inconclusive: boolean; warnings: string[]; limitations: string[]; + integrations: IntegrationComparison[]; assessments: AssessmentComparison[]; +} diff --git a/tests/comparison.test.ts b/tests/comparison.test.ts new file mode 100644 index 0000000..4e6913d --- /dev/null +++ b/tests/comparison.test.ts @@ -0,0 +1,155 @@ +import { mkdtemp, mkdir, writeFile, readFile, rm, rename, chmod, symlink } from 'node:fs/promises'; +import { resolve } from 'node:path'; +import { randomUUID, createHash } from 'node:crypto'; +import { beforeEach, afterEach, expect, it } from 'vitest'; +import { captureReportContext } from '../src/reports/context.ts'; +import { buildReportView } from '../src/reports/view.ts'; +import { buildSavedScan } from '../src/history/validation.ts'; +import { compareSavedScans } from '../src/comparison/index.ts'; +import { formatComparisonHtml, formatComparisonJson, formatComparisonTerminal } from '../src/reports/comparison.ts'; +import { LocalScanHistoryRepository } from '../src/history/repository.ts'; +import { runCli } from '../src/cli/run.ts'; +import type { SavedScan } from '../src/types/history.ts'; + +let sandbox: string; let target: string; let history: string; let before: SavedScan; let after: SavedScan; +async function capture(projectId: string) { + const c = await captureReportContext(target, {}); + // Simulated stable build identity for synthetic comparison tests, not production provenance. + c.metadata.scannerVersion = '0.1.0'; c.metadata.buildFingerprint = 'b'.repeat(64); + return buildSavedScan(c.scan, buildReportView(c.scan, c.metadata), projectId); +} +beforeEach(async () => { + sandbox = await mkdtemp(resolve('.scanner-test-comparison-')); target = resolve(sandbox, 'target'); history = resolve(sandbox, 'history'); await mkdir(target); + await writeFile(resolve(target, 'package.json'), '{"dependencies":{"openai":"4.0.0"}}'); + await writeFile(resolve(target, 'index.ts'), "import OpenAI from 'openai'; const c=new OpenAI();c.completions.create({model:'text-davinci-003'});throw new Error('NEVER_EXECUTE');"); + const p = await LocalScanHistoryRepository.init(history); before = await capture(p.projectId); + await writeFile(resolve(target, 'index.ts'), "import OpenAI from 'openai'; const c=new OpenAI();c.completions.create({model:process.env.MODEL});throw new Error('NEVER_EXECUTE');"); + after = await capture(p.projectId); + before.snapshot.metadata.startedAt = '2026-10-19T10:00:00.000Z'; after.snapshot.metadata.startedAt = '2026-10-20T10:00:00.000Z'; + before.savedAt = '2026-10-19T10:01:00.000Z'; after.savedAt = '2026-10-20T10:01:00.000Z'; +}); +afterEach(async () => { await rm(sandbox, { recursive: true, force: true }); }); +async function cli(args: string[]) { + const stdout: string[] = []; const stderr: string[] = []; + const code = await runCli(args, { stdout: (s) => stdout.push(s), stderr: (s) => stderr.push(s) }); + return { code, stdout: stdout.join('\n'), stderr: stderr.join('\n') }; +} +const blocks = (r: SavedScan) => r.snapshot.sections.find((s) => s.kind === 'findings')!.blocks; +function removeFinding(r: SavedScan) { + const f = r.identities.findings.pop()!; + r.snapshot.sections.find((s) => s.kind === 'findings')!.blocks = blocks(r).filter((b) => !b.rows.some((row) => row.label === 'Finding reference' && row.value === f.reference)); + r.snapshot.stats[2]!.value = r.identities.findings.filter((v) => ['affected', 'potentially-affected'].includes(v.status)).length; + r.snapshot.stats[3]!.value = r.identities.findings.filter((v) => v.status === 'unknown').length; +} + +it('reports genuine Affected → Unknown, unchanged integration identity and evidence differences without mutation', () => { + const original = structuredClone([before, after]); const c = compareSavedScans(before, after); + expect([before, after]).toEqual(original); expect(c.integrations[0]!.category).toBe('unchanged'); + const changed = c.assessments.find((r) => r.before[0]!.identity.status === 'affected')!; + expect(changed.category).toBe('changed'); expect(changed.uncertainty).toBe('increased'); expect(changed.evidenceChanged).toBe(true); + expect(changed.after[0]!.identity.status).toBe('unknown'); expect(formatComparisonTerminal(c)).toContain('Affected → Unknown'); + expect(c.inconclusive).toBe(false); expect(formatComparisonJson(c)).not.toContain(sandbox); +}); +it('compares the same record deterministically with explicit same-record and clock warnings', () => { + const a = compareSavedScans(before, before); expect(compareSavedScans(before, before)).toEqual(a); + expect(a.assessments.every((r) => r.category === 'unchanged')).toBe(true); expect(a.warnings.join(' ')).toContain('same saved scan'); +}); +it('does not reorder reversed timestamps and reports reduced Unknown only as an evidence status change', () => { + const c = compareSavedScans(after, before); expect(c.before.scanId).toBe(after.scanId); + expect(c.warnings.join(' ')).toContain('not reordered'); expect(c.assessments.some((r) => r.uncertainty === 'reduced')).toBe(true); +}); +it('rejects cross-project, unsupported schema and raw legacy reports', () => { + after.projectId = randomUUID(); expect(() => compareSavedScans(before, after)).toThrow('same explicit project'); + const bad = structuredClone(before); (bad as { schemaVersion: number }).schemaVersion = 2; + expect(() => compareSavedScans(before, bad)).toThrow(); expect(() => compareSavedScans(before, {} as SavedScan)).toThrow(); +}); +it.each(['scannerVersion', 'buildFingerprint', 'catalogVersion', 'catalogFingerprint'] as const)('discloses differing %s while preserving recorded statuses', (key) => { + if (key === 'scannerVersion') after.snapshot.metadata[key] = '0.2.0'; + else if (key === 'catalogVersion') after.snapshot.metadata[key] = '4'; + else after.snapshot.metadata[key] = 'f'.repeat(64); + const c = compareSavedScans(before, after); expect(c.inconclusive).toBe(true); expect(c.warnings.join(' ')).toMatch(/differs/u); + expect(c.assessments.some((r) => r.category === 'changed')).toBe(true); +}); +it('discloses unavailable build metadata, partial/truncated coverage and different budgets', () => { + after.snapshot.metadata.buildFingerprint = null; after.snapshot.complete = false; + after.snapshot.coverage.find((r) => r.label === 'Assessments truncated')!.value = 'true'; + after.snapshot.sections.find((s) => s.kind === 'scope')!.blocks[0]!.rows.find((r) => r.label === 'Scan budget: maxFiles')!.value = '1'; + const c = compareSavedScans(before, after); expect(c.inconclusive).toBe(true); + for (const word of ['unknown', 'incomplete', 'truncated', 'budgets']) expect(c.warnings.join(' ')).toContain(word); +}); +it('marks missing coverage metadata inconclusive even when both snapshots omit the same fields', () => { + before.snapshot.coverage = []; after.snapshot.coverage = []; + expect(compareSavedScans(before, after).warnings.join(' ')).toContain('coverage metadata is incomplete'); +}); +it('does not assert a like-for-like transition when embedded change content differs', () => { + after.identities.findings.forEach((f) => { f.recordFingerprint = 'f'.repeat(64); }); + expect(compareSavedScans(before, after).assessments.every((r) => r.category === 'inconclusive')).toBe(true); +}); +it('reports missing/new assessments without claiming fixed or inferring renamed clients', () => { + removeFinding(after); const missing = compareSavedScans(before, after).assessments.find((r) => r.category === 'missing')!; + expect(missing.reason).toContain('does not prove a fix'); + after = structuredClone(before); after.scanId = randomUUID(); + after.identities.findings.forEach((f) => { f.clientReference = 'f'.repeat(64); }); + const c = compareSavedScans(before, after); expect(c.assessments.filter((r) => r.category === 'missing')).toHaveLength(2); expect(c.assessments.filter((r) => r.category === 'new')).toHaveLength(2); +}); +it('refuses to match line-moved client scope even with an unchanged opaque client reference', () => { + blocks(after).forEach((b) => b.rows.forEach((r) => { if (r.label === 'Original client definition') r.value = 'index.ts:2'; })); + expect(compareSavedScans(before, after).assessments.every((r) => ['new', 'missing'].includes(r.category))).toBe(true); +}); +it('does not guess when multiple findings share the same client/change scope', () => { + const f = structuredClone(after.identities.findings[0]!); const b = structuredClone(blocks(after).find((b) => b.rows.some((r) => r.label === 'Finding reference' && r.value === f.reference))!); + f.reference = 'a'.repeat(64); b.rows.find((r) => r.label === 'Finding reference')!.value = f.reference; + after.identities.findings.push(f); blocks(after).push(b); after.snapshot.stats[3]!.value++; + expect(compareSavedScans(before, after).assessments.some((r) => r.category === 'inconclusive')).toBe(true); +}); +it('compares explicit integration fallback without inventing client identity', () => { + for (const r of [before, after]) { r.identities.findings.forEach((f) => { f.clientReference = null; }); blocks(r).forEach((b) => { b.rows = b.rows.filter((v) => v.label !== 'Original client definition'); }); } + expect(compareSavedScans(before, after).assessments.every((r) => r.category === 'changed')).toBe(true); +}); +it('reports newly detected/missing integrations with no inferred runtime installation or removal', () => { + const empty = structuredClone(after); empty.identities.integrations = []; empty.identities.findings = []; + empty.snapshot.stats[1]!.value = 0; empty.snapshot.stats[2]!.value = 0; empty.snapshot.stats[3]!.value = 0; + empty.snapshot.sections.find((s) => s.kind === 'findings')!.blocks = []; empty.snapshot.sections.find((s) => s.kind === 'integrations')!.blocks = []; + expect(compareSavedScans(before, empty).integrations[0]!.category).toBe('missing'); + expect(compareSavedScans(empty, before).integrations[0]!.category).toBe('new'); +}); +it('retains manifest declaration associations when integration rows are reordered', () => { + after = structuredClone(before); after.scanId = randomUUID(); + after.snapshot.sections.find((s) => s.kind === 'integrations')!.blocks[0]!.rows.reverse(); + expect(compareSavedScans(before, after).integrations[0]!.evidenceChanged).toBe(true); +}); +it('escapes injected snapshot text, hashes static CSS and has no executable or external resources', () => { + after.snapshot.notices.push(''); + const html = formatComparisonHtml(compareSavedScans(before, after)); const css = html.match(/