feat(plugin-audit): the parent-record read gates write one ReadableIds leaf per parent object, with no pre-scan; D5's three classes keep the bounded probe (ADR-0139 E5)

.changeset/23673-parent-gate-readable-ids.md

@@ -0,0 +1,18 @@
+---
+'@objectstack/plugin-audit': minor
+---
+
+feat(plugin-audit): the `sys_activity` and `sys_audit_log` read gates write one `ReadableIds` leaf per parent object and do no pre-scan, so a broad read is judged over every row, not a 2,000-row window (ADR-0139 D4, D5)
+
+Clause-②: no
+
+Both gates serve a row only when the caller may read the record it is about. Until now each read first scanned at most 2,000 candidate rows as the system, then asked which of those rows' parent records the caller could read. Past that window the gate failed closed: the unscanned rows were left out, the list's `total` agreed with the short answer, and a warning said so.
+
+Each gate now writes one branch per object the metadata registry holds (outside the audit writer's excluded objects and the gated object itself): `{ object_name: P, record_id: { $in: ReadableIds.of(P) } }`. The engine composes the leaf from the security service's `canReadObject` and `getReadFilter` for `P`, and the SQL drivers evaluate it as one subquery against `P`'s table, inside their own tenant wall. A list, its `total` and a grouped count are each one statement over every row the caller may read. A parent object the caller may not read is answered with no rows, without reading it as the caller, so no denial is logged for it.
+
+What a reader sees:
+
+- `sys_audit_log`: a broad read is served every row about a record the caller may read, and every row about no record, with a true `total`, whatever the table size. The rows about no record (`config_change`, `import`, `platform_admin_standing_change`, an auth event with no session) are now a filter on the ledger's own columns rather than a list of ids the scan saw. A holder of `view_all_audit_log` is still not narrowed. A row about a session is still served when that session is readable.
+- `sys_activity`: the gate itself has no window any more, but the stream's withheld-update rule still pre-scans 2,000 rows as the system, so a broad read stays bounded by that rule until it becomes a predicate (ADR-0139 E6). A read about one record is unchanged.
+- Three classes still read through the bounded probe, chosen before any leaf is written: a delegated (on-behalf-of) caller; a parent an object-scoped read middleware narrows (for example `sys_attachment`, `sys_comment_reaction`, `sys_approval_request` and the settings objects); and a parent bound to another datasource than the gated object, which covers every parent on a deployment that registers the `telemetry` datasource. The whole read also stays on the probe when the security service offers no `canReadObject`, or when none is registered, as before.
+- On a driver that does not compile the leaf (`driver-mongodb`), a read of either object now fails with `INTERNAL_ERROR` / 500 naming the driver and the leaf, instead of serving a capped list. The gates do not turn that refusal into an empty answer. This is ADR-0139's ruled behaviour until a pipeline form is measured for that driver.

packages/plugins/plugin-audit/src/activity-milestone-kind.integration.test.ts

@@ -105,7 +105,7 @@ describe('[#22771] a fired milestone is served with metadata.kind, judged like t
   const allFields = (object: string): string[] =>
     Object.keys(((engine as any).getSchema(object)?.fields ?? {}) as Record<string, unknown>);
 
-  /** The double: the two contract members, answered per reader. No field is served masked. */
+  /** The double: the two field members, answered per reader, and the two read halves (below). No field is served masked. */
   const security = {
     async getReadableFields(object: string, context?: any): Promise<string[] | undefined> {
       const all = allFields(object);
@@ -116,6 +116,15 @@ describe('[#22771] a fired milestone is served with metadata.kind, judged like t
     async getQueryableFields(object: string, context?: any): Promise<string[] | undefined> {
       return security.getReadableFields(object, context);
     },
+    // [ADR-0139 E5] Every reader reads every parent record here: the two halves
+    // of a read the engine composes the parent-record gate's ReadableIds
+    // leaves from — the object-level admission and the read scope (none).
+    async canReadObject(): Promise<boolean> {
+      return true;
+    },
+    async getReadFilter(): Promise<Record<string, unknown> | undefined> {
+      return undefined;
+    },
   };
 
   /** The console's activity read: one record's rows, as `context`. */

packages/plugins/plugin-audit/src/activity-read-visibility.test.ts

@@ -2,16 +2,28 @@
 
 /**
  * The `sys_activity` read gate's branches that a real engine cannot easily be
- * made to take — a failing pre-scan, a failing parent read, the scan bound —
- * plus the shape of what it emits. What the emitted filter SELECTS is pinned
- * against a real driver in `activity-read-visibility.integration.test.ts`.
+ * made to take, plus the shape of what it emits:
+ *
+ *  - the leaf path (ADR-0139 D4) against a stand-in engine that answers the
+ *    three questions the gate classifies by — one `ReadableIds` leaf per
+ *    enumerated parent object, nothing scanned, nothing asked;
+ *  - D5's three classes routed to the bounded probe, by name, up front;
+ *  - the bounded probe itself — a failing pre-scan, a failing parent read,
+ *    the scan bound, the object-level admission — against a stand-in engine
+ *    that answers none of the three, so the whole read is the probe.
+ *
+ * What the emitted filter SELECTS is pinned against a real engine and driver
+ * in `parent-record-read-leaf.integration.test.ts` (the leaf) and
+ * `activity-read-visibility.integration.test.ts` (the probe).
  */
 
 import { describe, it, expect, vi } from 'vitest';
+import { isReadableIds } from '@objectstack/spec/data';
 import { installActivityReadVisibility, parseActivityParent } from './activity-read-visibility.js';
 import { LEDGER_AUDIT_CAPABILITY } from './audit-log-read-visibility.js';
+import { AUDIT_EXCLUDED_OBJECTS } from './audit-writers.js';
 import type { CommentAccessEngine, CommentReadMiddlewareCtx } from './comment-access-hooks.js';
-import type { ParentObjectReadAdmissionSource } from './parent-record-read-gate.js';
+import type { ParentObjectReadAdmissionSource, ParentRecordGateEngine } from './parent-record-read-gate.js';
 
 const DENY = { id: '__activity_parent_denied__' };
 const silentLogger = () => ({ info: vi.fn(), warn: vi.fn(), debug: vi.fn() });
@@ -125,9 +137,9 @@ describe('installActivityReadVisibility', () => {
     expect(calls.probes).toEqual([]);
   });
 
-  it('leaves a read that matches nothing alone', async () => {
+  it('a read the probe finds no row for is still narrowed, so a row written meanwhile is judged', async () => {
     const { mw } = install({ scan: () => [] });
-    expect((await read(mw)).where).toBeUndefined();
+    expect((await read(mw)).where).toEqual(DENY);
   });
 
   it('a parent object whose read throws contributes nothing (fail closed)', async () => {
@@ -328,4 +340,274 @@ describe('installActivityReadVisibility', () => {
     const engine: CommentAccessEngine = { registerHook: () => {}, find: async () => [], findOne: async () => null };
     expect(() => installActivityReadVisibility(engine, silentLogger(), () => undefined)).not.toThrow();
   });
+
+  it('an engine that cannot classify a parent sends the whole read to the probe, and says so once', async () => {
+    const logger = silentLogger();
+    const admission = () => ({ canReadObject: async () => true });
+    const { mw, calls } = install({ scan: () => [{ object_name: 'crm_case', record_id: 'c1' }], readable: { crm_case: ['c1'] }, logger, admission });
+    await read(mw);
+    await read(mw);
+    expect(calls.scans).toHaveLength(2);
+    const said = logger.warn.mock.calls.filter(([line]) => String(line).includes('cannot say which parent objects'));
+    expect(said).toHaveLength(1);
+  });
+});
+
+/** A stand-in engine that answers the three questions the gate classifies a parent by. */
+function installClassifying(opts: {
+  objects: string[];
+  /** Parent objects an object-scoped read middleware narrows. */
+  middleware?: string[];
+  /** object → datasource; absent: the default driver (`undefined`). */
+  datasource?: Record<string, string>;
+  scan?: (options: any) => Array<Record<string, unknown>>;
+  readable?: Record<string, string[]>;
+  admission?: ParentObjectReadAdmissionSource;
+  logger?: ReturnType<typeof silentLogger>;
+  registry?: unknown;
+}) {
+  let mw!: Mw;
+  const calls = { scans: [] as any[], probes: [] as Array<{ object: string; ids: string[]; context: any }> };
+  const base: CommentAccessEngine = {
+    registerHook: () => {},
+    registerMiddleware: (fn) => {
+      mw = fn as Mw;
+    },
+    find: async (object: string, options: any) => {
+      if (object === 'sys_activity') {
+        calls.scans.push(options);
+        return (opts.scan?.(options) ?? []) as any;
+      }
+      const ids: string[] = (options?.where?.id?.$in ?? []).map(String);
+      calls.probes.push({ object, ids, context: options?.context });
+      const list = opts.readable?.[object] ?? [];
+      return ids.filter((id) => list.includes(id)).map((id) => ({ id })) as any;
+    },
+    findOne: async () => null,
+  };
+  // The classifying questions, over the comment-gate double above.
+  const engine: ParentRecordGateEngine = Object.assign(base, {
+    registry: (opts.registry ?? { getAllObjects: () => opts.objects.map((name) => ({ name })) }) as any,
+    hasObjectMiddleware: (object: string) => (opts.middleware ?? []).includes(object),
+    resolveEffectiveDatasource: (object: string) => opts.datasource?.[object],
+  });
+  // A security service that answers `canReadObject` unless the case says otherwise:
+  // without it the gate writes no leaf (see the `no-object-admission` case).
+  installActivityReadVisibility(
+    engine,
+    opts.logger ?? silentLogger(),
+    opts.admission ?? (() => ({ canReadObject: async () => true })),
+  );
+  return { mw, calls };
+}
+
+/** The `$or` branches of a gate WHERE (or the one branch). */
+const branchesOf = (where: any): any[] => (where?.$or ? where.$or : [where]);
+const leafObjectOf = (branch: any): string | null => {
+  const comparand = branch?.record_id?.$in;
+  return isReadableIds(comparand) ? comparand.object : null;
+};
+
+describe('[ADR-0139 D4] the leaf path: one ReadableIds leaf per parent object, and no pre-scan', () => {
+  it('writes one branch per registered parent object, in name order, each a branded leaf on record_id', async () => {
+    const { mw, calls } = installClassifying({ objects: ['crm_deal', 'crm_case'] });
+    const { where, ran } = await read(mw);
+    expect(ran).toBe(true);
+    const branches = branchesOf(where);
+    expect(branches.map((b) => b.object_name)).toEqual(['crm_case', 'crm_deal']);
+    expect(branches.map(leafObjectOf)).toEqual(['crm_case', 'crm_deal']);
+    // Nothing scanned, nothing read as the caller.
+    expect(calls.scans).toEqual([]);
+    expect(calls.probes).toEqual([]);
+  });
+
+  it('enumerates outside the audit writer’s excluded objects, the gated object and non-machine names', async () => {
+    const excluded = AUDIT_EXCLUDED_OBJECTS.filter((o) => o !== 'sys_activity').slice(0, 3);
+    expect(excluded.length).toBeGreaterThan(0);
+    const { mw } = installClassifying({ objects: ['crm_case', 'sys_activity', 'Not A Name', ...excluded] });
+    expect(branchesOf((await read(mw)).where).map(leafObjectOf)).toEqual(['crm_case']);
+  });
+
+  it('asks the security service nothing: the engine composes each leaf, so a refused object is never read', async () => {
+    const canReadObject = vi.fn(async () => false);
+    const { mw, calls } = installClassifying({ objects: ['crm_case', 'crm_secret'], admission: () => ({ canReadObject }) });
+    expect(branchesOf((await read(mw)).where).map(leafObjectOf)).toEqual(['crm_case', 'crm_secret']);
+    expect(canReadObject).not.toHaveBeenCalled();
+    expect(calls.probes).toEqual([]);
+  });
+
+  it('ANDs onto the caller’s where, on all four reads, and leaves writes, system and context-less reads alone', async () => {
+    const existing = { type: 'updated' };
+    for (const operation of ['find', 'findOne', 'count', 'aggregate'] as const) {
+      const { mw } = installClassifying({ objects: ['crm_case'] });
+      const where = (await read(mw, { operation, ast: { object: 'sys_activity', where: existing } })).where as {
+        $and: unknown[];
+      };
+      expect(where.$and[0]).toBe(existing);
+      expect(leafObjectOf(where.$and[1])).toBe('crm_case');
+    }
+    const { mw } = installClassifying({ objects: ['crm_case'] });
+    expect((await read(mw, { operation: 'insert' })).where).toBeUndefined();
+    expect((await read(mw, { context: { isSystem: true } })).where).toBeUndefined();
+    expect((await read(mw, { context: undefined })).where).toBeUndefined();
+  });
+
+  it('a fresh leaf per read: nothing is shared between two reads', async () => {
+    const { mw } = installClassifying({ objects: ['crm_case'] });
+    const first = (await read(mw)).where as any;
+    const second = (await read(mw)).where as any;
+    expect(first.record_id.$in).not.toBe(second.record_id.$in);
+  });
+
+  it('no parent object to enumerate: the deny-all sentinel', async () => {
+    const { mw } = installClassifying({ objects: ['sys_activity'] });
+    expect((await read(mw)).where).toEqual(DENY);
+  });
+
+  it('[#21260] a holder of the ledger’s audit capability is narrowed here exactly like any caller', async () => {
+    const { mw } = installClassifying({ objects: ['crm_case'] });
+    const holder = { userId: 'u1', systemPermissions: [LEDGER_AUDIT_CAPABILITY] };
+    expect(leafObjectOf((await read(mw, { context: holder })).where)).toBe('crm_case');
+  });
+});
+
+describe('[ADR-0139 D5] three classes keep the bounded probe, by name, chosen before any leaf is written', () => {
+  it('a parent an object-scoped read middleware narrows is probed — its rows only — beside the other parents’ leaves', async () => {
+    const { mw, calls } = installClassifying({
+      objects: ['crm_case', 'sys_attachment'],
+      middleware: ['sys_attachment'],
+      scan: () => [
+        { object_name: 'sys_attachment', record_id: 'f1' },
+        { object_name: 'sys_attachment', record_id: 'f2' },
+      ],
+      readable: { sys_attachment: ['f2'] },
+    });
+    const branches = branchesOf((await read(mw)).where);
+    expect(branches.map(leafObjectOf)).toEqual(['crm_case', null]);
+    expect(branches[1]).toEqual({ object_name: 'sys_attachment', record_id: { $in: ['f2'] } });
+    expect(calls.scans).toEqual([
+      {
+        where: { object_name: { $in: ['sys_attachment'] } },
+        fields: ['object_name', 'record_id'],
+        limit: 2000,
+        context: { isSystem: true },
+      },
+    ]);
+    expect(calls.probes.map((p) => p.object)).toEqual(['sys_attachment']);
+  });
+
+  it('a parent bound to another datasource than the stream is probed', async () => {
+    const { mw, calls } = installClassifying({
+      objects: ['crm_case', 'ext_ticket'],
+      datasource: { ext_ticket: 'warehouse' },
+      scan: () => [{ object_name: 'ext_ticket', record_id: 't1' }],
+      readable: { ext_ticket: ['t1'] },
+    });
+    const branches = branchesOf((await read(mw)).where);
+    expect(branches.map(leafObjectOf)).toEqual(['crm_case', null]);
+    expect(branches[1]).toEqual({ object_name: 'ext_ticket', record_id: { $in: ['t1'] } });
+    expect(calls.scans[0].where).toEqual({ object_name: { $in: ['ext_ticket'] } });
+  });
+
+  it('every parent federated (a telemetry-split stream): no leaf, the probe answers every row', async () => {
+    const { mw } = installClassifying({
+      objects: ['crm_case', 'crm_deal'],
+      datasource: { sys_activity: 'telemetry' },
+      scan: () => [{ object_name: 'crm_case', record_id: 'c1' }],
+      readable: { crm_case: ['c1'] },
+    });
+    expect((await read(mw)).where).toEqual({ object_name: 'crm_case', record_id: { $in: ['c1'] } });
+  });
+
+  it('a delegated caller: no leaf at all, the whole read is the probe, read with the caller’s envelope', async () => {
+    const onBehalfOf = { userId: 'u_delegator' };
+    const { mw, calls } = installClassifying({
+      objects: ['crm_case', 'crm_deal'],
+      scan: () => [{ object_name: 'crm_case', record_id: 'c1' }],
+      readable: { crm_case: ['c1'] },
+    });
+    const { where } = await read(mw, { context: { userId: 'u_agent', onBehalfOf } as any });
+    expect(where).toEqual({ object_name: 'crm_case', record_id: { $in: ['c1'] } });
+    expect(calls.scans[0].where).toEqual({});
+    expect(calls.probes[0].context).toMatchObject({ userId: 'u_agent', onBehalfOf });
+  });
+
+  it('no object-level admission (no canReadObject, or no security service): no leaf, the whole read is the probe, said once', async () => {
+    // A leaf would then be admitted by explain(), which refuses the SYSTEM
+    // re-read of this WHERE the withheld-update rule makes (gate module header).
+    const sources: ParentObjectReadAdmissionSource[] = [
+      () => undefined,
+      () => ({}),
+      () => {
+        throw new Error("service 'security' not registered");
+      },
+    ];
+    for (const admission of sources) {
+      const logger = silentLogger();
+      const { mw, calls } = installClassifying({
+        objects: ['crm_case', 'crm_deal'],
+        scan: () => [{ object_name: 'crm_case', record_id: 'c1' }],
+        readable: { crm_case: ['c1'] },
+        admission,
+        logger,
+      });
+      expect((await read(mw)).where).toEqual({ object_name: 'crm_case', record_id: { $in: ['c1'] } });
+      await read(mw);
+      expect(calls.scans.map((s) => s.where)).toEqual([{}, {}]);
+      const said = logger.warn.mock.calls.filter(([line]) => String(line).includes('answers no canReadObject()'));
+      expect(said).toHaveLength(1);
+    }
+  });
+
+  it('the probe’s cap warning names the class it was probing for', async () => {
+    const logger = silentLogger();
+    const rows = Array.from({ length: 2000 }, (_, i) => ({ object_name: 'sys_attachment', record_id: `f${i}` }));
+    const { mw } = installClassifying({ objects: ['crm_case', 'sys_attachment'], middleware: ['sys_attachment'], scan: () => rows, logger });
+    await read(mw);
+    expect(logger.warn).toHaveBeenCalledWith(expect.stringContaining('(object-middleware: sys_attachment) hit the 2000-row cap'));
+  });
+
+  it('a probe that fails denies the whole read, and says so', async () => {
+    const logger = silentLogger();
+    const { mw } = installClassifying({
+      objects: ['crm_case', 'sys_attachment'],
+      middleware: ['sys_attachment'],
+      scan: () => {
+        throw new Error('pre-scan exploded');
+      },
+      logger,
+    });
+    expect((await read(mw)).where).toEqual(DENY);
+    expect(logger.warn).toHaveBeenCalledWith(expect.stringContaining('failed, denying all'));
+  });
+});
+
+describe('[ADR-0139 D3] the gate catches nothing the read itself raises', () => {
+  it('a refusal the read raises after the gate hands it on reaches the caller unchanged', async () => {
+    const { mw } = installClassifying({ objects: ['crm_case'] });
+    const refusal = Object.assign(new Error('[driver-x] where.$or[0].record_id.$in holds a ReadableIds leaf'), {
+      code: 'INTERNAL_ERROR',
+      status: 500,
+    });
+    const ctx: CommentReadMiddlewareCtx = { object: 'sys_activity', operation: 'find', ast: { object: 'sys_activity' }, context: { userId: 'u1' } };
+    await expect(mw(ctx, async () => {
+      throw refusal;
+    })).rejects.toBe(refusal);
+  });
+
+  it('a fault in deciding how to read the parents fails the read: it is never served as the deny-all sentinel', async () => {
+    const registry = {
+      getAllObjects: () => {
+        throw new Error('registry exploded');
+      },
+    };
+    const { mw } = installClassifying({ objects: [], registry });
+    let ran = false;
+    const ctx: CommentReadMiddlewareCtx = { object: 'sys_activity', operation: 'find', ast: { object: 'sys_activity' }, context: { userId: 'u1' } };
+    await expect(mw(ctx, async () => {
+      ran = true;
+    })).rejects.toThrow('registry exploded');
+    expect(ran).toBe(false);
+    expect(ctx.ast?.where).toBeUndefined();
+  });
 });