Skip to content

feat(playground): persist per-node trace of graph agents with PII redaction - #69

Merged
kartikmanimuthu merged 1 commit into
offloadmemory:mainfrom
fabicarvano:feat/playground-node-trace-pii
Oct 19, 2026
Merged

kartikmanimuthu merged 1 commit into
offloadmemory:mainfrom
fabicarvano:feat/playground-node-trace-pii

Conversation

@fabicarvano

Copy link
Copy Markdown
Contributor

Summary

Graph-agent playground executions already emitted per-node events over SSE, but only the final text was persisted, so the trace was lost when the tab was closed. This PR:

  • Persists the per-node trace in AgentExecution.output.trace (existing JSON column, no migration) in both the playground and playground/resume routes, including failed and paused nodes (the executor does not emit node_complete for them).
  • Redacts the persisted trace and the node_*, state_update and execution_complete SSE events with irreversible placeholders: [CPF], [CNPJ], [CARTAO], [EMAIL], [TELEFONE], [IP], [TOKEN]. Originals are discarded (no vault, no mapping). The agent still receives the real data.
  • Validates matches to limit false positives (CPF/CNPJ check digits, Luhn, IPv4 octets, Brazilian DDD). Redaction runs before truncation, so a cut never leaves part of an identifier.
  • Caps each trace field at 4000 chars and the trace at 200 entries.
  • Renders graph nodes in the playground console trace tab.
  • Adds a reserved, empty health category to the redactor, to be filled later (LGPD).

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactor / performance improvement
  • Dependency update
  • CI/CD or build system change

Testing

  • libs/shared: 36 new tests (redactor and trace collector) pass. Full libs/shared suite: 754 passed in 64 files.
  • tsc --noEmit in libs/shared: no errors.
  • tsc --noEmit in apps/web-ui: 0 errors in the changed files. 6 errors remain, all pre-existing and in untouched files (Fumadocs generated @/.source missing in app/(docs) and lib/docs-source.ts; it is produced by the codegen step).
  • Simulation with the real GraphExecutor (fake node executors, no database or LLM): full flow → pause → resume → failure. The persisted trace ended as completed, paused, failed, with no CPF or e-mail in the persisted trace or in the SSE events.
  • Not run: build, e2e, integration tests against a real database and LLM, lint, other projects' unit tests, and visual verification of the UI. The CI or a maintainer may need to cover these.

Known limitations and out of scope:

  • Only structured identifiers are detected. Names, addresses, free text and health data are not.
  • text_delta (the chat answer) is intentionally not redacted.
  • The execution error message (output.error, the SSE error event and the executor log) is still stored/emitted as-is, as before.
  • There is no execution history screen; the persisted trace is available through the existing GET /api/agents/[id]/playground/executions endpoint.
  • Not included: API v1 (production), per-tenant PII toggle and per-tenant custom patterns.

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

…action

Graph-agent playground executions streamed per-node events over SSE but
persisted only the final text, so the trace was lost when the tab closed.

- Persist a per-node trace in AgentExecution.output.trace (JSON column,
  no schema change) in both the playground and the resume routes, including
  failed and paused nodes, which the executor does not emit node_complete for.
- Redact the persisted trace and the node_*/state_update/execution_complete
  SSE events with irreversible placeholders ([CPF], [CNPJ], [CARTAO],
  [EMAIL], [TELEFONE], [IP], [TOKEN]). Originals are discarded; the agent
  still receives the real data. Redaction runs before truncation.
- Validate matches to limit false positives (CPF/CNPJ check digits, Luhn,
  IPv4 octets, BR DDD). Add a reserved, empty health category for later.
- Cap each trace field at 4000 chars and the trace at 200 entries.
- Render graph nodes in the playground console trace tab.

Only structured identifiers are detected; names, addresses and free text
are not. Text deltas (the chat answer) are intentionally left unredacted.

@kartikmanimuthu kartikmanimuthu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @fabicarvano — this is careful work, and the parts I'd normally worry about are the ones you got right: the types are structural so @chatbot/shared doesn't have to import @chatbot/agent-studio (no cycle), redaction runs before truncation so a cut can never leave half an identifier, and the CPF/CNPJ check digits, Luhn, octet range and BR DDD validation are all real rather than regex-only. The CHANGELOG entry and the honest "only structured identifiers are detected" limitation are appreciated too.

I executed the two modules under Node to check the claims rather than eyeball them. Two gaps I'd like closed before merge, because both defeat the stated purpose:

  1. The answer text is persisted unredacted. playground/route.ts:431 and resume/route.ts:180 write output: { text: fullText, ...trace }. The trace is redacted, but text — the field most likely to contain the PII, since it's the model echoing a CPF or an e-mail back — is not, and neither is error: errorMessage on the failure paths. If the intent is LGPD-safe persistence, it fails in the same JSON column. text: redactText(fullText) is a one-line fix; if the answer body is deliberately out of scope, that needs saying in the CHANGELOG.
  2. There's no aggregate size cap. Per-field is 4000 chars, per-trace is 200 entries, but a single entry with 40 keys measured 160,453 bytes — so 200 entries can reach ~32 MB in one AgentExecution.output row. A whole-trace byte budget (~256 KB, dropping oldest) would bound it.

Smaller things I confirmed while testing, all good follow-ups rather than blockers:

  • PII carried as a number passes through untouched ({ cpf: 52998224725 } survives) — tool args often do this.
  • Non-plain objects are returned as-is by design (redact.ts:257), so a Map keeps its raw values.
  • False positive: order reference 79927398713 → order reference [TELEFONE]. The 11-digit mobile branch doesn't require separators, so any 11-digit run with a valid DDD and 9 in position 3 gets redacted.
  • sanitizeExecutionEvent spreads ...event, so an unknown event type or an extra field is streamed raw — an allow-list would be safer at a privacy boundary.
  • The collector tracks a single current node; two concurrent node_starts lose the first one's nodeType. Today's GraphExecutor is sequential so it's latent, but keying by nodeId costs nothing.
  • At the cap it keeps the oldest 200 and drops the newest — for diagnosing a failure you want the tail.
  • startedAt/durationMs are reconstructed from Date.now() at event-handling time; the executor already computes durationMs, so under SSE backpressure the durations inflate.

One design thought, not a request: irreversible redaction with no vault means you can never answer "what did the agent actually see?" — for a customer's erasure request or a support ticket. Worth being deliberate about, since it's cheap now and expensive later.

Approving. If the two items above are quick, I'd rather have them here than in a follow-up.

@kartikmanimuthu
kartikmanimuthu merged commit 1d1a607 into offloadmemory:main Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants