Repository navigation
feat(playground): persist per-node trace of graph agents with PII redaction - #69
Conversation
…action Graph-agent playground executions streamed per-node events over SSE but persisted only the final text, so the trace was lost when the tab closed. - Persist a per-node trace in AgentExecution.output.trace (JSON column, no schema change) in both the playground and the resume routes, including failed and paused nodes, which the executor does not emit node_complete for. - Redact the persisted trace and the node_*/state_update/execution_complete SSE events with irreversible placeholders ([CPF], [CNPJ], [CARTAO], [EMAIL], [TELEFONE], [IP], [TOKEN]). Originals are discarded; the agent still receives the real data. Redaction runs before truncation. - Validate matches to limit false positives (CPF/CNPJ check digits, Luhn, IPv4 octets, BR DDD). Add a reserved, empty health category for later. - Cap each trace field at 4000 chars and the trace at 200 entries. - Render graph nodes in the playground console trace tab. Only structured identifiers are detected; names, addresses and free text are not. Text deltas (the chat answer) are intentionally left unredacted.
kartikmanimuthu
left a comment
There was a problem hiding this comment.
Thanks @fabicarvano — this is careful work, and the parts I'd normally worry about are the ones you got right: the types are structural so @chatbot/shared doesn't have to import @chatbot/agent-studio (no cycle), redaction runs before truncation so a cut can never leave half an identifier, and the CPF/CNPJ check digits, Luhn, octet range and BR DDD validation are all real rather than regex-only. The CHANGELOG entry and the honest "only structured identifiers are detected" limitation are appreciated too.
I executed the two modules under Node to check the claims rather than eyeball them. Two gaps I'd like closed before merge, because both defeat the stated purpose:
- The answer text is persisted unredacted.
playground/route.ts:431andresume/route.ts:180writeoutput: { text: fullText, ...trace }. The trace is redacted, buttext— the field most likely to contain the PII, since it's the model echoing a CPF or an e-mail back — is not, and neither iserror: errorMessageon the failure paths. If the intent is LGPD-safe persistence, it fails in the same JSON column.text: redactText(fullText)is a one-line fix; if the answer body is deliberately out of scope, that needs saying in the CHANGELOG. - There's no aggregate size cap. Per-field is 4000 chars, per-trace is 200 entries, but a single entry with 40 keys measured 160,453 bytes — so 200 entries can reach ~32 MB in one
AgentExecution.outputrow. A whole-trace byte budget (~256 KB, dropping oldest) would bound it.
Smaller things I confirmed while testing, all good follow-ups rather than blockers:
- PII carried as a number passes through untouched (
{ cpf: 52998224725 }survives) — tool args often do this. - Non-plain objects are returned as-is by design (
redact.ts:257), so aMapkeeps its raw values. - False positive:
order reference 79927398713→order reference [TELEFONE]. The 11-digit mobile branch doesn't require separators, so any 11-digit run with a valid DDD and9in position 3 gets redacted. sanitizeExecutionEventspreads...event, so an unknown event type or an extra field is streamed raw — an allow-list would be safer at a privacy boundary.- The collector tracks a single
currentnode; two concurrentnode_starts lose the first one'snodeType. Today'sGraphExecutoris sequential so it's latent, but keying bynodeIdcosts nothing. - At the cap it keeps the oldest 200 and drops the newest — for diagnosing a failure you want the tail.
startedAt/durationMsare reconstructed fromDate.now()at event-handling time; the executor already computesdurationMs, so under SSE backpressure the durations inflate.
One design thought, not a request: irreversible redaction with no vault means you can never answer "what did the agent actually see?" — for a customer's erasure request or a support ticket. Worth being deliberate about, since it's cheap now and expensive later.
Approving. If the two items above are quick, I'd rather have them here than in a follow-up.
Summary
Graph-agent playground executions already emitted per-node events over SSE, but only the final text was persisted, so the trace was lost when the tab was closed. This PR:
AgentExecution.output.trace(existing JSON column, no migration) in both theplaygroundandplayground/resumeroutes, including failed and paused nodes (the executor does not emitnode_completefor them).node_*,state_updateandexecution_completeSSE events with irreversible placeholders:[CPF],[CNPJ],[CARTAO],[EMAIL],[TELEFONE],[IP],[TOKEN]. Originals are discarded (no vault, no mapping). The agent still receives the real data.Type of Change
Testing
libs/shared: 36 new tests (redactor and trace collector) pass. Fulllibs/sharedsuite: 754 passed in 64 files.tsc --noEmitinlibs/shared: no errors.tsc --noEmitinapps/web-ui: 0 errors in the changed files. 6 errors remain, all pre-existing and in untouched files (Fumadocs generated@/.sourcemissing inapp/(docs)andlib/docs-source.ts; it is produced by the codegen step).GraphExecutor(fake node executors, no database or LLM): full flow → pause → resume → failure. The persisted trace ended ascompleted,paused,failed, with no CPF or e-mail in the persisted trace or in the SSE events.Known limitations and out of scope:
text_delta(the chat answer) is intentionally not redacted.output.error, the SSEerrorevent and the executor log) is still stored/emitted as-is, as before.GET /api/agents/[id]/playground/executionsendpoint.Checklist