Repository navigation
0.86.0: redaction exceptions, hit counts, scan cache - #383
Merged
Merged
Conversation
Exceptions live in security.redact.exceptions as SHA-256 of the decoded value with its masked form, the rule that first recognised it and when it was added. A hit whose value is in the set is dropped before overlaps are merged, for every rule and both modes; saved secrets are never excepted. Findings carry the value hash for the security log, trials flag excepted values, and RuleSet exposes a fingerprint of the exception set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Schema 29 adds security_events.value_sha256. Secret items and log views carry the hash in process only (never serialized), and log views gain excepted/exceptable flags. Protocol 57 adds the exception endpoints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unts
GET/POST /security/exceptions, POST /security/events/{id}/exception and
DELETE /security/exceptions/{sha256}. Adding from the log takes the row
id and copies the hash stored with the row, so the hash never leaves
core; log views are annotated with excepted/exceptable at read time.
GET /security/counts sums log counts per rule for one guard since a time,
read from a covering index.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clients resend the whole conversation every turn. A per-runtime ScanCache now keeps the raw scan hits of each piece of the request body (each object in a top-level array such as messages/input/tools, and each run of other top-level fields), keyed by SHA-256 of the rule set fingerprint, the date and the piece's text. Only hit ranges and rule ids are stored, never values; at most 2,000 pieces and 65,536 hits, pieces under 64 bytes or over 1 MiB are not kept. Results are identical to a whole-body scan: placeholder removal, the written-text filter and merging still run on the whole body. Rules that look across neighbouring messages (keyword-preceded numbers, labeled fields) disable the cache; a command left open at the end of a piece is checked against what follows it. Differential tests run multi-turn conversations and adversarial bodies through both paths. RuleSet gains fingerprint(), destructured field by field so a new field that changes hits fails to compile until it is fed in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The steps after scanning (dropping hits on placeholders, outside the written text, on keys and base64 payloads) only look at where each hit lies, so they now run per piece and are cached with it. A request whose history contains a hit no longer parses the whole body each turn to find the written strings: an array element is matched as the only element of an array under the same key, which is all the written-text rules look at. Merging still runs on the whole request. Consecutive top-level fields form one piece, since some rules read neighbouring fields together. Keys that differ only in case or underscores fall back to the whole-body scan. tests/scan_cache.rs (ignored) times one look per turn at turns 10/100/200 of a 200-turn chat with and without the cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runtime::build makes a fresh ScanCache next to the redaction rules, so a config reload drops every result scanned under the old rules; a plugin-only swap keeps it. The inbound look, plugin re-look, per-hop replace and WebSocket frames pass it to the cached flow entry points. Also: per piece, the written-text ranges are read even without hits, so a body whose written text cannot be read (an out-of-range number, a lone surrogate) counts nothing on either path, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A finding carries a value hash only when every occurrence is a span some rule matched as it is; merged overlaps and record-only fragments get none, so listing a logged value as an exception always takes effect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The request database no longer stores any hash of a found value; schema
goes back to 28. The recorder keeps the hashes of the latest 20,000
redaction log rows in memory, which drive exceptable/excepted and
POST /security/events/{id}/exception (409 security.exception_forgotten
for rows no longer held). Duplicate and cap checks for adding an
exception run inside the config write, retried on a concurrent write.
The configuration manual notes that exceptions are plain SHA-256.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found bodies where the cache cost more than it saved: tens of thousands of short messages were parsed and scanned one by one (and never stored, being under the size floor), and every uncached piece parsed its written-text ranges even without hits. - Consecutive array elements are scanned and stored together until a run reaches 4 KiB; runs ahead of the appended tail stay byte-for-byte the same across turns. - A conversation seen for the first time is scanned whole as before and stored per piece by a background thread from that result; a stored conversation is recognised by a mark (first 4 KiB and a window late in its history) without splitting the body. - Written-text ranges are read only for pieces with hits, and for the rest only when the request has hits elsewhere; the result is kept. - Bodies under 64 KiB, or with more pieces than the cache holds, skip the cache; the open-command check finds command names with an automaton instead of tokenising every piece. The benchmark moves into the crate (it waits for the background store) and covers the reviewer's shapes; the reviewer's seam and Bedrock / other-shape comparisons are kept as tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A body ending right after a top-level `"key":` indexed past its end when split, which killed the background store and failed later requests of the same conversation. The worker now also outlives a panic in one job. Results found on one day are no longer stored under the next day's key when a scan crosses midnight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Version 0.86.0 with release notes for redaction exceptions, per-rule hit counts and the scan cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On slower runners the worker had not stored anything yet when a test counted the cache's segments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Redaction exceptions, per-rule hit counts and a scan cache for unchanged history. Version 0.86.0; full notes in
release-notes/0.86.0.md.security.redact.exceptions(SHA-256 + masked form, up to 500) excepts a value from every redaction rule; saved credentials cannot be excepted. Endpoints to list, add (by value or from a security-log row), and remove. Adding from the log uses hashes kept in memory for the latest 20,000 rows only; the request store holds no hash of any value. Log views and the trial mark excepted values.GET /security/countsreturns per-rule counts for one guard since a time, from an index.Upgrade: protocol 56 -> 57; request store schema unchanged (28); new message codes listed in the notes.
Reviewed in two adversarial rounds (hash exposure in the request store, merged spans, concurrent adds, cache cold-path cost, a panic on a truncated body, a midnight date edge); each finding has a regression test.
🤖 Generated with Claude Code