Skip to content

fix(supervisor): rename OCSF log files from .log to .jsonl - #4263

Merged
matthewgrossman merged 8 commits into
mainfrom
fix/4256-ocsf-jsonl-retention/matthewgrossman
Oct 14, 2026
Merged

matthewgrossman merged 8 commits into
mainfrom
fix/4256-ocsf-jsonl-retention/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Summary

Change the supervisor OCSF output filename from /var/log/openshell-ocsf.YYYY-MM-DD.log to /var/log/openshell-ocsf.YYYY-MM-DD.jsonl. The extension changes from .log to .jsonl; the directory remains /var/log. This is backward-incompatible for collectors and scripts matching the old filenames. Distinct suffixes prevent shorthand retention from pruning newly generated OCSF JSONL files. Shorthand filenames, daily rotation, and the existing three-file limits remain unchanged.

Related Issue

Closes #4256

Replaces the closed draft #4257 on a fresh branch from main.

Changes

  • Change only the supervisor OCSF appender's suffix from log to jsonl; keep the existing appender setup. Gateway/MXC filenames remain unchanged.
  • Do not migrate or protect legacy openshell-ocsf.*.log history. It was already subject to early shorthand pruning and remains eligible for it. Remove the migration helper and its tests.
  • Keep the denial-reader implementation unchanged and test exclusion of current OCSF .jsonl filenames.
  • Update collection paths and the embedded policy-advisor skill.

Compatibility and release

OCSF consumers must change their collection pattern from openshell-ocsf.*.log to openshell-ocsf.*.jsonl. Legacy files are not migrated or protected from shorthand cleanup; collect any needed history before upgrading. New Docker/Podman supervisor containers and Kubernetes supervisor pods start with empty log storage, while host-based VM supervisors may encounter existing files. Gateway/MXC consumers are unaffected.

This backward-incompatible filename change likely needs a minor version bump rather than a patch release. Confirm the release/versioning decision with the release owners before shipping; this PR does not change version files.

Why there is no legacy-log migration

This fix intentionally applies only to newly generated .jsonl files. Old OCSF .log history already had broken retention and could be pruned early by the shorthand writer; we are not introducing a history-preservation guarantee for those files. New Docker/Podman supervisor containers use fresh tmpfs storage, and new Kubernetes supervisor pods use fresh emptyDir storage, so those replacement resources have no old files to migrate. Keeping this change to a suffix update avoids migration code and failure handling for already best-effort, usually ephemeral history.

This is not a claim that every upgrade clears logs: Kubernetes container restarts within the same pod preserve emptyDir, and VM supervisors can use persistent host storage. Any remaining old .log files stay eligible for shorthand pruning. Operators needing that history must collect it before upgrading; this PR neither migrates nor explicitly deletes it.

Filename dependency audit

  • The published jq example in docs/observability/ocsf-json-export.mdx explicitly matched openshell-ocsf.*.log. That example, the filename references in docs/observability/logging.mdx and docs/observability/accessing-logs.mdx, and the embedded crates/openshell-supervisor-process/src/skills/policy_advisor.md are updated in this PR. Users following the old documented collection pattern must update it.
  • crates/openshell-driver-mxc/examples/run-ocsf-audit.ps1 has an actual openshell-ocsf*.log file search, but consumes the Windows gateway sink, whose filenames remain unchanged.
  • The /v1/denials reader in crates/openshell-supervisor-network/src/policy_local.rs depends on shorthand openshell.*.log filenames, which remain unchanged.
  • The local collector deployed by tasks/scripts/helm-k3s-local.sh receives traces over OTLP rather than tailing these files. Repository searches found no OTel filelog, Filebeat, or Fluent Bit configuration consuming the supervisor OCSF files.

The compatibility risk has concrete documentation/example evidence, but no external deployment was verified. Absence of an in-repo collector configuration does not establish that external consumers are unaffected.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated
  • Deployment E2E tests run (not run; no deployment or driver configuration changes)

Passed locally on macOS:

  • mise exec -- cargo test -p openshell-supervisor --bin openshell-supervisor --locked (13 tests after merging main)
  • mise exec -- cargo test -p openshell-supervisor-network --lib policy_local::tests --locked (38 tests)
  • mise exec -- cargo test -p openshell-supervisor-process --lib skills::tests --locked (3 tests)
  • mise exec -- cargo clippy -p openshell-supervisor --all-targets --locked -- -D warnings
  • Targeted Rust formatting, Markdown lint, mise run docs, and git diff --check. Fern reported zero errors and three existing warnings.

After merging main, the suffix change lives in its existing logging.rs helper, and the existing console/file test asserts the .jsonl extension. Main's JSON stderr output is preserved. The denial-reader test excludes current .jsonl OCSF files; legacy filename coverage and legacy-history guidance in the published docs are removed. The helper module and its retention/migration tests are removed to keep this fix to the suffix change and collection guidance. Before this simplification, independent startup and real rollover with distinct suffixes passed on a filesystem exposing creation timestamps; those tests are not retained in this PR.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (not applicable; no architecture change)

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman changed the title fix(supervisor): use jsonl files for independent OCSF retention fix(supervisor): rename OCSF log files from .log to .jsonl Oct 13, 2026
@matthewgrossman matthewgrossman self-assigned this Oct 13, 2026
@matthewgrossman
matthewgrossman marked this pull request as ready for review October 13, 2026 17:13
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman changed the title fix(supervisor): rename OCSF log files from .log to .jsonl fix(supervisor): rename OCSF log files from .log to .jsonl Oct 13, 2026
krishicks
krishicks previously approved these changes Oct 13, 2026
Comment thread crates/openshell-supervisor-network/src/policy_local.rs Outdated
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
matthewgrossman added this pull request to the merge queue Oct 14, 2026
Merged via the queue into main with commit 42e0d00 Oct 14, 2026
78 of 79 checks passed
@matthewgrossman
matthewgrossman deleted the fix/4256-ocsf-jsonl-retention/matthewgrossman branch October 14, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(logging): shorthand retention can prune supervisor OCSF JSONL history

2 participants