Repository navigation
feat: add native JSON output (-J/-j flags) - #353
Merged
Merged
Conversation
Add Fjson/Fjsonl global flags, option parsing for -J (nested JSON) and -j (JSON Lines), mutual exclusivity validation, and default field selection when -F is not explicitly given.
Emits {"lsof_version":"...","processes":[...]} around process
objects. Handles empty results and suppresses -F marker output
in JSON modes during repeat cycles.
Refactors json_open_envelope/json_close_envelope into print.c which already includes version.h via the correct build ordering. Updates all dialect Makefiles to add version.h as a dependency for print.o.
Contributor
Author
CI StatusAll 14 checks pass except the Cirrus CI / FreeBSD 14.3 job, which is a known pre-existing failure unrelated to this PR. Evidence:
This PR touches only the CLI output path ( |
jiegec
reviewed
Mar 12, 2026
| } | ||
| } | ||
| if (!has_fields) { | ||
| for (i = 0; FieldSel[i].nm; i++) { |
Member
There was a problem hiding this comment.
Lots of code duplication with existing -F handling, can we deduplicate them?
Contributor
Author
There was a problem hiding this comment.
Done in bb8bab1. Extracted select_default_fields() static helper in main.c — now called from both the case 'F': default path and the -J/-j default field setup. Removed ~40 lines of duplicated #ifdef logic.
jiegec
reviewed
Mar 12, 2026
jiegec
reviewed
Mar 12, 2026
jiegec
reviewed
Mar 12, 2026
| static void json_print_str(int *sep, const char *key, const char *val) { | ||
| if (*sep) | ||
| putchar(','); | ||
| printf("\"%s\":\"", key); |
Member
There was a problem hiding this comment.
The key might also require escape?
Contributor
Author
There was a problem hiding this comment.
Good catch. Fixed in bb8bab1 — extracted json_print_key() helper that escapes the key via json_puts_escaped(). All json_print_* functions now route through it.
jiegec
reviewed
Mar 12, 2026
| if (FieldSel[LSOF_FIX_ACCESS].st) { | ||
| char a[2] = {access_to_char(Lf->access), '\0'}; | ||
| if (a[0] != ' ') | ||
| json_print_str(sep, "access", a); |
Member
There was a problem hiding this comment.
Maybe a json_print_char makes it simpler?
Contributor
Author
There was a problem hiding this comment.
Done in bb8bab1. Added json_print_char() and switched the access and lock fields to use it — avoids the char[2] array construction.
- Extract select_default_fields() to deduplicate field selection logic shared between -F and -J/-j default handling (jiegec review) - Escape JSON keys via json_print_key() helper (jiegec review) - Add json_print_char() for single-char fields like access and lock, avoiding unnecessary char[2] array construction (jiegec review)
|
FYI, as currently written, as strings in the output are encoded as JSON strings, and JSON kind of mandates UTF-8, that doesn't work properly for file names that are encoded in a character encoding other than ASCII or UTF-8. From testing it looks like it replaces all sequences of 1 or more bytes that can't be decoded into UTF-8 with one There's no good way to address that. That's a shortcoming of the JSON format.
I'm not saying See https://unix.stackexchange.com/questions/757832/how-to-process-json-with-strings-containing-invalid-utf-8 for more background on that. |
My bad, I messed up my testing. Looks like lsof behaves like lsfd and many other tools that dump those bytes as-is, so producing invalid JSON, but one that contains the information in a theoretically parseable way. Still worth documenting. |
This was referenced Mar 12, 2026
Contributor
|
Out of curiosity, how much of this was AI-generated code? |
kj299
pushed a commit
to kj299/lsof
that referenced
this pull request
Oct 17, 2026
…1, 90, 68) lsof-rs printed a JSON schema of its own: `user` for uid and login, `node` for inode and protocol, `size` a number, `local`/`remote`/`state` keys, no `lsof_version`, `pgid`, `offset`, `flags` or `tcp_info`, and a task's object the same as its process's. The C has had -J/-j since its PR lsof-org#353, and a script moved between the two broke. This is a breaking change, so the version is 2.0.0 (Cargo.toml, both lock files, the bug template); no tag is created. Model: OpenFile.name_addition (Option<NonZeroU16>, in the row's padding: it stays 192 bytes) records where NAME's addition starts, the C's Lf->nma. add_name_addition() appends it, base_name() returns the C's Lf->nm, both checked: a length past the name or inside a character is no addition, never a panic. Every backend addition goes through it: (readlink: E), (stat: E), NOFD's whole name, (-e FS)/(+e FS), the maps rows' (stat: …)/(path …), and Windows' -E endpoints. Prevents JSON names carrying text the C keeps apart. Renderer (render/json.rs, rewritten): the C's envelope, key order and types (print.c:1896-2175), measured key by key against the C over 85 option variants; written into the sink a process at a time. device_number and inode come from fs_device and node raw, so sockets and -e rows print them as the C does; NOFD is {"fd":"NOFD"}; tcp_info on TCP, UDP, UDP-Lite and unix rows only, a unix row's always {} and UDP-Lite's never with a state. The row predicates -F, JSON and the table share (node_is_protocol, device_text, access/lock blanks, tcp_info_socket) live once in render/mod.rs, and a property test holds every JSON key to its -F letter. JSON escapes with the table's own is_unprintable_unicode, so the two cannot drift apart on what a terminal must not see raw (U+2029 is now pinned too). Two C-DEFECTs are not reproduced and are ledgered: an Internet row keeps a `name` (134), and DEL, C1 and U+2028/2029 stay escaped (135); uid is unsigned (68, closed as C-DEFECT, the USER column's digits too). CLI: -F selects JSON's keys in either order (a later -F no longer replaces -J); with no selection the C's default set, flags forced on (hex only after a last -f G) and every tcp_info member, so Linux reads the queues and -J -T is -J. The post-loop errors are the C's, in its order and joined: -o/-s, -F and -t (now for -F too, closing 90), -x, -J and -j, -J/-j and -t; +J/+j say "+J is not supported" and the parse reads on. -j -r prints the C's blank line after each cycle (137). Consumers migrated with it: oracle_diff.py (endpoints from name, state from tcp_info) and its tests, capture.ps1, windows_live.rs, both smoke tests. oracle_diff.py's -F parser also stops letting a TQR=/TQS=/TRW= token overwrite the TST= state, which made a -Tq capture compare queue numbers as states. CHANGELOG's link references point [Unreleased] at lsof-rs-v2.0.0...HEAD and add [2.0.0]; the tag is the maintainer's. Fixed on the way: /proc/<pid>/status is split on '\n' alone. str::lines() also dropped a '\r' before it, so a process named "tail\r" was listed as "tail" and `-c 'tail\r'` found nothing, where the C keeps the CR (fixture CR, three differential cases, a parse_status test; the DIVERGENCES fuzz note that said lines() splits only on '\n' is corrected). Measured and left open: the C names a row a file argument locates by that argument as typed (`lsof fx/reg.txt` prints `fx/reg.txt`, lsof-rs the absolute path). It is row 17's mechanism, a DECISION, so it is ledgered as OPEN row 139 with two cases rather than reversed here. Gates: golden tests for each rule; args tests for the refusals, order, selection, tcp_info members and flags; fuzz target render_json (one line per file, no raw control, strings decoded back exactly); 72 json-* differential cases, every lsof-rs JSON output parsed by linux_diff.py, which now passes linux-rules.json (the kit's defaults plus lsof_version, checked by --self-test); mutants/json-schema.toml. New DIVERGENCES rows 134-139; rows 17, 29, 93 amended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWsXG6DnuShZ7NgrM5xxb1
kj299
pushed a commit
to kj299/lsof
that referenced
this pull request
Oct 17, 2026
…1, 90, 68) lsof-rs printed a JSON schema of its own: `user` for uid and login, `node` for inode and protocol, `size` a number, `local`/`remote`/`state` keys, no `lsof_version`, `pgid`, `offset`, `flags` or `tcp_info`, and a task's object the same as its process's. The C has had -J/-j since its PR lsof-org#353, and a script moved between the two broke. This is a breaking change, so the version is 2.0.0 (Cargo.toml, both lock files, the bug template); no tag is created. Model: OpenFile.name_addition (Option<NonZeroU16>, in the row's padding: it stays 192 bytes) records where NAME's addition starts, the C's Lf->nma. add_name_addition() appends it, base_name() returns the C's Lf->nm, both checked: a length past the name or inside a character is no addition, never a panic. Every backend addition goes through it: (readlink: E), (stat: E), NOFD's whole name, (-e FS)/(+e FS), the maps rows' (stat: …)/(path …), and Windows' -E endpoints. Prevents JSON names carrying text the C keeps apart. Renderer (render/json.rs, rewritten): the C's envelope, key order and types (print.c:1896-2175), measured key by key against the C over 85 option variants; written into the sink a process at a time. device_number and inode come from fs_device and node raw, so sockets and -e rows print them as the C does; NOFD is {"fd":"NOFD"}; tcp_info on TCP, UDP, UDP-Lite and unix rows only, a unix row's always {} and UDP-Lite's never with a state. The row predicates -F, JSON and the table share (node_is_protocol, device_text, access/lock blanks, tcp_info_socket) live once in render/mod.rs, and a property test holds every JSON key to its -F letter. JSON escapes with the table's own is_unprintable_unicode, so the two cannot drift apart on what a terminal must not see raw (U+2029 is now pinned too). Two C-DEFECTs are not reproduced and are ledgered: an Internet row keeps a `name` (134), and DEL, C1 and U+2028/2029 stay escaped (135); uid is unsigned (68, closed as C-DEFECT, the USER column's digits too). CLI: -F selects JSON's keys in either order (a later -F no longer replaces -J); with no selection the C's default set, flags forced on (hex only after a last -f G) and every tcp_info member, so Linux reads the queues and -J -T is -J. The post-loop errors are the C's, in its order and joined: -o/-s, -F and -t (now for -F too, closing 90), -x, -J and -j, -J/-j and -t; +J/+j say "+J is not supported" and the parse reads on. -j -r prints the C's blank line after each cycle (137). Consumers migrated with it: oracle_diff.py (endpoints from name, state from tcp_info) and its tests, capture.ps1, windows_live.rs, both smoke tests. oracle_diff.py's -F parser also stops letting a TQR=/TQS=/TRW= token overwrite the TST= state, which made a -Tq capture compare queue numbers as states. CHANGELOG's link references point [Unreleased] at lsof-rs-v2.0.0...HEAD and add [2.0.0]; the tag is the maintainer's. Fixed on the way: /proc/<pid>/status is split on '\n' alone. str::lines() also dropped a '\r' before it, so a process named "tail\r" was listed as "tail" and `-c 'tail\r'` found nothing, where the C keeps the CR (fixture CR, three differential cases, a parse_status test; the DIVERGENCES fuzz note that said lines() splits only on '\n' is corrected). Measured and left open: the C names a row a file argument locates by that argument as typed (`lsof fx/reg.txt` prints `fx/reg.txt`, lsof-rs the absolute path). It is row 17's mechanism, a DECISION, so it is ledgered as OPEN row 139 with two cases rather than reversed here. Gates: golden tests for each rule; args tests for the refusals, order, selection, tcp_info members and flags; fuzz target render_json (one line per file, no raw control, strings decoded back exactly); 72 json-* differential cases, every lsof-rs JSON output parsed by linux_diff.py, which now passes linux-rules.json (the kit's defaults plus lsof_version, checked by --self-test); mutants/json-schema.toml. New DIVERGENCES rows 134-139; rows 17, 29, 93 amended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWsXG6DnuShZ7NgrM5xxb1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
-Jflag for nested JSON output (single JSON object with processes→files hierarchy)-jflag for flat JSON Lines output (one denormalized JSON object per open file, one per line)-Ffield selection mechanism for choosing which fields to includeMotivation
The legacy
-Ffield output uses single-character field IDs with newline/NUL terminators. While machine-parseable, it requires custom parsers. Native JSON enables direct ingestion into logging pipelines (Datadog, Splunk, Elastic Stack) and seamless parsing by Python, Go, Nushell, andjq.Changes
src/store.cFjson,Fjsonl,Fjson_first_procglobal flagslib/common.hsrc/main.c-J/-j, mutual exclusivity validation, default field selectionsrc/print.cjson_print_file(),json_print_proc_fields(), JSON branch inprint_proc(), envelope functionslib/proto.hjson_open_envelope/json_close_envelopelib/dialects/*/Makefileversion.hdependency forprint.oLsof.8-Jand-jtests/case-30-json-output.bash-Jnested JSONtests/case-31-jsonl-output.bash-jJSON LinesMakefile.amDesign decisions
size,offset,inodeemitted as JSON strings to avoid IEEE 754 precision losstcp_info: {state, recv_queue, send_queue, ...}in both formatsTest plan
-Jproduces valid JSON (parseable bypython3 -m json.tool)-jproduces valid JSON Lines (each line parseable)-Fpcfnlimits output to selected fields only-Jand-jare mutually exclusive (error if both specified)-J/-jand-tare mutually exclusive{"lsof_version":"...","processes":[]}for-J, empty for-j)+J/+jproduce error messages