Skip to content

Document full-text paging and configurable search limits - #739

Open
kylebernhardy wants to merge 50 commits into
mainfrom
kyle/docs-fulltext-continuation
Open

kylebernhardy wants to merge 50 commits into
mainfrom
kyle/docs-fulltext-continuation

Conversation

@kylebernhardy

@kylebernhardy kylebernhardy commented Oct 20, 2026 •

Copy link
Copy Markdown
Member
Kyle fills search result pages using adjustable card trays

⊙ Problem

The earlier draft described a separate full-text continuation API. Fill full-text and autocomplete pages using Harper's existing paging now fills ordinary limit/offset pages and removes that API.

❓ Your call: Keep paging guidance in the existing REST and query references, as requested, with no separate continuation API or new guide.

💡 Solution

Explain paging, limits and recovery in the existing references. There are no continuation functions, tokens, close operations or retained searches, and no separate paging page or sidebar entry. Each page runs a new query and can see later changes.

⚖️ Alternatives

❓ Your call: Framing-Verdict: option-set-too-narrow (0364589df96e) — resolved by adopting the concrete smaller structure: one restart-reference owner with links, existing storage/default owners, and source-grounded error/audit guidance. Earlier verdict 9d0c7d605cb1's existing-owner alternative was also adopted. A vendored core-default snapshot checker would create another artifact to synchronize, so it is deferred; product-wide error-code changes belong to core qualification rather than prose.

Document native full-text search owns the broader declaration/query guide. The reconciliation qualifies its historical 100-record prefix limit as pre-v5.4 and gives both guides the same independent-page and work-budget semantics. The two branches merge cleanly and their combined documentation build passes; numeric defaults and restart requirements remain in the existing configuration references. The broader guide also removes unsupported filterFields declarations and consolidates its duplicate 5.3 release-note sections. Publish this reference-owner change before the broader guide. Only the broader guide carries a Depends-on marker, pointing here; no reciprocal marker is added.

❓ Your call: Framing-Verdict: better-alternative-exists (b1934a2d4c64; c9884425285f) — resolved by adopting one existing Depends-on marker in the broader guide, pointing here. No reciprocal marker or repository-wide anchor policy is added. The combined tree is checked against exact core contracts; no clean clearing planning verdict is claimed.

🔧 Changes

  • reference/configuration/options.md — Owns all three request defaults and six native process budgets; corrects auditLog to the source default true.

  • reference/configuration/operations.md — Owns the request-worker versus native-process restart contract, states that full-text updates use individual parameter names rather than a whole storage_fullText object, and uses the actual configuration response text.

  • reference/configuration/overview.md — Links file-edit restart guidance to its reference owner.

  • reference/database/storage-tuning.md — Adds a versioned full-text tuning pointer before Related, without repeating numeric defaults.

  • reference/database/transaction.md — Includes full-text indexing among transaction-log consumers and warns that per-table audit opt-out prevents full-text index registration.

  • reference/operations-api/operations.md — Makes full-text comparators discoverable in search_by_conditions, documents full_text_indexes/readiness directly under describe_table, and names runtime budgets in the replicated full-process restart exception.

  • reference/resources/query-optimization.md — Explains independent offset requests, an offset-20/limit-10 example counting eligible records, filtering, raw/result/time budgets, native capture, freshness condition fields, common error statuses and recovery with describe_table readiness inspection, audit retention and concurrency costs; also covers response-envelope overflow from multi-hit ordinary batches.

  • reference/resources/resource-api.md — Lists positive and all six negated full-text comparators, requires a positive anchor for negation, and removes a promise of a missing Table.search example.

  • reference/rest/querying.md — Documents positive and negated comparator syntax, filtered page filling, concurrent-write caveats and the difference between exhausted matches and budget errors, plus count behavior including the absence of an exact total for full-text and autocomplete queries.

  • release-notes/v5-lincoln/5.4.md — Describes full-text paging, the three request controls and native process-capacity budgets without introducing a continuation API.

  • reference/logging/configuration.md — Keeps both auditLog examples enabled, including Complete Example, and explains the full-text per-table audit prerequisite.

✅ Verification

Standalone and combined documentation builds, formatting, typecheck and workflow tests pass. CI is green. Publish this reference-owner PR before the native full-text guide.

Verification evidence and review history

Main integration verified October 26

Merged current main into this branch and resolved the v5.4 release-note conflict by retaining both the full-text paging section and main's pointer-compression addon section. Paging behavior, limits and restart requirements are unchanged.

Standalone build (410 documents), repository-wide formatting, typecheck and workflow tests pass. The exact updated native-guide and paging branches merge cleanly and pass the combined build (414 documents), formatting, types and workflow tests. Updated core verification passes 128 focused tests (two LMDB-only skips), 32 real-native tests, source TypeScript worker loading, and all 21 deployed full-text/example tests independently under npm 11.

The final Claude/Composer review and domain adjudication completed at this merge head (13 rounds / 10 full cumulatively). No conflict-resolution blocker was identified; the canonical result remains non-converged with minor completeness/prose concerns. The declaration and example contract is supplied by the coordinated native guide. Existing freshness/recovery discoverability and native-limit ownership tradeoffs remain recorded below; unrelated console-default and archived-v4 questions are outside this update. The previously adopted framing dispositions are retained, not replaced with a new planning-clearance claim.

The overlapping Document nested relationship filters and pagination uses the same inclusive-start/exclusive-end range semantics; current core computes the two-argument limit as end minus offset. The schema-sharding companion also corrects the audit default; retain both its placement guidance and the full-text audit prerequisite when integrating it. Publication ordering remains one-way: this reference-owner PR before the native guide. No PR merge or new review request was made.

Earlier verification and review history

Kris approved the prior head and left seven inline follow-ups plus an unanchored audit-example nit. The follow-up corrects all eight: replicated restart exceptions, 400 versus 503 budget failures, bare unavailable/wait-path 503s, all six negated comparators, a directly available describe_table readiness reference, rejected pre-v5.4 prefix windows, verified native version wording, and audit-enabled logging examples. Each inline thread has a source-grounded response and is resolved. No approval of this newer head is inferred from the earlier review.

❓ Your call: Post-review delta: five documentation files, 55 insertions and 21 deletions, not sent through another outside-model round. These are narrow factual corrections requested by the human review, including an unchanged copy of the companion guide’s previously reviewed metadata reference and two audit-example values. No runtime API, dependency or implementation behavior changed. The standalone production build (410 documents), exact combined build with the guide (414), formatting, typecheck and companion-workflow tests pass. The canonical review and earlier adopted framing dispositions remain recorded below; no new convergence or planning-clearance claim is made.

Review closeout on the paired guide clarifies the explicit audit prerequisite, consumes freshness results without buffering, separates generic describe_table metadata from full-text metadata, removes obsolete prerelease guidance and explains POST versus GET and streaming failures. The exact final guide and paging branches pass the combined production build (414 documents), formatting and typecheck. Fresh unchanged-core verification passes 119 focused tests (two LMDB-only skips in RocksDB mode) and 21 deployed full-text/example integration tests without skips.

❓ Your call: Reconciliation verification: version-qualified prefix paging, shared error contracts, configuration ownership and coordinated publication. Both individual builds and the combined build pass. All nine defaults match the actual resolver from the final core commit, and all seven documented-example integration tests pass on that commit. Kyle authorized new outside review. The first review completed, and the narrow response-overflow and audit-reference findings were corrected; the planning alternative to use one existing dependency marker was adopted in the broader guide. Final outside review and domain adjudication completed (12 rounds / nine full cumulatively), without canonical convergence. The protocol-specific worker-code, readiness-inspection and runtime-release-note findings were then corrected within reviewed files and rebuilt. Declaration/examples/freshness-reference findings are supplied by the broader guide in the verified combined tree. Numeric defaults and native limits keep their existing owners; unrelated logging.console and archived-v4 findings remain outside this change.

Production build passes and processes 410 documents; repository-wide formatting passes after the paging and configuration clarification. The clarification and nine defaults were checked against the final paging core commit, including the accepted-record window, replay work and explicit budget-error contract. Default values and request/runtime names were checked against core's defaultConfig, validation, policy resolver and native binding. Error statuses were checked against query error mapping; the table is explicitly common failures, with 413 / other 500 behavior described separately. Local anchors and the public v0.6.0 native-limits link were checked; the build alone is not claimed to validate every anchor.

Earlier core verification covered 101 focused tests and all 14 deployed HTTP tests on both Node 24.16 and CI Bun 1.4.3. Its runtime fixture separately verifies unchanged captured budgets across replacement workers after configuration changes. The HTTP tests alone are not claimed to prove that negative restart boundary.

Earlier coverage comprised 10 rounds, eight full, with Claude, Cursor Composer/Muse and completed domain adjudication. Native Gemini CLI supplemented that earlier state and raised no documentation defects. The current cumulative canonical coverage is 12 rounds, nine full, as recorded above and in the generated footer. The last planning review did not clear automatically: its concrete placement, error, audit and reservation findings were adopted as described above; a generic build-source checker and product-wide error redesign were deferred rather than claimed complete. Native Gemini is supplemental coverage, not a canonical Gemini receipt.

Verify maxHeaderSize defaults across Harper HTTP configuration references remains an unrelated accuracy question. No version4 docs or ambiguous console default were changed. Before this main update, the core and documentation heads passed their active CI checks; all three bot review threads are resolved, and no new actionable feedback was found. The paging overlap with the broader full-text guide has been reconciled locally; the native-guide review comments have since been addressed with code-grounded replies and all inline threads resolved. Its older changes-requested review still requires a fresh human review; thread resolution is not an approval. Kyle authorized moving this PR out of Draft after readiness checks clear. Neither PR may be merged by this task.

Comment generated by kAIle (GPT-6)

Release target: main — reference documentation for v5.4 full-text paging and limits.

Related PRs: #691 overlaps (Document native full-text search, coordinated publication), #736 overlaps (Document nested relationship filters and pagination, consistent range semantics), #753 overlaps (Document schema-defined sharding and regional placement, shared audit-default correction), 3 others independent

Complexity: medium

Review-Coverage: authored=codex; ran=cursor-composer,cursor-muse,claude; adjudicated=domain; declined=gemini,cursor-grok,cursor-kimi; rounds=13; full=10 @ b5294b0

Review-Attention: skim ~2m (decisions: companion-guide-ordering, native-limit-source, configuration-surface) @ b5294b0

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive documentation for the new Full-Text Paging feature (continuation paging) in version 5.4.0, including a new dedicated guide, configuration options, REST header details, and sidebar integration. The review feedback suggests improving the formatting and placement of the <VersionBadge> component across several files to avoid awkward usage in running prose, recommending plain text or standalone placement under headings instead.

Comment thread reference/configuration/options.md Outdated
Comment thread reference/rest/headers.md Outdated
Comment thread reference/resources/full-text-paging.md Outdated
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@github-actions
github-actions Bot temporarily deployed to pr-739 October 20, 2026 16:10 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@kylebernhardy kylebernhardy changed the title Document full-text and autocomplete snapshot paging Document full-text paging and configurable search limits Oct 21, 2026
@github-actions
github-actions Bot temporarily deployed to pr-739 October 21, 2026 06:14 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@github-actions
github-actions Bot temporarily deployed to pr-739 October 21, 2026 13:27 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@github-actions
github-actions Bot temporarily deployed to pr-739 October 21, 2026 16:38 Inactive
@kylebernhardy
kylebernhardy marked this pull request as ready for review October 21, 2026 22:39
@kylebernhardy
kylebernhardy requested a review from a team as a code owner October 21, 2026 22:39
@github-actions
github-actions Bot temporarily deployed to pr-739 October 21, 2026 23:53 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@kriszyp kriszyp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, looks good!
🤖 Reviewed with Claude


Proposed inline comments (anchors failed):

  • reference/logging/configuration.md:342: [nit] The "Complete Example" still sets auditLog: false. This PR changes the example in options.md to true and adds full-text indexing to the features that need the transaction log (reference/database/transaction.md:26). Anyone who copies the complete example disables audit logging and can no longer register full-text indexes (outside-this-diff)

Comment thread reference/operations-api/operations.md Outdated
Comment thread reference/resources/query-optimization.md Outdated
Comment thread reference/resources/query-optimization.md Outdated
Comment thread reference/rest/querying.md Outdated
Comment thread reference/resources/query-optimization.md
Comment thread reference/resources/query-optimization.md Outdated
Comment thread reference/resources/query-optimization.md Outdated
@kylebernhardy

kylebernhardy commented Oct 24, 2026 •

Copy link
Copy Markdown
Member Author

Thanks, Kris. All seven inline comments are now addressed in f1f55da with individual source-grounded replies. I also fixed the unanchored logging example nit: both auditLog snippets now use true, including Complete Example, and the text explains the full-text audit prerequisite. Updated logging examples.

The standalone production build passes (410 documents). The exact combined tree with Document native full-text search passes a 414-document build, formatting, typecheck and all companion-workflow tests. These are documentation corrections to the existing implementation; no runtime or dependency change was made.

Comment generated by kAIle (GPT-6)

@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@github-actions
github-actions Bot temporarily deployed to pr-739 October 24, 2026 14:20 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

This branch was successfully deployed

1 active deployment
pr-739 — b5294b09 Deployed Oct 26, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants