Skip to content

Document full-text paging and configurable search limits - #739

Merged
kriszyp merged 54 commits into
mainfrom
kyle/docs-fulltext-continuation
Oct 28, 2026
Merged

kriszyp merged 54 commits into
mainfrom
kyle/docs-fulltext-continuation

Conversation

@kylebernhardy

@kylebernhardy kylebernhardy commented Oct 20, 2026 •

Copy link
Copy Markdown
Member
Kyle fills search result pages using adjustable card trays

⊙ Problem

🤔 Scope

The engine upgrade changes persisted indexes and matching behavior. Updating the existing search references and release notes keeps the operator guidance with the feature.

Users adopting Harper v5.4 need clear paging guidance and must plan for the fulltext 0.7.0 / Tantivy 0.27.0 index rebuild, stemming changes and unsupported direct downgrade to fulltext 0.5.x.

💡 Solution

Keep paging, budgets and errors in their existing references. Add the engine/rebuild release warning; the native guide supplies the broader declaration and operator walkthrough.

⚖️ Alternatives

Adopt the planning review’s smaller approach: one detailed upgrade runbook in the native guide, with reference links once it is present. Publish the independently correct reference companion before the guide. No new migration page, continuation API or generated contract artifact.

🔧 Changes

✅ Verification

Exact standalone reference (410 documents) and combined guide (414) builds, whole-repository formatting, typecheck and workflow tests pass. Generated upgrade/recovery anchors were checked. Core upgrade tests support the documented JSON error and rebuild behavior; no new runtime or performance qualification is claimed. Lint and all 14 example/fixture checks also pass against the retained core fixture. Fresh validation, lint, formatting, type and workflow checks pass. The preview job built successfully, then the target server returned HTTP 500 (Unknown system error -122, write). Tracked in PR preview deploys fail on every PR: the preview host ran out of disk quota and now drops TLS connections; these runs do not establish its current root cause.

Review and release scope

Claude and Composer completed full review with domain adjudication at 3640af2; closing-full is done. Final direct wording fixes and the reviewed main-merge conflicts were checked locally with a second reader, without another outside pass. Canonical convergence remains unclaimed. The docs target the planned v5.4 adoption; confirm Upgrade full-text search to Tantivy 0.27 is included before merging/releasing these docs. Harper 5.3.0/fulltext 0.3.0 remains outside the qualified procedure and is tracked in Implement full-text generation lifecycle, rebuild, eviction, backup, and status.

Historical verification and review history (through October 26)

Main integration verified October 26

Merged current main into this branch and resolved the v5.4 release-note conflict by retaining both the full-text paging section and main's pointer-compression addon section. Paging behavior, limits and restart requirements are unchanged.

Standalone build (410 documents), repository-wide formatting, typecheck and workflow tests pass. The exact updated native-guide and paging branches merge cleanly and pass the combined build (414 documents), formatting, types and workflow tests. Updated core verification passes 128 focused tests (two LMDB-only skips), 32 real-native tests, source TypeScript worker loading, and all 21 deployed full-text/example tests independently under npm 11.

The final Claude/Composer review and domain adjudication completed at this merge head (13 rounds / 10 full cumulatively). No conflict-resolution blocker was identified; the canonical result remains non-converged with minor completeness/prose concerns. The declaration and example contract is supplied by the coordinated native guide. Existing freshness/recovery discoverability and native-limit ownership tradeoffs remain recorded below; unrelated console-default and archived-v4 questions are outside this update. The previously adopted framing dispositions are retained, not replaced with a new planning-clearance claim.

The overlapping Document nested relationship filters and pagination uses the same inclusive-start/exclusive-end range semantics; current core computes the two-argument limit as end minus offset. The schema-sharding companion also corrects the audit default; retain both its placement guidance and the full-text audit prerequisite when integrating it. Publication ordering remains one-way: this reference-owner PR before the native guide. No PR merge or new review request was made.

Earlier verification and review history

Kris approved the prior head and left seven inline follow-ups plus an unanchored audit-example nit. The follow-up corrects all eight: replicated restart exceptions, 400 versus 503 budget failures, bare unavailable/wait-path 503s, all six negated comparators, a directly available describe_table readiness reference, rejected pre-v5.4 prefix windows, verified native version wording, and audit-enabled logging examples. Each inline thread has a source-grounded response and is resolved. No approval of this newer head is inferred from the earlier review.

❓ Your call: Post-review delta: five documentation files, 55 insertions and 21 deletions, not sent through another outside-model round. These are narrow factual corrections requested by the human review, including an unchanged copy of the companion guide’s previously reviewed metadata reference and two audit-example values. No runtime API, dependency or implementation behavior changed. The standalone production build (410 documents), exact combined build with the guide (414), formatting, typecheck and companion-workflow tests pass. The canonical review and earlier adopted framing dispositions remain recorded below; no new convergence or planning-clearance claim is made.

Review closeout on the paired guide clarifies the explicit audit prerequisite, consumes freshness results without buffering, separates generic describe_table metadata from full-text metadata, removes obsolete prerelease guidance and explains POST versus GET and streaming failures. The exact final guide and paging branches pass the combined production build (414 documents), formatting and typecheck. Fresh unchanged-core verification passes 119 focused tests (two LMDB-only skips in RocksDB mode) and 21 deployed full-text/example integration tests without skips.

❓ Your call: Reconciliation verification: version-qualified prefix paging, shared error contracts, configuration ownership and coordinated publication. Both individual builds and the combined build pass. All nine defaults match the actual resolver from the final core commit, and all seven documented-example integration tests pass on that commit. Kyle authorized new outside review. The first review completed, and the narrow response-overflow and audit-reference findings were corrected; the planning alternative to use one existing dependency marker was adopted in the broader guide. Final outside review and domain adjudication completed (12 rounds / nine full cumulatively), without canonical convergence. The protocol-specific worker-code, readiness-inspection and runtime-release-note findings were then corrected within reviewed files and rebuilt. Declaration/examples/freshness-reference findings are supplied by the broader guide in the verified combined tree. Numeric defaults and native limits keep their existing owners; unrelated logging.console and archived-v4 findings remain outside this change.

Production build passes and processes 410 documents; repository-wide formatting passes after the paging and configuration clarification. The clarification and nine defaults were checked against the final paging core commit, including the accepted-record window, replay work and explicit budget-error contract. Default values and request/runtime names were checked against core's defaultConfig, validation, policy resolver and native binding. Error statuses were checked against query error mapping; the table is explicitly common failures, with 413 / other 500 behavior described separately. Local anchors and the public v0.6.0 native-limits link were checked; the build alone is not claimed to validate every anchor.

Earlier core verification covered 101 focused tests and all 14 deployed HTTP tests on both Node 24.16 and CI Bun 1.4.3. Its runtime fixture separately verifies unchanged captured budgets across replacement workers after configuration changes. The HTTP tests alone are not claimed to prove that negative restart boundary.

Earlier coverage comprised 10 rounds, eight full, with Claude, Cursor Composer/Muse and completed domain adjudication. Native Gemini CLI supplemented that earlier state and raised no documentation defects. The current cumulative canonical coverage is 12 rounds, nine full, as recorded above and in the generated footer. The last planning review did not clear automatically: its concrete placement, error, audit and reservation findings were adopted as described above; a generic build-source checker and product-wide error redesign were deferred rather than claimed complete. Native Gemini is supplemental coverage, not a canonical Gemini receipt.

Verify maxHeaderSize defaults across Harper HTTP configuration references remains an unrelated accuracy question. No version4 docs or ambiguous console default were changed. Before this main update, the core and documentation heads passed their active CI checks; all three bot review threads are resolved, and no new actionable feedback was found. The paging overlap with the broader full-text guide has been reconciled locally; the native-guide review comments have since been addressed with code-grounded replies and all inline threads resolved. Its older changes-requested review still requires a fresh human review; thread resolution is not an approval. Kyle authorized moving this PR out of Draft after readiness checks clear. Neither PR may be merged by this task.

Comment generated by kAIle (GPT-6)

Release target: main — reference documentation for v5.4 full-text paging and limits.

Related PRs: #3680 overlaps (Upgrade full-text search to Tantivy 0.27, engine adoption), #691 overlaps (Document native full-text search, coordinated publication), #736 overlaps (Document nested relationship filters and pagination, consistent range semantics), #753 overlaps (Document schema-defined sharding and regional placement, shared audit-default correction), #671 overlaps (Add production reliability guides for Harper operators, shared reference edits), #733 overlaps (docs: clarify RocksDB no-fsync durability and LMDB-only writeAsync, shared reference edits), #743 overlaps (Document the operations API source allowlist (set_network_access, get_network_access), shared reference edits), #749 overlaps (Document native concurrency controls and their consistency limits, shared reference edits), #756 overlaps (Document the seal lockdown modes and the seal-after-load default, shared reference edits)

Complexity: medium

Deferred findings

Review-Coverage: authored=codex; ran=cursor-composer,cursor-muse,claude; adjudicated=domain; declined=gemini,cursor-grok,cursor-kimi,muse; rounds=15; full=11 @ 94bb9a2

Review-Attention: skim ~2m (decisions: inline-runbook-then-link, upgrade-notes-scope, native-limits-external, fulltext-config-placement, restart-hedge, merge-not-squash) @ 94bb9a2

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive documentation for the new Full-Text Paging feature (continuation paging) in version 5.4.0, including a new dedicated guide, configuration options, REST header details, and sidebar integration. The review feedback suggests improving the formatting and placement of the <VersionBadge> component across several files to avoid awkward usage in running prose, recommending plain text or standalone placement under headings instead.

Comment thread reference/configuration/options.md Resolved
Comment thread reference/rest/headers.md Resolved
Comment thread reference/resources/full-text-paging.md Resolved
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@github-actions
github-actions Bot temporarily deployed to pr-739 October 20, 2026 16:10 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@kylebernhardy
kylebernhardy marked this pull request as ready for review October 21, 2026 22:39
@kylebernhardy
kylebernhardy requested a review from a team as a code owner October 21, 2026 22:39
@github-actions
github-actions Bot temporarily deployed to pr-739 October 21, 2026 23:53 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@kriszyp kriszyp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, looks good!
🤖 Reviewed with Claude


Proposed inline comments (anchors failed):

  • reference/logging/configuration.md:342: [nit] The "Complete Example" still sets auditLog: false. This PR changes the example in options.md to true and adds full-text indexing to the features that need the transaction log (reference/database/transaction.md:26). Anyone who copies the complete example disables audit logging and can no longer register full-text indexes (outside-this-diff)

Comment thread reference/operations-api/operations.md Resolved
Comment thread reference/rest/querying.md Resolved
@kylebernhardy

kylebernhardy commented Oct 24, 2026 •

Copy link
Copy Markdown
Member Author

Thanks, Kris. All seven inline comments are now addressed in f1f55da with individual source-grounded replies. I also fixed the unanchored logging example nit: both auditLog snippets now use true, including Complete Example, and the text explains the full-text audit prerequisite. Updated logging examples.

The standalone production build passes (410 documents). The exact combined tree with Document native full-text search passes a 414-document build, formatting, typecheck and all companion-workflow tests. These are documentation corrections to the existing implementation; no runtime or dependency change was made.

Comment generated by kAIle (GPT-6)

@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

@github-actions
github-actions Bot temporarily deployed to pr-739 October 24, 2026 14:20 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739

This preview will update automatically when you push new commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants