Repository navigation
Document full-text paging and configurable search limits - #739
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive documentation for the new Full-Text Paging feature (continuation paging) in version 5.4.0, including a new dedicated guide, configuration options, REST header details, and sidebar integration. The review feedback suggests improving the formatting and placement of the <VersionBadge> component across several files to avoid awkward usage in running prose, recommending plain text or standalone placement under headings instead.
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739 This preview will update automatically when you push new commits. |
kriszyp
left a comment
There was a problem hiding this comment.
Cool, looks good!
🤖 Reviewed with Claude
Proposed inline comments (anchors failed):
reference/logging/configuration.md:342: [nit] The "Complete Example" still setsauditLog: false. This PR changes the example inoptions.mdtotrueand adds full-text indexing to the features that need the transaction log (reference/database/transaction.md:26). Anyone who copies the complete example disables audit logging and can no longer register full-text indexes (outside-this-diff)
|
Thanks, Kris. All seven inline comments are now addressed in f1f55da with individual source-grounded replies. I also fixed the unanchored logging example nit: both auditLog snippets now use true, including Complete Example, and the text explains the full-text audit prerequisite. Updated logging examples. The standalone production build passes (410 documents). The exact combined tree with Document native full-text search passes a 414-document build, formatting, typecheck and all companion-workflow tests. These are documentation corrections to the existing implementation; no runtime or dependency change was made. Comment generated by kAIle (GPT-6) |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-739 This preview will update automatically when you push new commits. |
⊙ Problem
🤔 Scope
The engine upgrade changes persisted indexes and matching behavior. Updating the existing search references and release notes keeps the operator guidance with the feature.
Users adopting Harper v5.4 need clear paging guidance and must plan for the fulltext 0.7.0 / Tantivy 0.27.0 index rebuild, stemming changes and unsupported direct downgrade to fulltext 0.5.x.
💡 Solution
Keep paging, budgets and errors in their existing references. Add the engine/rebuild release warning; the native guide supplies the broader declaration and operator walkthrough.
⚖️ Alternatives
Adopt the planning review’s smaller approach: one detailed upgrade runbook in the native guide, with reference links once it is present. Publish the independently correct reference companion before the guide. No new migration page, continuation API or generated contract artifact.
🔧 Changes
✅ Verification
Exact standalone reference (410 documents) and combined guide (414) builds, whole-repository formatting, typecheck and workflow tests pass. Generated upgrade/recovery anchors were checked. Core upgrade tests support the documented JSON error and rebuild behavior; no new runtime or performance qualification is claimed. Lint and all 14 example/fixture checks also pass against the retained core fixture. Fresh validation, lint, formatting, type and workflow checks pass. The preview job built successfully, then the target server returned HTTP 500 (
Unknown system error -122, write). Tracked in PR preview deploys fail on every PR: the preview host ran out of disk quota and now drops TLS connections; these runs do not establish its current root cause.Review and release scope
Claude and Composer completed full review with domain adjudication at 3640af2; closing-full is done. Final direct wording fixes and the reviewed main-merge conflicts were checked locally with a second reader, without another outside pass. Canonical convergence remains unclaimed. The docs target the planned v5.4 adoption; confirm Upgrade full-text search to Tantivy 0.27 is included before merging/releasing these docs. Harper 5.3.0/fulltext 0.3.0 remains outside the qualified procedure and is tracked in Implement full-text generation lifecycle, rebuild, eviction, backup, and status.
Historical verification and review history (through October 26)
Main integration verified October 26
Merged current main into this branch and resolved the v5.4 release-note conflict by retaining both the full-text paging section and main's pointer-compression addon section. Paging behavior, limits and restart requirements are unchanged.
Standalone build (410 documents), repository-wide formatting, typecheck and workflow tests pass. The exact updated native-guide and paging branches merge cleanly and pass the combined build (414 documents), formatting, types and workflow tests. Updated core verification passes 128 focused tests (two LMDB-only skips), 32 real-native tests, source TypeScript worker loading, and all 21 deployed full-text/example tests independently under npm 11.
The final Claude/Composer review and domain adjudication completed at this merge head (13 rounds / 10 full cumulatively). No conflict-resolution blocker was identified; the canonical result remains non-converged with minor completeness/prose concerns. The declaration and example contract is supplied by the coordinated native guide. Existing freshness/recovery discoverability and native-limit ownership tradeoffs remain recorded below; unrelated console-default and archived-v4 questions are outside this update. The previously adopted framing dispositions are retained, not replaced with a new planning-clearance claim.
The overlapping Document nested relationship filters and pagination uses the same inclusive-start/exclusive-end range semantics; current core computes the two-argument limit as end minus offset. The schema-sharding companion also corrects the audit default; retain both its placement guidance and the full-text audit prerequisite when integrating it. Publication ordering remains one-way: this reference-owner PR before the native guide. No PR merge or new review request was made.
Earlier verification and review history
Kris approved the prior head and left seven inline follow-ups plus an unanchored audit-example nit. The follow-up corrects all eight: replicated restart exceptions, 400 versus 503 budget failures, bare unavailable/wait-path 503s, all six negated comparators, a directly available describe_table readiness reference, rejected pre-v5.4 prefix windows, verified native version wording, and audit-enabled logging examples. Each inline thread has a source-grounded response and is resolved. No approval of this newer head is inferred from the earlier review.
Review closeout on the paired guide clarifies the explicit audit prerequisite, consumes freshness results without buffering, separates generic describe_table metadata from full-text metadata, removes obsolete prerelease guidance and explains POST versus GET and streaming failures. The exact final guide and paging branches pass the combined production build (414 documents), formatting and typecheck. Fresh unchanged-core verification passes 119 focused tests (two LMDB-only skips in RocksDB mode) and 21 deployed full-text/example integration tests without skips.
Production build passes and processes 410 documents; repository-wide formatting passes after the paging and configuration clarification. The clarification and nine defaults were checked against the final paging core commit, including the accepted-record window, replay work and explicit budget-error contract. Default values and request/runtime names were checked against core's defaultConfig, validation, policy resolver and native binding. Error statuses were checked against query error mapping; the table is explicitly common failures, with 413 / other 500 behavior described separately. Local anchors and the public v0.6.0 native-limits link were checked; the build alone is not claimed to validate every anchor.
Earlier core verification covered 101 focused tests and all 14 deployed HTTP tests on both Node 24.16 and CI Bun 1.4.3. Its runtime fixture separately verifies unchanged captured budgets across replacement workers after configuration changes. The HTTP tests alone are not claimed to prove that negative restart boundary.
Earlier coverage comprised 10 rounds, eight full, with Claude, Cursor Composer/Muse and completed domain adjudication. Native Gemini CLI supplemented that earlier state and raised no documentation defects. The current cumulative canonical coverage is 12 rounds, nine full, as recorded above and in the generated footer. The last planning review did not clear automatically: its concrete placement, error, audit and reservation findings were adopted as described above; a generic build-source checker and product-wide error redesign were deferred rather than claimed complete. Native Gemini is supplemental coverage, not a canonical Gemini receipt.
Verify maxHeaderSize defaults across Harper HTTP configuration references remains an unrelated accuracy question. No version4 docs or ambiguous console default were changed. Before this main update, the core and documentation heads passed their active CI checks; all three bot review threads are resolved, and no new actionable feedback was found. The paging overlap with the broader full-text guide has been reconciled locally; the native-guide review comments have since been addressed with code-grounded replies and all inline threads resolved. Its older changes-requested review still requires a fresh human review; thread resolution is not an approval. Kyle authorized moving this PR out of Draft after readiness checks clear. Neither PR may be merged by this task.
Comment generated by kAIle (GPT-6)
Release target: main — reference documentation for v5.4 full-text paging and limits.
Related PRs: #3680 overlaps (Upgrade full-text search to Tantivy 0.27, engine adoption), #691 overlaps (Document native full-text search, coordinated publication), #736 overlaps (Document nested relationship filters and pagination, consistent range semantics), #753 overlaps (Document schema-defined sharding and regional placement, shared audit-default correction), #671 overlaps (Add production reliability guides for Harper operators, shared reference edits), #733 overlaps (docs: clarify RocksDB no-fsync durability and LMDB-only writeAsync, shared reference edits), #743 overlaps (Document the operations API source allowlist (set_network_access, get_network_access), shared reference edits), #749 overlaps (Document native concurrency controls and their consistency limits, shared reference edits), #756 overlaps (Document the seal lockdown modes and the seal-after-load default, shared reference edits)
Complexity: medium
Deferred findings
b61768680773482dminor documentation —reference/configuration/options.md— logging.console default contradicts logging reference — filed Reconcile logging.console defaults across configuration references #7691d2f385dfce65f6bnit documentation —reference_versioned_docs/version-v4/rest/querying.md— Archived v4 range syntax remains ambiguous — filed Verify and document archived v4 REST limit endpoint semantics #770Review-Coverage: authored=codex; ran=cursor-composer,cursor-muse,claude; adjudicated=domain; declined=gemini,cursor-grok,cursor-kimi,muse; rounds=15; full=11 @ 94bb9a2
Review-Attention: skim ~2m (decisions: inline-runbook-then-link, upgrade-notes-scope, native-limits-external, fulltext-config-placement, restart-hedge, merge-not-squash) @ 94bb9a2