[kylebernhardy]

⊙ Problem

🤔 Scope

The engine upgrade changes persisted indexes and matching behavior. Updating the existing search references and release notes keeps the operator guidance with the feature.

Users adopting Harper v5.4 need clear paging guidance and must plan for the fulltext 0.7.0 / Tantivy 0.27.0 index rebuild, stemming changes and unsupported direct downgrade to fulltext 0.5.x.

💡 Solution

The native guide owns the upgrade checklist and recovery procedure. The included reference companion owns request/native budgets and restart requirements; API references link to those owners.

⚖️ Alternatives

Adopt the planning review’s smaller approach: one detailed upgrade runbook in the native guide, with reference links once it is present. Publish the independently correct reference companion before the guide. No new migration page, continuation API or generated contract artifact.

🔧 Changes

Paging compatibility

Version	Prefix paging	Between pages
Before v5.4	Offset plus limit cannot exceed 100	Each page is a separate query
v5.4	Bounded pages can pass native windows within request budgets	Concurrent changes can cause repeated or omitted records

• CONTRIBUTING.md — Version and storage-engine annotation conventions.
• learn/developers/full-text-search-product-catalog.mdx — Executable catalog tutorial, local freshness and retry guidance.
• reference/backups/overview.md — Derived-index rebuilds after restore; read availability and write lag.
• reference/components/javascript-environment.md — Full-text JavaScript API discovery.
• reference/configuration/operations.md — Worker versus process restart requirements and configuration updates.
• reference/configuration/options.md — Request/native budgets, defaults and corrected audit default.
• reference/configuration/overview.md — Links to the restart contract owner.
• reference/database/overview.md — Native full-text feature discovery.
• reference/database/schema.md — Field declarations and explicit audit prerequisite.
• reference/database/storage-tuning.md — Versioned tuning guidance without duplicated defaults.
• reference/database/transaction.md — Audit prerequisite and retained-history requirements.
• reference/full-text-search/configuration.md — Schema options, rebuild triggers and permission-filtered metadata.
• reference/full-text-search/overview.md — Engine versions, rebuild checklist, ranking changes and recovery.
• reference/full-text-search/querying.md — Modes, permissions, scoring, highlights, freshness and paging.
• reference/graphql-querying/overview.md — GraphQL boundary and supported full-text entry points.
• reference/index.md — Native guide navigation.
• reference/logging/configuration.md — Audit-enabled examples and table-level requirement.
• reference/operations-api/operations.md — Comparators, readiness permissions and restart exceptions.
• reference/resources/query-optimization.md — Paging budgets, local coverage, common errors and recovery links.
• reference/resources/resource-api.md — Positive and negated full-text comparator contract.
• reference/rest/headers.md — Search metadata header discoverability.
• reference/rest/querying.md — Comparator syntax, paging/counts and HTTP-200 terminal errors.
• release-notes/v5-lincoln/5.3.md — Consolidated native feature release notes.
• release-notes/v5-lincoln/5.4.md — Prominent upgrade warning, engine/stemming changes and paging limits.
• sidebarsLearn.ts — Catalog tutorial navigation.
• sidebarsReference.ts — Native guide navigation.

✅ Verification

Exact standalone reference (410 documents) and combined guide (414) builds, whole-repository formatting, typecheck and workflow tests pass. Generated upgrade/recovery anchors were checked. Core upgrade tests support the documented JSON error and rebuild behavior; no new runtime or performance qualification is claimed. Lint and all 14 example/fixture checks also pass against the retained core fixture. Fresh validation, lint, formatting, type and workflow checks pass. The preview job built successfully, then the target server returned HTTP 500 (Unknown system error -122, write). Tracked in PR preview deploys fail on every PR: the preview host ran out of disk quota and now drops TLS connections; these runs do not establish its current root cause. The companion check waits for PR 739 to merge.

Review and release scope

Claude and Composer completed full review with domain adjudication at 48d547b; closing-full is done. Final direct wording fixes and the reviewed main-merge conflicts were checked locally with a second reader, without another outside pass. Canonical convergence remains unclaimed. The docs target the planned v5.4 adoption; confirm Upgrade full-text search to Tantivy 0.27 is included before merging/releasing these docs. Harper 5.3.0/fulltext 0.3.0 remains outside the qualified procedure and is tracked in Implement full-text generation lifecycle, rebuild, eviction, backup, and status.

Historical verification and review history (through October 26)

Main integration verified October 26

Merged current main and resolved the schema-table conflict by retaining its replica-expiration controls beside the full-text audit prerequisite. Main's backup and native-addon guidance is preserved. No runtime API or dependency changed.

Standalone and exact paired-tree builds pass (414 documents), as do formatting, typecheck and workflow tests. The paging companion independently builds 410 documents. Updated core verification passes 128 focused tests (two LMDB-only skips), 32 real-native tests, source TypeScript worker loading, and 21 deployed full-text/example tests independently under npm 11. The broader core gates have failures recorded in the core PR; this is not a whole-suite-green claim.

Fresh CI: validation passes, but the preview deployment job failed after the production build, 45.4 MiB upload and origin-node deployment succeeded. Replication to the peer failed with “WebSocket was closed before the connection was established” (deployment d63d60df-c9d4-4600-b10b-116828b4ff7f). This is a deployment/peer-connection failure, not a failed MDX/build check. No replication-error bypass, retry or deployment-setting change was made. The companion status remains pending and the prior human changes-requested review remains in place.

Final Claude/Composer review and domain adjudication completed on merge commit d9f1b2e (15 rounds / nine full cumulatively), with no conflict-resolution blocker. The canonical result remains non-converged. Its configuration/default/release-note concerns are supplied by the explicitly linked paging companion, whose exact combined tree was built. Four confirmed guide findings were then corrected: local coverage does not wait for inter-node replication; replicas require the schema declaration as well as row transactions; describe_table metadata filters unreadable fields; and Table.clear() returns 501 until full-text declarations are removed. Remaining editorial placement/style nits do not change the contract.

❓ Your call: Final-artifact delta after that review: four insertions and three deletions in four already-reviewed documentation files, not sent through another outside-model round. These are narrow source-backed corrections to that review, with no runtime or API change. Standalone and paired builds, formatting, types and workflow tests pass after the corrections. Earlier adopted framing decisions remain recorded below; no new planning clearance or canonical convergence is claimed.

The range wording in Document nested relationship filters and pagination agrees with this guide and the unchanged core parser. Document schema-defined sharding and regional placement also touches the schema table; retain its placement guidance and this guide's audit prerequisite when combining it. The publication dependency remains one-way: Document full-text paging and configurable search limits before this guide. It is a companion status, not a claim that branch protection makes an out-of-order merge impossible. No PR merge, auto-merge or new reviewer request was made.

Earlier verification and review history

The final guide head and paging companion merge cleanly and pass a combined 414-document production build, typecheck and formatting. Fresh unchanged-core verification passes 119 focused schema/runtime/query tests (two LMDB-only skips in RocksDB mode) and all 21 deployed full-text/example integration tests.

❓ Your call: Final-artifact delta after the completed full review of 06b8ff0: seven already-reviewed files, 22 insertions and 20 deletions, not re-reviewed. These are narrow corrections requested by that review plus restoration of unrelated formatting to main; there is no runtime, API or schema change. Individual and exact combined-tree builds, typecheck and formatting pass. The canonical result remains non-converged: audit defaults, budget references and v5.4 release notes are supplied by the explicit paging dependency. Count and header guidance were narrowed, and the streaming, range-version and dictionary-link findings were corrected. Earlier framing alternatives remain resolved by the recorded adoption; no clean clearing planning verdict is claimed.

• Production build: 414 documents; typecheck and repository-wide formatting pass.
• Paging companion build: 410 documents; repository-wide formatting passes.
• The exact combined branches merge without conflicts and pass production build (414 documents), typecheck and formatting.
• Historical prefix-window checks were traced to the core base; current paging semantics are covered by the core companion's passing CI, including all Node 22/24/26 resource and LMDB stages and native full-text tests on Linux, macOS and Windows.
• All seven existing documented-example integration tests were rerun and pass on the final paging core commit, covering declarations and Table.search, REST and Operations API query shapes. This is not a new whole-feature qualification run.
• A static compatibility check found two unsupported schema declarations before the fix and zero afterward against the exact released and final-core option contracts.
• A narrow combined-tree check evaluated the actual core resolver and verified all nine documented defaults, the paging owner anchor, 400/503/413 guidance, the 5.4 paging release note, and one 5.3 full-text heading.
• Kyle authorized outside review of both documentation diffs. Claude and Cursor reviewed the reconciliation; their unsupported-schema, duplicate-heading, error-status and fuzzy-prefix wording findings were corrected. Final outside review and domain adjudication completed (14 rounds / eight full cumulatively), but the canonical result remains non-converged. The retained major is the isolated-branch configuration gap supplied by the paging companion; the marker below and exact combined-tree verification record that dependency. The final narrow corrections make budget errors explicit, add a REST range example, fix date-rule placement, cross-link the error summary, and restore badge styling exactly to main. The latest closeout also corrects the REST version attribution, links dictionary fields directly, explains terminal streaming failures, separates table counts from full-text match totals, scopes coverage guidance to full-text, and removes unrelated list/badge restyling. Those corrections address reviewed findings inside reviewed files and were verified locally, without another full round.
• All 12 human inline threads now have factual replies and are resolved: the freshness loop no longer buffers records; describe_table has a generic introduction and separate full-text subsection; obsolete prerelease guidance is removed; highlighting, Tantivy, audit defaults and POST versus GET are explained; unsupported filter metadata and duplicate release notes are removed. The prior changes-requested review remains until a human reviewer updates it. No new approval, reviewer request or merge is claimed.

Comment generated by kAIle (GPT-6)

Release target: main — native full-text guide with version-qualified v5.4 paging.

Related PRs: #3680 overlaps (Upgrade full-text search to Tantivy 0.27, engine adoption), #739 overlaps (Document full-text paging and configurable search limits, publication dependency), #753 overlaps (Document schema-defined sharding and regional placement, shared schema table), #736 overlaps (Document nested relationship filters and pagination, consistent range semantics), #671 overlaps (Add production reliability guides for Harper operators, shared reference edits), #733 overlaps (docs: clarify RocksDB no-fsync durability and LMDB-only writeAsync, shared reference edits), #743 overlaps (Document the operations API source allowlist (set_network_access, get_network_access), shared reference edits), #749 overlaps (Document native concurrency controls and their consistency limits, shared reference edits), #756 overlaps (Document the seal lockdown modes and the seal-after-load default, shared reference edits), 10 others independent

Publication dependency: Document full-text paging and configurable search limits supplies configuration and 5.4 release-note owners before this native guide. The combined tree is the verified publication result. The one-way companion status waits for that PR; neither PR is a standalone replacement for the other. No merge or auto-merge is authorized.

Depends-on: #739

Complexity: medium

_{Review-Coverage: authored=codex; ran=claude,cursor-composer,cursor-grok; adjudicated=domain; blocked=gemini(auth); declined=cursor-kimi,cursor-muse,muse; rounds=17; full=10 @ 58456b3}

_{Review-Attention: read ~4m (decisions: carry-739-content, inline-version-split-prose, contributing-badge-policy, auditlog-default-fixed-here) @ 58456b3}

[gemini-code-assist]

Code Review

This pull request introduces comprehensive documentation for the new native BM25-ranked Full-Text Search feature in Harper v5.3.0, including a new developer guide, configuration references, querying guides, and updates to existing database, REST, GraphQL, and Operations API documentation. The review feedback highlights several style and formatting improvements: separating a critical security warning into distinct sentences for better visibility, replacing the <VersionBadge> component with plain text when used mid-sentence in prose, and replacing em dashes with colons as field separators in Markdown lists.

[cb1kenobi]

Native full-text documentation is consistent on authoritative versus derived state, REST and JavaScript permission boundaries, rebuild and lag error codes, and prefix-window limits. Existing inline comments already cover VersionBadge placement, em dashes, and the learn-guide security-warning wording. Nothing remaining on the changed lines is a confirmed blocking defect.

—
Reviewed 45c1c5d

[cb1kenobi]

The field-only FullText schema, REST versus trusted in-process permission boundary, rebuild and lag error codes, and backup or log-retention recovery guidance are consistent across the new pages and updated API references. Existing discussion already covers VersionBadge placement, em dashes, and the older learn-guide warning wording. Nothing remaining on the changed lines is a confirmed blocking defect.

—
Reviewed 6855bed

[kriszyp]

The Resource code example basically ignores all of our best practices. It should:

• Not use loadAsInstace
• Use static methods
• Don't accumulate search results in an array (memory hazard, slower latency).

And I don't know why you would need to set audit: true, that's the default.

_{🤖 Reviewed with Claude}

[cb1kenobi]

No additional blocking findings were confirmed. Existing discussion already covers the product-guide issues found at this commit.

—
Reviewed 6855bed

[kylebernhardy]

Thanks, Kris. All four points are addressed:

• Highlight behavior now matches core: every configured source can match and rank, while only fields with highlight: true appear in $highlights. The redundant highlight switch was removed from the example.
• The custom resource now uses the static Resource pattern, checks the authenticated caller in post, passes context, and opts into table and source-field authorization with checkPermission: true. The Resource API now documents this opt-in and makes clear that passing context alone is not enough.
• The freshness example consumes the async iterator, which is when coverage is checked.
• The engine badge now sits with the version badge at the top of the page.

The accuracy pass also tightened the audit migration path, all five readiness states, write backpressure, rebuild and replica behavior, Blob limits, and the handling of derived Tantivy files.

Validation is green: formatting, typecheck, workflow tests, production docs build (411 documents), 11 focused Harper integration tests, and 96 focused Harper unit tests.

Comment generated by kAIle (GPT-6 Astra)

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-691

This preview will update automatically when you push new commits.

[kylebernhardy]

Executable coverage for the published schema, JavaScript, REST, and Operations API examples is now in Harper PR #2915, "Test documented full-text examples". The test fixture also pins permissions, index coverage, reduced-index metadata, highlight limits, synonyms, multiple indexes, and Blob sources.\n\nComment generated by kAIle (GPT-6 Astra)

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-691

This preview will update automatically when you push new commits.

[kylebernhardy]

Thanks for the nudge, Ethan. The remaining inline concerns have now been checked against the current core implementation, addressed, and resolved with individual replies. The guide fixes cover audit-default wording, freshness iteration without buffering, generic describe_table layout, unsupported filter metadata, duplicate release notes, POST versus GET, highlighting, Tantivy and obsolete prerelease guidance. The final cleanup also clarifies streaming errors and count behavior and removes unrelated formatting churn.

The individual and combined documentation builds pass (414 documents), as do typecheck and formatting. Fresh core verification passes 119 focused tests (two LMDB-only skips in RocksDB mode) and 21 deployed full-text/example tests without skips. Claude and Cursor completed another outside review; its paired-reference findings remain covered by Document full-text paging and configurable search limits, which this guide explicitly depends on. The older changes-requested review still needs a fresh human assessment of the updated head.

Comment generated by kAIle (GPT-6)

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-691

This preview will update automatically when you push new commits.

[kriszyp]

_{🤖 Reviewed with Claude}

[kriszyp]

minor — "Rejected records do not count toward the requested page" has no version qualifier, but the tutorial is badged v5.3.0 and the reference says this only holds from v5.4.

[kriszyp]

nit — the example labelled "relevant fragment" leaves out stop_words, surface_terms, synonyms and highlighting, and doesn't say the fields were left out.

[kriszyp]

[minor] Unavailable-totals lists full-text queries as always reporting *. Per reference/rest/querying.md:99 and reference/full-text-search/querying.md:84, that only holds with exactCount: true; otherwise the request is downgraded to count=estimated and can return a number. A client built from line 247 handles the estimate case wrong. Qualify the full-text case to "with exact counts enabled".

_{(reference/rest/querying.md:247 is not part of this PR's diff — anchored to the nearest line this PR's diff can hold)}

[kriszyp]

[nit] "Only the directive options listed above are accepted; an unknown option returns 400" sits under Structured filters. There it reads as a rule about query-condition properties. Move it under Directive options.