From d5b50ebec12bd6eaa1d9454482cc5ce862495d01 Mon Sep 17 00:00:00 2001 From: MPCoreDeveloper Date: Sun, 11 Oct 2026 15:03:09 +0200 Subject: [PATCH 1/6] perf(bench): publish the v2.1.0 numbers and fix the three harness defects they exposed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restamp and re-tabulate the six published pages (README, docs/BENCHMARK_RESULTS.md, docs/manual/performance.md §7.1, docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md, docs/benchmarks/default-config-pk.md and the plan's row 10) from one same-run set at 92794084 / net11.0: default posture fair-PK INSERT 1,17x (1,03-1,27) / READ 1,13x (1,12-1,15) / UPDATE 0,53x (0,42-0,53) / DELETE 0,47x (0,46-0,62) against that run's own SQLite, plus the record-layout A/B (fixed width 1,2-1,5x ahead of legacy variable length on UPDATE/DELETE at a 1,2-1,9x per-record-encryption cost) and the document-CRUD ladder. Every published ratio is the same-run one, because SQLite's reference drifts between sessions and single-run absolutes are not comparable across them. The correction the publication carries: the "fair-PK layout ahead of SQLite on UPDATE (0,8x) / DELETE (0,5x)" claims came from the window in which the comparator left the SQLite arm allocating a command and two parameters per row. With one prepared command SQLite's own PK references are 3,05x higher (288.108 -> 878.557 UPDATE, 385.116 -> 1.172.704 DELETE), so those two cells are behind, the earlier reference band is superseded rather than better, and the pages now state the direction the measurement gives (improved from 0,39x / 0,31-0,35x). Two further published figures are corrected with their cause: the inline-capacity default is 24 bytes, not the 16 the README still said (+34 % on the batched multi-row INSERT shape, fair-PK INSERT ratio 0,63x -> 0,87x), and the "top v2.1 priority" line is now the historical sentence it was plus where the ratio stands. Three harness defects, found while publishing and fixed here. BuildConfig's layout switch was `AutoFixedWidthRecords = !fixedWidth`, which cannot express "legacy variable length" because the product default is true, so an arm that merely declined to force the layout still got it for any new PRIMARY KEY table: both layout arms measured ~3 % apart (223.728/562.562 against 230.591/582.384) while the banner promised the layout the plan last measured 2,3x away on UPDATE. It is now an explicit `legacyVariableLength` parameter scoped to that one call site and defaulting to false. The --pk [diag] line printed the requested parameters instead of the resolved config, so the encrypted stock arm read `noEncrypt=True` ("no encryption") on the arm whose whole point is per-record encryption at rest; it now prints config.NoEncryptMode / FixedWidthRecordLayout / AutoFixedWidthRecords (the stock arm reads `noEncrypt=False autoFixedWidth=True atRest=True inline=24`). RunPkMedian builds its own median object and never stamped Reps, so every pk_comparative_*.json archive - the archive the plan publishes beside the fair-PK table so a reader can reproduce the protocol - described a 3-rep median after 8 discarded warm-ups as `"Reps": 1`. Both are guarded red->green by BenchmarkProtocolArchiveTests (Category=Manual, skips when the harness exe is absent): PkLegacyArm_ResolvesToVariableLengthRecords_AsItsBannerClaims and PkComparisonArchive_RecordsTheMeasuredRepCount_NotTheSingleShotDefault. Validation: core suite 1998 / 0 failed / 0 errors / 0 skipped through the MTP exe with CI's filter, test project builds with 0 errors, 2 / 0 on the guards. No engine code, config default, threshold, package or version metadata changed (legacyVariableLength defaults false, so no existing mode's DatabaseConfig moves), no baseline re-recorded, nothing weakened to publish a number. The four archives the pages cite are committed beside them; the rest of the untracked results stay untracked. Local commit on a feature branch, not pushed. --- README.md | 60 ++-- docs/BENCHMARK_RESULTS.md | 59 +++- docs/CHANGELOG.md | 2 + .../SHARPCOREDB_COMPARATIVE_BENCHMARKS.md | 54 +++- docs/benchmarks/default-config-pk.md | 45 +++ docs/manual/performance.md | 72 +++-- .../INSERT_UPDATE_PERFORMANCE_PLAN.md | 2 +- docs/performance/WORKLOG.md | 21 ++ .../BenchmarkProtocolArchiveTests.cs | 271 ++++++++++++++++++ .../Program.cs | 60 +++- .../results/comparative_20261011_125436.json | 72 +++++ .../pk_comparative_20261011_123404.json | 58 ++++ .../pk_comparative_20261011_125346.json | 58 ++++ .../results/pk_default_20261011_125413.json | 30 ++ 14 files changed, 796 insertions(+), 68 deletions(-) create mode 100644 tests/SharpCoreDB.Tests/Benchmarks/BenchmarkProtocolArchiveTests.cs create mode 100644 tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/comparative_20261011_125436.json create mode 100644 tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_comparative_20261011_123404.json create mode 100644 tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_comparative_20261011_125346.json create mode 100644 tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_default_20261011_125413.json diff --git a/README.md b/README.md index 0e29fc28..194720a5 100644 --- a/README.md +++ b/README.md @@ -29,14 +29,19 @@ Use it when you need: > **Current release: v2.1.0 (2026-10-10) — the .NET 11 / C# 15 line, on `master`.** **Published on > NuGet.org** (all 27 packages) from `master`, `net11.0` only. Highlights: the fixed-width inline capacity -> ships at 16 by default on both storage paths (+12–19 % on the batched multi-row INSERT shape, per -> the tracked measurement), the overflow arena appends a row's payloads in one call instead of one -> file open per value, and in-place UPDATE/DELETE with deferred index maintenance put the -> fixed-width fair-PK layout **ahead of SQLite on UPDATE (0.8×) and DELETE (0.5×)**. +> ships at 24 bytes by default on both storage paths (worth **+34 %** on the batched multi-row INSERT +> shape and **0,63× → 0,87×** on the tracked fair-PK INSERT ratio), the overflow arena appends a row's payloads in one call instead of one +> file open per value, and in-place UPDATE/DELETE with deferred index maintenance now run the +> fixed-width fair-PK layout **1,2–1,5× ahead of the legacy variable-length layout** on UPDATE and DELETE. > Validated by **2,897 tests across 16 suites (0 failed, 16 skipped)** and the write-path regression -> gate (PASSED). Honest caveats: INSERT is still ~1.4× behind SQLite on the fair-PK shape, the -> PageBased engine's UPDATE is 4.9× behind, and the pure default (encrypted) config is 1.3–1.6× -> behind the `NoEncryptMode` raw-speed posture. Full notes: +> gate (PASSED). Benchmarks re-measured and republished on 2026-10-11 (`92794084`), every ratio +> same-run: on the default-posture fair-PK shape SharpCoreDB is **INSERT 1,17× (1,03–1,27) and READ +> 1,13× (1,12–1,15) ahead** of SQLite and **UPDATE 0,53× (0,42–0,53) and DELETE 0,47× (0,46–0,62)** +> of it, and per-record encryption costs the otherwise-identical fixed-width arm **1,2–1,9×**. This +> corrects the earlier "ahead on UPDATE/DELETE" reading, which was taken while the benchmark left the +> *SQLite* arm allocating a command and two parameters per row — SQLite's own PK UPDATE/DELETE +> references rose **3,05×** (288.108 → 878.557 / 385.116 → 1.172.704) once the comparator held one +> prepared command, so that band is superseded, not better. Full notes: > [`docs/2.1.0-RC.3_WHAT_CHANGED.md`](docs/2.1.0-RC.3_WHAT_CHANGED.md) — the RC.3 validation record; > the promotion itself re-verified the core suite at **1995 / 0 failed** and is documented in > [`docs/net11/PROMOTION_TO_MAIN.md`](docs/net11/PROMOTION_TO_MAIN.md). @@ -51,7 +56,9 @@ Use it when you need: > UPDATE ~270–315K, DELETE ~353–420K, INSERT ~186–190K, READ ~95–107K). Honest default-config > caveat: the pure default config runs ~1.3–1.6x slower because the file-level wrapper still pays > AES work while per-record at-rest encryption is off (`NoEncryptMode` root cause, P3d). Full -> report: [`docs/2.0.0.2_WHAT_CHANGED.md`](docs/2.0.0.2_WHAT_CHANGED.md) and +> note: the SQLite references in this block are the pre-comparator-correction ones described above, +> so read them as that line's history rather than as today's reference. Full report: +> [`docs/2.0.0.2_WHAT_CHANGED.md`](docs/2.0.0.2_WHAT_CHANGED.md) and > [`docs/benchmarks/default-config-pk.md`](docs/benchmarks/default-config-pk.md). > Full documentation: **`docs/INDEX.md`** · Manual: **`docs/manual/README.md`** · Performance: **`docs/manual/performance.md`** @@ -84,22 +91,35 @@ Use it when you need: ## What's new in v2.x (current engine line) -The v1.x benchmark gap (point reads/updates/deletes **16–52x behind SQLite**) is closed. The -numbers below are the **fair-PK harness (median-of-3, tuned config, ascending-PK batches)** from -`docs/2.0.0.2_WHAT_CHANGED.md`: +The v1.x benchmark gap (point reads/updates/deletes **16–52x behind SQLite** because every update +appended a new row) is closed as an engine behaviour. The numbers below are re-measured on +**2026-10-11** (`92794084`, v2.1.0 / .NET 11) on the **fair-PK harness (median of 3 runs, `id INTEGER +PRIMARY KEY`, ascending-PK batches)**, and every ratio is taken against **that run's own SQLite arm** — +this project reports ratios, never lone absolutes, because SQLite's own reference drifts between runs. -| Operation (ops/s) | **SharpCoreDB v2.x (Columnar fixed-width)** | SQLite | gap | +**Default posture, `--pk-default` (pure `DatabaseConfig`: encryption on, no harness flags).** Ratio is +SharpCoreDB ÷ SQLite, so **> 1,00x means SharpCoreDB is ahead**; the range is the paired per-rep range, +and one that straddles 1,00x means the cell is inside this box's noise: + +| Operation (ops/s) | **SharpCoreDB v2.1.0 (default, encrypted)** | SQLite (same run) | ratio | |---|---:|---:|---:| -| **UPDATE** | **~163–245K** | ~270–315K | ~0.8–1.7x of SQLite | -| **DELETE** | **~106–172K** | ~353–420K | ~2.1–3.5x | -| **INSERT** | **~125–152K** | ~186–190K | competitive | -| **READ** | **~72–110K** | ~95–107K | competitive | +| **INSERT** | **204.263** | 174.469 | **1,17x** (1,03–1,27) ✅ | +| **READ** | **134.634** | 116.916 | **1,13x** (1,12–1,15) ✅ | +| **UPDATE** | **398.709** | 773.323 | **0,53x** (0,42–0,53) | +| **DELETE** | **509.941** | 1.085.140 | **0,47x** (0,46–0,62) | + +**Record-layout A/B on the tuned `--pk` harness** — same workload, three interchangeable layouts, +plaintext so the layout is the only variable: the **fixed-width** layout is **1,2–1,5× ahead of the +legacy variable-length layout** on UPDATE (534–629K vs 360–372K) and DELETE (687–716K vs 425–466K) +while level on INSERT/READ. Per-record encryption costs the same fixed-width arm **1,2–1,9×** +(INSERT 1,18–1,25× · READ 1,22–1,32× · UPDATE 1,83–1,91× · DELETE 1,41–1,57×). Honest notes: these are ascending-PK batches on the fixed-width Columnar fast path — other query -shapes fall back to correct-but-slower generic paths. The **pure default config** runs ~1.3–1.6x -slower than `NoEncryptMode=true` (documented `NoEncryptMode` root cause, see -`docs/benchmarks/default-config-pk.md`). SQLite comparison, honest guidance and the API ladder: -**[Performance Guide](docs/manual/performance.md)**. Roadmap: +shapes fall back to correct-but-slower generic paths. SQLite's PK UPDATE/DELETE references are really +~0,80M / ~1,00M ops/s on this box; the ~270–315K / ~353–420K references published earlier were +measured while the harness left the SQLite arm allocating a command and two parameters per row, so +they understate SQLite and are superseded. Full tables and the API ladder: +**[Performance Guide](docs/manual/performance.md)** · **[`docs/BENCHMARK_RESULTS.md`](docs/BENCHMARK_RESULTS.md)**. Roadmap: **[`docs/performance/V2_PERFORMANCE_PLAN.md`](docs/performance/V2_PERFORMANCE_PLAN.md)**. Headline changes: diff --git a/docs/BENCHMARK_RESULTS.md b/docs/BENCHMARK_RESULTS.md index 869263d9..417ceb79 100644 --- a/docs/BENCHMARK_RESULTS.md +++ b/docs/BENCHMARK_RESULTS.md @@ -4,13 +4,59 @@ - OS: Windows 11 - CPU: Intel i7-10850H @ 2.70GHz (6 cores/12 threads) - RAM: 16GB -- Runtime: .NET 10.0.4, RyuJIT x86-64-v3 -- Benchmark Tool: BenchmarkDotNet v0.15.8 -- **Last Updated: March 14, 2026** · **v2.0.0 update: August 28, 2026** +- Runtime: .NET 10.0.4 (the v1.x BenchmarkDotNet suite further down) · **.NET 11 (C# 15 preview), RyuJIT x86-64-v3** for the v2.1.0 comparative numbers +- Benchmark Tool: BenchmarkDotNet v0.15.8 (v1.x suite) · `tests/benchmarks/SharpCoreDB.Benchmarks.Comparative` (v2.1.0) +- **Last Updated: March 14, 2026** · **v2.0.0 update: August 28, 2026** · **v2.1.0 update: October 11, 2026** --- -## ⚡ v2.0.0 Update (August 28, 2026) +## ⚡ v2.1.0 Update (October 11, 2026, .NET 11) + +Re-measured on the promoted tree `92794084` (v2.1.0) with the same harness. **Every ratio below is +same-run** — SQLite is measured in the same process run, because its own reference drifts between +sessions and ratios, not lone absolutes, are the comparable part. Results: +`pk_default_20261011_125413.json` (fair-PK), `comparative_20261011_125436.json` (ladder). + +**Fair-PK shape, default posture (`--pk-default`: pure `DatabaseConfig`, encryption on, no harness +flags, `id INTEGER PRIMARY KEY`) — ratio is SharpCoreDB ÷ SQLite, so > 1,00x means SharpCoreDB is +ahead, and the range is the paired per-rep range:** + +| Operation | SharpCoreDB v2.1.0 (default, encrypted) | SQLite (same run) | ratio | +|---|---:|---:|---:| +| INSERT | **204.263** | 174.469 | **1,17x** (1,03–1,27) | +| READ | **134.634** | 116.916 | **1,13x** (1,12–1,15) | +| UPDATE | **398.709** | 773.323 | 0,53x (0,42–0,53) | +| DELETE | **509.941** | 1.085.140 | 0,47x (0,46–0,62) | + +**Document CRUD shape (no PK, the v1.x-era workload of this file), median of 3:** + +| Database | INSERT | READ | UPDATE | DELETE | +|---|---:|---:|---:|---:| +| SharpCoreDB (Direct) | 186.718 | **191.474** 🥇 | 221.092 | 326.445 | +| SharpCoreDB (StructRow) | **187.838** | 169.684 | — | — | +| SharpCoreDB (SQL) | 144.706 | 105.250 | 176.570 | 134.022 | +| SQLite | 175.138 | 109.429 | **717.926** 🥇 | **975.609** 🥇 | +| LiteDB | 70.410 | 15.774 | 11.038 | 14.880 | + +**What closed and what is left.** The v2.0-era warning in this file — "single-row UPDATE/DELETE vs +SQLite (append-on-update vs in-place C records) — top v2.1 priority" — is **closed as an engine +behaviour and honestly restated as a remaining ratio**. UPDATE/DELETE now run in place with deferred +index maintenance (fixed-width layout is the default for new PK tables), which makes the fixed-width +fair-PK arm **1,2–1,5× faster than the legacy variable-length arm** on UPDATE/DELETE and puts the +default posture at 0,53× / 0,47× of SQLite — but it did **not** close the ratio, because SQLite's own +PK references on this box are ~796–811K UPDATE and ~1,00–1,04M DELETE. Earlier publications quoted +SQLite at ~270–315K UPDATE / ~353–420K DELETE and therefore claimed the layout was *ahead*; those +references were measured while the harness left the SQLite arm allocating a command and two +parameters per row, and they are superseded (SQLite's PK references rose 3,05× once the comparator +held one prepared command — 288.108 → 878.557 UPDATE, 385.116 → 1.172.704 DELETE). + +> Full analysis + API ladder: [`docs/manual/performance.md`](manual/performance.md) · +> comparative report: [`docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md`](benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md) + +--- + + +## ⚡ v2.0.0 Update (historical, August 28, 2026) This report covers the v1.x-era BenchmarkDotNet micro-benchmark suite. The **v2.0 performance-first release** additionally closed the comparative CRUD gap against SQLite/LiteDB — measured on the @@ -21,8 +67,9 @@ release** additionally closed the comparative CRUD gap against SQLite/LiteDB — - **Batch INSERT is SQLite-competitive** — 91–133K vs 145–150K ops/s - **SIMD columnar analytics remain dominant** — `GROUP BY` SUM over 10M rows in ~2 ms -Remaining known gap: single-row UPDATE/DELETE vs SQLite (append-on-update vs in-place C records) — -top v2.1 priority. +Remaining known gap at that time: single-row UPDATE/DELETE vs SQLite (append-on-update vs in-place C +records) — **closed as an engine behaviour in v2.1.0** (UPDATE/DELETE now run in place with deferred +index maintenance); see the v2.1.0 section above for where the ratio stands today. > Full analysis + API ladder: [`docs/manual/performance.md`](manual/performance.md) · > comparative report: [`docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md`](benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index b88ecedf..7da00352 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -17,6 +17,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **The published benchmark pages now carry the 2026-10-11 v2.1.0 numbers, and the fair-PK "ahead on UPDATE/DELETE" claim they used to carry is corrected rather than dropped — because it was measured against a crippled SQLite arm.** `README.md`, `docs/BENCHMARK_RESULTS.md`, `docs/manual/performance.md`, `docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md` and `docs/benchmarks/default-config-pk.md` are restamped to v2.1.0 / .NET 11 / `92794084` and re-tabulated from one **same-run** measurement set (`pk_default_20261011_125413.json`, `comparative_20261011_125436.json`): default posture fair-PK **INSERT 1,17× (1,03–1,27), READ 1,13× (1,12–1,15), UPDATE 0,53× (0,42–0,53), DELETE 0,47× (0,46–0,62)** against that run's own SQLite, and the record-layout A/B shows the fixed-width layout **1,2–1,5× ahead of the legacy variable-length layout** on UPDATE/DELETE at a **1,2–1,9×** per-record-encryption cost. Two published figures were wrong and are now stated with their cause: the inline-capacity default is **24** bytes, not the 16 the README still said (worth **+34 %** on the batched multi-row INSERT shape and fair-PK INSERT ratio **0,63× → 0,87×**), and the "fair-PK layout ahead of SQLite on UPDATE (0,8×) / DELETE (0,5×)" reading came from the window in which the harness left the ***SQLite*** arm allocating a command and two parameters per row — SQLite's own PK references are **3,05×** higher with the prepared comparator (288.108 → 878.557 / 385.116 → 1.172.704), so the honest reading is *behind* on those two cells and the earlier band is superseded, not better. Three harness defects were found while publishing and fixed in the same session: the `--pk` banner's "legacy variable-length" arm now genuinely opts out of the fixed-width layout (`BuildConfig(legacyVariableLength: true)` → `AutoFixedWidthRecords = false`; before the fix the two layout arms read ~3 % apart instead of the 1,3–2,3× the layout is worth), the `[diag]` line prints the **resolved** config instead of the requested parameters (the encrypted stock arm had printed `noEncrypt=True`), and `RunPkMedian` now stamps `Reps`, so a `pk_comparative_*.json` archive describes its protocol as measured instead of claiming `"Reps": 1` for a 3-rep median. Pinned by `BenchmarkProtocolArchiveTests` (`PkLegacyArm_ResolvesToVariableLengthRecords_AsItsBannerClaims`, `PkComparisonArchive_RecordsTheMeasuredRepCount_NotTheSingleShotDefault`; both **red before, green after**), Manual-tagged and skipping when the harness exe is absent. No engine code, config default, threshold or version moved. + - **A Columnar table now registers an automatic hash index for a column the first time an operation filters on it, instead of registering one per column at `CREATE TABLE` — so a write stops maintaining indexes the workload never uses.** `DatabaseConfig.EnableHashIndexes` (default `true`) had always meant "auto-create a hash index per column": `SqlParser.DDL.cs` registered the primary key *and every other column*, and because `Table.EnsureAllRegisteredIndexesLoaded()` loads every **registered** index before a write on the append/Columnar engines, the five-column `docs` shape paid a hash remove+add per row on all five columns while the workload filtered on one — the measured 20.000 `index-maint` calls per 10.000 updates. Registration is now on demand: a user-declared primary key is still registered up front, every other column registers itself through the single gate `Table.EnsureAutoHashIndexRegistered`, which every lookup path now shares (the read path, the position-aware update locate, both delete cores, the parallel scan, both struct-scan fast paths and structured DML), and the hidden `_rowid` fallback is never registered because the PK B-tree already serves its point lookups. *Building* was already lazy (`Table.EnsureIndexLoaded`), so nothing else in the index lifecycle changed, `HashIndexAutoCreationGateTests` was rewritten to the new contract (primary key eagerly, others on the first filtered operation, none with the dial off) rather than loosened, and `SHARPCOREDB_HASH_INDEXES=0` still turns automatic indexes off entirely. **Measured on arms B and C (`--pk-default-batch` / `--docs-batch`, INSERT vs SQLite, median with the run's own per-rep range): arm B batch 0,95× → 1,60× (1,05–1,92) and arm B SQL 0,83× → 1,05× (0,94–1,26); arm C batch 1,26× (1,19–1,54) and arm C SQL 0,90× → 0,98× (0,98–1,29)** — with same-build `SHARPCOREDB_HASH_INDEXES=0` runs as the ceiling on the day (arm B 1,68× batch / 1,31× SQL, arm C 1,25× batch / 0,95× SQL), i.e. what is left of the margin is the eagerly registered declared-primary-key index. On the query side the cost is now visible instead of hidden: `--auto-index-benefit` reports the **first** filtered query separately, because that is the query that pays for building the index — 20.000 rows, 2.000 point queries per column, two runs: the **first** query costs 35,5–71,0 ms on `email`/`score`/`age` and the warm queries after it cost 4,7–14,1 µs (`email`, `score`) against 4,8–5,0 ms with the dial off (**354×–988×** and **3.106×–7.656×**), with the explicitly indexed control unchanged (63–68 µs first, 5,9–6,3 µs repeated) and identical row counts in both configurations. Core suite **1952 / 0 failed / 0 errors**; `--gate` **PASSED, 8/8 metrics `ok`**, all eight ahead of the recorded baseline (default INSERT 0,78×, default UPDATE 0,31×). - **The single-file growth minimum is now 1 MiB (was 10 MiB), which takes every small `.scdb` database from 14,7 MB to 6,4 MB.** The 10 MiB minimum was what made a 100-row `.scdb` file 14,7 MB: a fresh file is 1.037 pages and the extension is `max(requiredPages, currentSize / 2, minExtensionPages)`, where 10 MiB is 2.560 pages at a 4 KiB page size — so the minimum won the contest and the first extension landed on 3.597 pages = **14.733.312 B**, the same size at 1 row and at 2.000 rows (measured 2026-09-23, which is what proved it was a floor rather than growth). At the new 1 MiB default the minimum no longer binds (256 pages against the halving term's 518) and the same database lands on **6.369.280 B, −56,8 %**, measured at **1 / 100 / 200 / 500 / 2.000 / 20.000 / 200.000 / 1.000.000 rows** with **byte-identical allocation per row** and no measurable throughput difference — the single-file blocks are compressed, so even a million rows of the benchmark schema stay inside that first extension. Existing databases are unaffected: the value is read **per open**, never changes how a file is read, and the historical 14,7 MB floor is one configuration line away (`SingleFileMinExtensionBytes = 10 * 1024 * 1024`), pinned by `SingleFileFileGrowthTests.ExplicitHistoricalTenMiBMinimum_StillProducesTheOldFloor` so the old behaviour stays reproducible byte for byte. - **The inline capacity's default is now 24 bytes (was 16), and it is worth 0,63× → 0,87× on the fair-PK INSERT ratio.** `DatabaseConfig.FixedWidthInlineValueBytes` decides how many payload bytes a variable-length column may keep **inside** its fixed-width record slot instead of writing them to the overflow arena, and 16 was two bytes short of the most common short-value shape: on the benchmark schema the 18-byte `user99999@test.com` overflowed for every row whose index is ≥ 1.000, so **99.000 of 100.000 rows paid an arena write each**. At 24 every value in that schema fits inline and `arena-write`, `arena-append` and `encode-scratch` fall to **zero calls**, measured on the harness's own `[diag]` counter and on `--pk-profile-insert`. diff --git a/docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md b/docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md index a8d960b9..b4ae58b8 100644 --- a/docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md +++ b/docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md @@ -1,12 +1,55 @@ # SharpCoreDB — Comparative Benchmark Report -**Date:** 2026-03-06 (original) · **Updated:** 2026-08-28 (v2.0.0 final) -**SharpCoreDB Version:** 1.7.0 (original) → **2.0.0** (current) +**Date:** 2026-03-06 (original) · **Updated:** 2026-10-11 (v2.1.0, .NET 11) +**SharpCoreDB Version:** 1.7.0 (original) → **2.1.0** (current) **Test Machine:** Intel i7-10850H (6C/12T), 32 GB RAM, Windows 10, NVMe SSD --- -## ⚡ v2.0.0 Update (final, 2026-08-28) +## ⚡ v2.1.0 Update (2026-10-11, .NET 11 / C# 15) + +Re-measured with the same harness, same schema and same machine as the v2.0.0 run below, on the +promoted tree `92794084` (v2.1.0). Ladder: `comparative_20261011_125436.json` (median of 3 runs); +fair-PK: `pk_default_20261011_125413.json`. **Every ratio is same-run**, against that run's own SQLite +and LiteDB arms. + +**Document CRUD shape (no PK — the workload of this report), ops/sec, median of 3:** + +| Database | INSERT | READ | UPDATE | DELETE | +|---|---:|---:|---:|---:| +| **SharpCoreDB v2.1.0 (Direct)** | 186.718 | **191.474** 🥇 | 221.092 | 326.445 | +| **SharpCoreDB v2.1.0 (StructRow)** | **187.838** 🥇 | 169.684 | — | — | +| **SharpCoreDB v2.1.0 (SQL)** | 144.706 | 105.250 | 176.570 | 134.022 | +| **SQLite** | 175.138 | 109.429 | **717.926** 🥇 | **975.609** 🥇 | +| **LiteDB** | 70.410 | 15.774 | 11.038 | 14.880 | + +`—` = that phase is deliberately not run on the StructRow arm (plan §5.5); the Direct row carries +those cells through the same batch call path, table and indexes. **BLite 4.0.1 is still skipped**, for +the same reason as in 2026-03-06: its `BsonDocumentBuilder` has no public setter API +(`Set`/`Add`/`Write` all missing), so `DynamicCollection.CreateDocument()` compiles but cannot be +populated — the harness prints that reason rather than a zero. + +**Headline takeaways:** + +- ✅ **READ beats SQLite on the Direct and StructRow APIs** — 191.474 / 169.684 vs 109.429 ops/sec + (**1,75x / 1,55x**), and INSERT is ahead on both (186.718 / 187.838 vs 175.138, **1,07x**). +- ✅ **Every CRUD operation beats LiteDB** on the best API — INSERT 2,7x, READ 12,1x, UPDATE 20,0x, + DELETE 21,9x. +- ⚠️ **The generic document shape still trails SQLite on UPDATE/DELETE** (Direct 221.092/326.445 = + **0,31x / 0,33x**; SQL path 0,25x / 0,14x) — improved over the v2.0 reading of ~0,1–0,2x by the + in-place UPDATE/DELETE work, but not closed. On the **fair-PK shape**, where that work is the design + target, the default posture reads UPDATE 398.709 (0,53x) and DELETE 509.941 (0,47x) with INSERT + 1,17x and READ 1,13x ahead — the fair-PK table and the encryption posture are in + [`docs/manual/performance.md`](../manual/performance.md) §7.1 and + [`docs/benchmarks/default-config-pk.md`](default-config-pk.md). +- ⚠️ **The SQL path pays for the generic planner** — same engine, same table: Direct is 1,29x faster + on INSERT, 1,82x on READ, 1,25x on UPDATE and 2,44x on DELETE. If a hot path is SQL, that is the + cost of the parse/resolve layer, measured rather than inferred. + +--- + + +## ⚡ v2.0.0 Update (historical, 2026-08-28) The v2.0 performance-first release **closed the benchmark gap**. Same harness, same schema, same machine, same protocol — two-run range (`comparative_20260828_*.json`): @@ -26,8 +69,9 @@ same machine, same protocol — two-run range (`comparative_20260828_*.json`): - ✅ **Every CRUD operation beats LiteDB** — reads ~5–8x, updates ~5x, deletes ~2–10x. - ✅ **INSERT remains SQLite-competitive** (0.6–0.9x, within 15–20%). - ⚠️ **Single-row UPDATE/DELETE still behind SQLite** (~0.1–0.2x) — SQLite's fixed-length C - record format writes in place; SharpCoreDB row stores still append-on-update. This is the - top v2.1 priority. + record format writes in place; SharpCoreDB row stores still append-on-update. This was the top v2.1 + priority, and the v2.1.0 section at the top of this file records what shipped and where the ratio + stands today (fair-PK default posture UPDATE 0,53x / DELETE 0,47x). - ✅ **Columnar SIMD analytics remain in a class of their own** — `GROUP BY` SUM over 10M rows in ~2 ms vs ~1,300 ms for SQLite (**~682x**). diff --git a/docs/benchmarks/default-config-pk.md b/docs/benchmarks/default-config-pk.md index 1eb898fc..16b3bb5b 100644 --- a/docs/benchmarks/default-config-pk.md +++ b/docs/benchmarks/default-config-pk.md @@ -100,3 +100,48 @@ encryption when `EnableAtRestRecordEncryption` is on. decisions) still needs a code-level explanation before any change. 3. The same-window A/B tooling now exists; re-run with 3 reps to quantify the paired ratio, then investigate the `NoEncryptMode` mechanism in code and only then consider a change. + +--- + +## Update 2026-10-11 (v2.1.0 / .NET 11, `92794084`) — the product posture, re-measured + +The 2026-09-04 tables above are kept as the P3 investigation record, and this file's variant list +(`async`, `bufferedio`, `novalidate`, `noadaptive`, `hsinsert`, `plain`, `tuned`) still describes the +switch. The default arm was re-measured on the promoted tree with the current harness — +`pk_default_20261011_125413.json`, 3 paired reps with the arm order alternated, ratio = **SharpCoreDB +÷ SQLite**, so **> 1,00x means SharpCoreDB is ahead**: + +| Operation | SharpCoreDB (default cfg) | SQLite (reference) | ratio | +|---|---:|---:|---:| +| INSERT | 204.263 | 174.469 | **1,17x** (1,03–1,27) | +| READ | 134.634 | 116.916 | **1,13x** (1,12–1,15) | +| UPDATE | 398.709 | 773.323 | **0,53x** (0,42–0,53) | +| DELETE | 509.941 | 1.085.140 | **0,47x** (0,46–0,62) | + +**Two corrections, both measured in the same window.** (1) The file's warning that SQLite's own +reference drifts still holds, and it is the reason the *direction* of the UPDATE/DELETE cells above +differs from the 2026-09-04 reading: those references were taken while the comparator left the SQLite +arm allocating a command and two parameters per row. With one prepared command SQLite's PK references +are ~0,77–0,89M UPDATE and ~1,01–1,11M DELETE, i.e. **3,05x** higher than the ~270–315K / ~353–420K +quoted earlier (performance plan §9 item 7, original jump 288.108 → 878.557 / 385.116 → 1.172.704). +The honest reading of this arm is therefore 0,53x UPDATE and 0,47x DELETE — behind, and improved from +the 0,39x / 0,31–0,35x this file's plan row last recorded. (2) The `[diag]` line now prints the +**resolved** config instead of the requested parameters: the stock arm's line reads +`noEncrypt=False autoFixedWidth=True atRest=True inline=24`, where before 2026-10-11 it printed +`noEncrypt=True` — the harness was reporting the parameter it had passed (`true` = "no encryption"), +not the encrypted config it had actually built, which is the opposite of what that arm measures. + +**Knob isolation re-run on this box and binary** (`SHARPCOREDB_PK_DEFAULT_VARIANT`, single runs, same +window): `hsinsert` (HighSpeedInsertMode) UPDATE 408.962 — **~0 %**; `noadaptive` +(EnableAdaptiveWalBatching off) 358.718 — **~10 % slower**, i.e. adaptive WAL batching is worth ~10 % +on this arm; an A/B on the tuned arms (Async → FullSync) moved the fixed-width at-rest UPDATE +291.720 → 303.282 (**+4 %**), which refutes the older +54 % FullSync hypothesis for the tuned arm as +well. The declared differences that remain between the tuned `--pk` config and this stock arm are the +ones no single switch isolates: page cache + capacity (10.000), memory mapping, buffered IO, WAL +buffer 8 MiB, buffer pool 128 MiB, `CollectGCAfterBatches=false`, compiled-plan / B-tree-selection / +SIMD-pushdown toggles, `SqlValidationMode=Disabled` and `StrictParameterValidation=false`. They are +recorded as **declared configuration differences, not an attributed cause**: the published comparisons +are all same-run ratios inside one configuration, so no published claim depends on cross-arm +comparability, and single-knob deltas below ~1,3x remain unattributable outside a same-window +interleaved A/B. + diff --git a/docs/manual/performance.md b/docs/manual/performance.md index 678a7ff3..a778d846 100644 --- a/docs/manual/performance.md +++ b/docs/manual/performance.md @@ -1,40 +1,64 @@ # 7. Performance Guide -> **The headline:** SharpCoreDB v2.0.0.3 is the current *performance-first* release on the v2.x -> engine line. The v1.x benchmark gap (point reads/updates/deletes ~16–52x behind SQLite) is -> closed: on the fair-PK harness (median-of-3, tuned config) **UPDATE is ~0.8–1.7x of SQLite, -> INSERT is competitive, and DELETE ~2.1–3.5x** — see §7.1 for the current table. +> **The headline:** SharpCoreDB v2.1.0 is the current *performance-first* release, on the .NET 11 / +> C# 15 line. The v1.x gap (point reads/updates/deletes ~16–52x behind SQLite, because every update +> appended a new row) is closed as an engine behaviour, and re-measured on 2026-10-11 (`92794084`) +> the **default posture is INSERT 1,17x (1,03–1,27) and READ 1,13x (1,12–1,15) ahead of SQLite** while +> **UPDATE 0,53x (0,42–0,53) and DELETE 0,47x (0,46–0,62)** stay behind — see §7.1. > > This chapter explains **when SharpCoreDB is fastest**, how to write code that gets there, and — -> honestly — where SQLite is still ahead (and why). Run the current fair-PK numbers yourself with -> `tests/benchmarks/SharpCoreDB.Benchmarks.Comparative` (Release, `--pk`), or use the published -> `docs/2.0.0.3_WHAT_CHANGED.md` / `docs/benchmarks/default-config-pk.md` results. +> honestly — where SQLite is still ahead (and why). Run the current numbers yourself with +> `tests/benchmarks/SharpCoreDB.Benchmarks.Comparative` (Release; `--pk-default` for the product +> posture, `--pk` for the record-layout A/B), or read `docs/BENCHMARK_RESULTS.md`. --- -## 7.1 The current v2.0.0.3 numbers (fair-PK harness) +## 7.1 The current v2.1.0 numbers -Harness: `tests/benchmarks/SharpCoreDB.Benchmarks.Comparative` fair-PK mode (`--pk`, median of 3 -runs), fixed-width Columnar fast path, ascending-PK batches. Methodology and SQLite reference: -`docs/2.0.0.3_WHAT_CHANGED.md` and `docs/benchmarks/default-config-pk.md`. +Harness: `tests/benchmarks/SharpCoreDB.Benchmarks.Comparative` — `--pk-default` (the product posture: +pure `DatabaseConfig` with the engine pinned to `AppendOnly`, encryption on, no harness flags) and +`--pk` (the tuned record-layout A/B), median of 3 runs, fixed-width Columnar fast path, ascending-PK +batches, measured 2026-10-11 on `92794084`. Every ratio is **same-run** against that run's own SQLite +arm, because SQLite's own reference drifts between sessions — ratios are the comparable part, never +lone absolutes. -| Operation (ops/s) | **SharpCoreDB v2.x (Columnar fixed-width)** | SQLite | gap | +**Default posture (`--pk-default`, encryption on) — ratio is SharpCoreDB ÷ SQLite, so > 1,00x means +SharpCoreDB is ahead; the parenthesised range is the paired per-rep range, and one that straddles +1,00x means the cell sits inside this box's noise and must be re-run, not rounded:** + +| Operation (ops/s) | **SharpCoreDB v2.1.0 (default, encrypted)** | SQLite (same run) | ratio | |---|---:|---:|---:| -| **UPDATE** | **~163–245K** | ~270–315K | ~0.8–1.7x of SQLite | -| **DELETE** | **~106–172K** | ~353–420K | ~2.1–3.5x | -| **INSERT** | **~125–152K** | ~186–190K | competitive | -| **READ** | **~72–110K** | ~95–107K | competitive | +| **INSERT** | **204.263** | 174.469 | **1,17x** (1,03–1,27) | +| **READ** | **134.634** | 116.916 | **1,13x** (1,12–1,15) | +| **UPDATE** | **398.709** | 773.323 | 0,53x (0,42–0,53) | +| **DELETE** | **509.941** | 1.085.140 | 0,47x (0,46–0,62) | + +**Record-layout A/B (`--pk`, tuned config, plaintext, two runs) — the ratio column here is SQLite ÷ +arm, exactly as the harness prints it, so > 1,00x means SQLite is faster:** + +| Arm (ops/s, range of two runs) | INSERT | READ | UPDATE | DELETE | +|---|---:|---:|---:|---:| +| fixed-width, plaintext | 227–237K | 138–142K | **534–629K** | **687–716K** | +| fixed-width, at-rest | 191–192K | 107–113K | 292–329K | 455–488K | +| legacy variable-length, plaintext | 218–233K | 134–175K | 360–372K | 425–466K | +| SQLite (same runs) | 177–183K | 112–113K | 796–811K | 999K–1,04M | +| gap vs SQLite (fixed-width, plaintext) | 0,8x | 0,6–0,8x | 1,3–1,5x | 1,4–1,5x | **Read the table honestly:** -- ✅ **UPDATE is now competitive with SQLite on the keyed fast path** (~0.8–1.7x) — fixed-width - record layout (default for new PK tables) makes UPDATE an in-place overwrite, and strictly - ascending `pk = literal` batches run as one contiguous range (B8/B9). -- ⚠️ **DELETE still trails SQLite ~2.1–3.5x** — SQLite’s row store is extremely fast at pure - deletes; closing the rest of that gap is the tracked v2.1 target. -- ⚠️ The **pure default config** runs ~1.3–1.6x slower than the tuned harness because the - file-level wrapper still pays AES work while per-record at-rest encryption is off (documented - `NoEncryptMode` root cause, see `docs/benchmarks/default-config-pk.md`). +- ✅ **INSERT and READ are ahead of SQLite in the default posture** (1,17x and 1,13x, neither range + straddling 1,00x), and the fixed-width layout is **1,2–1,5x ahead of the legacy variable-length + layout** on UPDATE/DELETE — the layout change paying off, measured with the layout as the only + variable. +- ⚠️ **SQLite is still faster on point UPDATE and DELETE** — 1,3–1,5x / 1,4–1,5x on the tuned + fixed-width arm and ~1,9x / ~2,1x in the default posture. Earlier published claims that the + fair-PK layout was *ahead* on UPDATE/DELETE were measured while the harness left the SQLite arm + allocating a command and two parameters per row; SQLite's own PK references jumped **3,05x** once + the comparator held one prepared command (288.108 → 878.557 UPDATE, 385.116 → 1.172.704 DELETE), + so those readings are superseded, not "better". +- ⚠️ **Per-record encryption costs 1,2–1,9x** on the otherwise-identical fixed-width arm (INSERT + 1,18–1,25x, READ 1,22–1,32x, UPDATE 1,83–1,91x, DELETE 1,41–1,57x) — a real, measured at-rest tax + rather than a hidden one. - ✅ **Analytics remain the crown jewel** — SIMD columnar aggregates run far faster than SQLite on `GROUP BY` SUM workloads (see §7.5). diff --git a/docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md b/docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md index 0dd1814e..a3567a1e 100644 --- a/docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md +++ b/docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md @@ -49,7 +49,7 @@ run the numbers are based on. | 7 | Per-row write durability under the default (FullSync) | **Stays write-through per value — recorded, not changed.** The default is a 34× cliff behind buffered appends on per-row statements (961–1,037 µs/row against ~30 µs/row, measured 2026-09-16), but a buffered *default* would trade the crash-durability promise (constraint 2: buffered rows are lost on process crash as well as power loss). The gain is already available exactly where a bulk caller wants it — `BulkImport` sets `EnableBufferedAppends = true` explicitly (with `Async` WAL, group commit and the query cache off), as does the write-once logging sink — so this is a deliberate posture, not an oversight. Any change to it is an owner call with both durability columns published. | | 8 | Inline-capacity default (§4b) | **Raised to 24 bytes (2026-09-22, was 16).** It is the knee: 24 removes the per-row overflow-arena write on this plan's tracked shapes, 32 buys no further speed at +20 % of the file, and the *combined* data+arena footprint moves −0,4 % / +2,0 % on the two benchmark schemas. Effect on the tracked ratio: fair-PK INSERT **0,63× → 0,87×** and absolute **130–135K → 162–171K ops/s**, so the plan's 150K absolute INSERT target is met; the ≥ 1,0× *ratio* target is not claimed (SQLite's own same-run reference moved from ~155K when the target was set to 176–199K on the quiet machine). The cost is disk on schemas whose values are shorter than the capacity, and an existing database is never rewritten — it keeps the capacity its records were written with. *(Re-confirmed unchanged in the 2026-09-24 review; the rows below are the decisions taken then.)* | | 9 | Single-file (`.scdb`) minimum file extension | **Lowered to 1 MiB (2026-09-24, was 10 MiB)** — *decision 5B of the 2026-09-24 review.* The 10 MiB minimum was why any small `.scdb` database was 14,7 MB whatever it held (a fresh file is 1.037 pages; the minimum, 2.560 pages at 4 KiB, won the `max(required, currentSize/2, min)` contest), and lowering it takes the same database to **6.369.280 B** because the halving term then decides. Measured at 1 / 100 / 200 / 2.000 / 20.000 / 200.000 / **1.000.000 rows** (seven sizes, each measured at both settings): file **6.369.280 B vs 14.733.312 B (−56,8 %)** and **byte-identical allocation per row** with no measurable rows/s difference (the apparent differences follow the order inside each pair). It is read **per open**, is **not** part of the on-disk format, and the historical 10 MiB is one config line away (`DatabaseConfig.SingleFileMinExtensionBytes = 10 * 1024 * 1024`), pinned by a test. | -| 10 | INSERT ratio target (decision 4, "beat SQLite") | **Absolute floor kept; the ≥ 1,0× ratio is closed as *not claimed*.** §8e measures the absolute target **met** (162–171K ops/s, floor 150K) while the ratio sits at **0,87×** against a same-run SQLite that reads 175,6–198,5K where the bar was set on ~155K — so the honest record is "absolute met, ratio 0,87× against a faster reference". What replaces the ratio as a tracked obligation is **publication**: both columns on every number (decision 6), and the **encrypted default arm** (`--pk-default`) becomes the tracked default-posture cell under a **no-regression rule against its own recorded values** — **and those values are the *symmetric-protocol* ones: INSERT 0,83×, READ 1,10× (straddling), UPDATE 0,39×, DELETE 0,41×** (session 32; re-measured 2026-09-27 on `a62ef168` as 0,81–0,87× / 1,01–1,03× / 0,39× / 0,31–0,35×). The `0,70× / 0,57× / 0,48× / 0,59×` reading this row used to carry is **superseded, not "better"**: it was taken while the benchmark left the *SQLite* arm allocating a command and two parameters per row, and SQLite's own PK UPDATE/DELETE references jumped **3,05×** (288.108 → 878.557, 385.116 → 1.172.704) once the comparator got one prepared command (see §9 item 7). Tracking the superseded band makes a correct engine read as a 20–40 % UPDATE/DELETE regression. No new numeric floor is invented here; if one is wanted for the encrypted arm, that is the one sub-decision left open in this row. | +| 10 | INSERT ratio target (decision 4, "beat SQLite") | **Absolute floor kept; the ≥ 1,0× ratio is closed as *not claimed*.** §8e measures the absolute target **met** (162–171K ops/s, floor 150K) while the ratio sits at **0,87×** against a same-run SQLite that reads 175,6–198,5K where the bar was set on ~155K — so the honest record is "absolute met, ratio 0,87× against a faster reference". What replaces the ratio as a tracked obligation is **publication**: both columns on every number (decision 6), and the **encrypted default arm** (`--pk-default`) becomes the tracked default-posture cell under a **no-regression rule against its own recorded values** — **and those values are the *symmetric-protocol* ones: INSERT 0,83×, READ 1,10× (straddling), UPDATE 0,39×, DELETE 0,41×** (session 32; re-measured 2026-09-27 on `a62ef168` as 0,81–0,87× / 1,01–1,03× / 0,39× / 0,31–0,35×). The `0,70× / 0,57× / 0,48× / 0,59×` reading this row used to carry is **superseded, not "better"**: it was taken while the benchmark left the *SQLite* arm allocating a command and two parameters per row, and SQLite's own PK UPDATE/DELETE references jumped **3,05×** (288.108 → 878.557, 385.116 → 1.172.704) once the comparator got one prepared command (see §9 item 7). Tracking the superseded band makes a correct engine read as a 20–40 % UPDATE/DELETE regression. No new numeric floor is invented here; if one is wanted for the encrypted arm, that is the one sub-decision left open in this row. **Re-measured again 2026-10-11 on `92794084` (the v2.1.0 publication run, `pk_default_20261011_125413.json`) as 1,17× (1,03–1,27) / 1,13× (1,12–1,15) / 0,53× (0,42–0,53) / 0,47× (0,46–0,62) — every cell improved on the 2026-09-27 band, and the same session's publication carries the corrected post-comparator SQLite references in `README.md`, `docs/BENCHMARK_RESULTS.md`, `docs/manual/performance.md` and `docs/benchmarks/*`.** | | 11 | Overflow-arena appends through the append buffer | **Stays write-through.** `arena-append` was the fair-PK profile's next lever at **63,3 ms = one `FileOptions.WriteThrough` open per row**, which is exactly what the append buffer removed for single-row INSERTs — but routing the arena through that buffer changes its durability window, so it is the same owner call as §5 item 2's, and it is **not taken**: `BulkImport` already opts into buffering where a bulk caller wants it. | | 12 | Encrypted commit's `commit-overwrites` half | **Left as-is (durability boundary).** It is 16,4 % of the encrypted UPDATE pass, of which the non-prep half is 14,4 ms, and it is the point where a batch's overwrites reach the disk — changing it trades the crash story, not a benchmark number. | | 13 | `index-maint` on the batched update path | **Left as-is (index freshness).** 7,6 % of the pass at 43 B × 20.000 calls. The deferred-index experiment already showed what removing in-session maintenance costs elsewhere: the O(n) reconcile at `Table.Flush()` took random-key DELETE from **294.185 → 70.248 ops/s**, which is why the threshold was raised rather than lowered. | diff --git a/docs/performance/WORKLOG.md b/docs/performance/WORKLOG.md index 1469b8ef..66868538 100644 --- a/docs/performance/WORKLOG.md +++ b/docs/performance/WORKLOG.md @@ -5079,3 +5079,24 @@ Nothing in the repository explains a stall: no `.git/*.lock`, no fsmonitor daemo + +### 2026-10-11 (session 81) — v2.1.0 benchmark publication: the fair-PK "ahead on UPDATE/DELETE" claim was measured against a SQLite arm that allocated a command per row, corrected in six published pages, with three harness defects found and fixed on the way (the `--pk` "legacy variable-length" arm now really opts out of fixed width, `[diag]` prints the resolved config, `RunPkMedian` stamps `Reps`) + +- Session: 1 of 1 for the plan's publication obligation (§2 deliverable 1 and §9 row 10 — the tracked `--pk-default` default-posture cell and the pages it feeds). No new plan item was started; the session's own findings were fixed inside it rather than filed. +- Command(s): the authoritative fair-PK arm `SharpCoreDB.Benchmarks.Comparative.exe --pk` → `artifacts/s82-pk.txt` (run 1) and `s82-pk2.txt` (run 2, identical protocol) · the product posture `--pk-default` → `s82-pkdefault.txt` · the docs ladder `SHARPCOREDB_LADDER_REPS=3` → `s82-ladder.txt` · the knob isolations `SHARPCOREDB_PK_DEFAULT_VARIANT=hsinsert` → `s82-pkdefault-hsinsert.txt` and `=noadaptive` → `s82-pkdefault-noadaptive.txt` · the durability A/B `SHARPCOREDB_WAL_DURABILITY=fullsync` → `s82-pk-fullsync.txt` · `pwsh scripts/quiet-machine.ps1` → `s82-quiet.txt` (NOISY → the 12 idle MSBuild servers cleared, leaving only the standing `WSearch` privilege-limit finding) · the guard proof `tests\SharpCoreDB.Tests\bin\Release\net11.0\SharpCoreDB.Tests.exe -filterVSTest 'FullyQualifiedName~BenchmarkProtocolArchiveTests'` → `artifacts/s81-guard-publish.txt` (the pre-fix red is kept as `s82-guard-red`), and the wider suite with CI's filter → `artifacts/s81-core-suite-publish.txt`. Archives cited by the published pages: `results/pk_default_20261011_125413.json`, `results/comparative_20261011_125436.json`, `results/pk_comparative_20261011_123404.json`, `results/pk_comparative_20261011_125346.json`. +- Regime: Release, JIT, `win-x64`, `net11.0` / C# 15 preview on `92794084` (= `origin/master`, v2.1.0); `SHARPCOREDB_BENCH_TEMP=D:\scdb-bench-tmp` (scratch dir left clean: 25 files / 0,14 GB before the runs); every published ratio is the **same-run** one — SQLite and LiteDB measured in the same process run as each SharpCoreDB arm, with its own per-rep range carried along — because SQLite's reference drifts between sessions and single-run absolutes are not comparable across them. +- Verdict: **KEPT, as publication with the direction corrected.** Published set: default posture fair-PK **INSERT 1,17× (1,03–1,27) / READ 1,13× (1,12–1,15) / UPDATE 0,53× (0,42–0,53) / DELETE 0,47× (0,46–0,62)** against that run's own SQLite, and the record-layout A/B shows fixed-width **1,2–1,5× ahead of legacy variable-length** on UPDATE/DELETE at a **1,2–1,9×** at-rest cost on the otherwise identical arm. The previously published "fair-PK layout ahead of SQLite on UPDATE (0,8×) / DELETE (0,5×)" reading is **superseded** — it came from the window in which the harness left the *SQLite* arm allocating a command and two parameters per row (plan §9 item 7: SQLite's PK references rose 3,05×, 288.108 → 878.557 / 385.116 → 1.172.704, once the comparator held one prepared command). Three harness defects **FIXED** in-session, each red→green (§2). The tuned-vs-stock UPDATE divergence is **documented, not attributed** (§3) — no published figure depends on it. +- Commit: local commit on a feature branch off `master` (`perf/autonomous-20261011`) — **never on `master`**, never pushed (brief §9). +- NEXT: resume the plan's item list from session 80's `NEXT:` line (§5.1 INSERT, its item (i): the identity-remap fast path for `InsertBatch(table, rows, columns)` whose column order already is the table order — 112 B/row on the array arm). This session added no new perf item, so the queue's head is unchanged. + +**1. What was published, and the correction it carries.** Six pages, all restamped to v2.1.0 / .NET 11 / `92794084` and all re-tabulated from one same-run set: `README.md` (release block + the v2.x headline table), `docs/BENCHMARK_RESULTS.md` (a new v2.1.0 section above the v2.0.0 one, which is now marked historical), `docs/manual/performance.md` §7.1, `docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md` (a new v2.1.0 section + header), `docs/benchmarks/default-config-pk.md` (a dated 2026-10-11 update appended to the P3 record) and the plan's row 10. Two direction conventions coexist in the harness — `--pk` prints **SQLite ÷ arm**, `--pk-default` prints **arm ÷ SQLite** — so each table in the pages now states its convention on the line above it, and the `--pk` table names the `--pk-default` posture separately rather than mixing the two. Three published figures that were simply wrong are corrected: the inline-capacity default is **24**, not the 16 the README still claimed (`DatabaseConfig.FixedWidthInlineValueBytes = 24`; the plan's decision 8 and the CHANGELOG both recorded the raise, the README had not), worth **+34 %** on the batched multi-row INSERT shape and **0,63× → 0,87×** on the fair-PK INSERT ratio; the fair-PK UPDATE/DELETE direction, above; and the "top v2.1 priority" line in `docs/BENCHMARK_RESULTS.md` and the comparative report, which is now the historical sentence it was plus where the ratio actually stands. + + +**2. Three harness defects, found here and fixed here (bug-fix duty).** (i) `BuildConfig`'s layout switch was `AutoFixedWidthRecords = !fixedWidth`, which cannot express "legacy variable-length": the product's `AutoFixedWidthRecords` default is **true**, so an arm that merely declined to *force* the layout still got it for any new table declaring a PRIMARY KEY — the two `--pk` layout arms measured **223.728 / 562.562** against **230.591 / 582.384** (INSERT/UPDATE, ~3 % apart) while the banner promised a variable-length arm the plan had last measured 2,3× away on UPDATE. Fixed with an explicit `legacyVariableLength` parameter (`AutoFixedWidthRecords = legacyVariableLength ? false : !fixedWidth`), scoped to that one call site, defaulting to `false` so no other mode's config moves — after the fix the arms separate as the layout deserves (legacy UPDATE 360–372K against fixed-width 534–629K). (ii) The `[diag]` line printed the **parameters**, not the built config: the stock `--pk-default` arm's line read `noEncrypt=True` — "no encryption" — on the arm whose whole point is per-record encryption at rest. It now prints `config.NoEncryptMode` / `config.FixedWidthRecordLayout` / `config.AutoFixedWidthRecords`, and the stock arm reads `noEncrypt=False autoFixedWidth=True atRest=True inline=24`, which is what a reader needs to interpret the number. (iii) `RunPkMedian` builds its own median object (rather than going through `MedianArm`) and never stamped `Reps`, so every published `pk_comparative_*.json` archive claimed `"Reps": 1` for a median of 3 taken after 8 discarded warm-ups — the field exists "so a reader can tell a median from a single-shot reading without the console log", so the archive said the opposite of what was measured. All three are pinned in `BenchmarkProtocolArchiveTests` (Manual-tagged, skips when the harness exe is absent): `PkLegacyArm_ResolvesToVariableLengthRecords_AsItsBannerClaims` (red pre-fix, reproducing the ~3 % collapse the fix removes) and `PkComparisonArchive_RecordsTheMeasuredRepCount_NotTheSingleShotDefault` (red: all arms `Reps: 1` on a 2-rep run). Green after: **2 / 0 failed** (`artifacts/s81-guard-publish.txt`). + +**3. The one thing the publication had to resolve: the tuned at-rest arm and the stock arm disagree on UPDATE, and the difference is declared configuration.** `--pk`'s fixed-width **at-rest** arm reads UPDATE **292–329K** while the stock `--pk-default` arm — same engine, same table, same workload, also encrypted — reads **398.709**. Two runs of each, so it is not a stray rep. Isolated by single-knob variants on the same binary and window: `noadaptive` (`EnableAdaptiveWalBatching=false`) **358.718**, i.e. adaptive WAL batching is worth ~**10 %** on this arm; `hsinsert` (`HighSpeedInsertMode=true`) **408.962**, ~**0 %**; an A/B on the tuned arm itself (Async → FullSync) moved the *at-rest* UPDATE **291.720 → 303.282**, ~**+4 %**, which refutes the old +54 % FullSync hypothesis for this arm too. That accounts for roughly a sixth of the gap. The rest is what `BuildConfig` declares and `BuildPkDefaultVariantConfig` does not: page cache (`EnablePageCache=true`, capacity 10.000), `UseMemoryMapping`, `UseBufferedIO`, `WalBufferSize` 8 MiB, `BufferPoolSize` 128 MiB, `CollectGCAfterBatches=false`, the compiled-plan / B-tree-selection / SIMD-pushdown toggles and `SqlValidationMode=Disabled` + `StrictParameterValidation=false` — read from the code (`Program.cs` L1431–1466), not inferred, but **not attributed**: no single switch isolates them, and the plan's standing rule is that sub-1,3× single-knob deltas outside an interleaved same-window A/B are not attributable. Recorded as such in `docs/benchmarks/default-config-pk.md`, with the reason it is publishable anyway: every published figure is a **same-run ratio inside one configuration**, so no claim in the pages depends on the tuned and stock arms being comparable to each other. Both postures are published side by side and labelled; inventing an attribution would be worse than naming the difference. + +**4. Validation.** Core suite through the MTP exe with CI's filter: **1998 / 0 failed / 0 errors / 0 skipped in 83,9 s** (`artifacts/s81-core-suite-publish.txt`) — the wider suite the duty rule requires, run after the docs and harness changes. The two new guards: **2 / 0 failed** (§2). Build: `dotnet build tests/SharpCoreDB.Tests/SharpCoreDB.Tests.csproj -c Release -f net11.0` → **0 errors**. No `--gate` run was needed and none was attempted: nothing on the product's write path changed (the session's code changes are the harness's config parameterisation, one `[diag]` string and one test file), which is the gate's subject. Quiet-check: cleared the 12 idle MSBuild servers this session's own builds had left, leaving only the standing `WSearch` finding — the documented **privilege** limit (needs elevation), not a blocker; and the published figures are same-run ratios with per-rep ranges, which is exactly the form that survives it. + +**5. What was not touched.** No engine code, no config default, no threshold, no package or version metadata; the harness edits are confined to `BuildConfig`'s parameterisation, one `[diag]` string and the `Reps` stamp, and no other mode's `DatabaseConfig` moves (`legacyVariableLength` defaults to `false`, so every existing caller keeps the config it had). Nothing was weakened to publish a number: the encrypted posture is the *default* column in the README table, no arm's `NoEncryptMode` or durability was changed to win a cell, and the UPDATE/DELETE direction was reversed in the pages because the measurement says so, not because a target was met. No baseline was re-recorded (that needs the owner, §9 row 7). The comparison archives under `tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/` are untracked by convention and stay that way, except the four the published pages cite, which are committed with them so the numbers can be reproduced from the archive the plan asks for. + diff --git a/tests/SharpCoreDB.Tests/Benchmarks/BenchmarkProtocolArchiveTests.cs b/tests/SharpCoreDB.Tests/Benchmarks/BenchmarkProtocolArchiveTests.cs new file mode 100644 index 00000000..5e57a77a --- /dev/null +++ b/tests/SharpCoreDB.Tests/Benchmarks/BenchmarkProtocolArchiveTests.cs @@ -0,0 +1,271 @@ +namespace SharpCoreDB.Tests.Benchmarks; + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Text.Json; +using Xunit; + +/// +/// Guards the protocol an archived benchmark run claims in its JSON, not the numbers themselves. +/// +/// Why this exists (harness defect found 2026-10-11 while publishing the current fair-PK numbers): +/// --pk measures a median of SHARPCOREDB_BENCH_REPS runs (default 3) after +/// SHARPCOREDB_WARMUP_REPS discarded reps (default 8), and archives each arm as +/// BenchmarkResult. RunPkMedian built its median object without stamping Reps, so +/// every pk_comparative_*.json — the archive the performance plan publishes beside the fair-PK +/// table (§2 deliverable 1) so a reader can reproduce the protocol — reported "Reps": 1: it +/// described a 3-rep median as a single-shot reading. BenchmarkResult.Reps exists for exactly the +/// opposite purpose ("so a reader can tell a median from a single-shot reading without the console log"). +/// +/// +/// Process-level on purpose: the harness is a standalone Exe +/// (tests/benchmarks/SharpCoreDB.Benchmarks.Comparative) that no test project references and that +/// SharpCoreDB.CI.slnf does not build, so this test is tagged Manual and skips when the +/// executable is absent rather than failing every CI run that never builds it. Run it locally after +/// dotnet build tests/benchmarks/SharpCoreDB.Benchmarks.Comparative -c Release -f net11.0. +/// +/// +[Trait("Category", "Manual")] +public sealed class BenchmarkProtocolArchiveTests +{ + /// + /// The archived rep count must equal the rep count the run was asked for. Red on the pre-fix binary + /// (all arms read Reps: 1 for a 2-rep run), green once RunPkMedian stamps it. + /// + [Fact] + public void PkComparisonArchive_RecordsTheMeasuredRepCount_NotTheSingleShotDefault() + { + // Arrange + const int reps = 2; + var root = FindRepositoryRoot(); + var executable = root is null ? null : FindHarnessExecutable(root); + if (root is null || executable is null) + { + Assert.Skip("SharpCoreDB.Benchmarks.Comparative is not built; build it in Release/net11.0 to run this guard."); + } + + var resultsDir = HarnessResultsDirectory(executable!); + Directory.CreateDirectory(resultsDir); + var startedAt = DateTime.UtcNow; + var tempDir = Path.Combine(Path.GetTempPath(), $"SCDB_BenchProto_{Guid.NewGuid():N}"); + Directory.CreateDirectory(tempDir); + + var output = new List(); + try + { + using var process = StartHarness(executable!, tempDir, reps, output); + var exited = process.WaitForExit(TimeSpan.FromMinutes(5)); + Assert.True(exited, $"The benchmark harness did not finish within 5 minutes:{Environment.NewLine}{Joined(output)}"); + + var archive = NewestArchive(resultsDir, startedAt); + Assert.True( + archive is not null, + $"No pk_comparative_*.json was written after {startedAt:O} into {resultsDir}. Harness output:{Environment.NewLine}{Joined(output)}"); + + // Assert — every archived arm carries the rep count it was measured over. + using var document = JsonDocument.Parse(File.ReadAllText(archive!.FullName)); + var arms = document.RootElement.EnumerateObject().ToList(); + Assert.NotEmpty(arms); + + foreach (var arm in arms) + { + Assert.True( + arm.Value.TryGetProperty("Reps", out var archivedReps), + $"Arm '{arm.Name}' in {archive.Name} has no Reps field, so a reader cannot tell a median from a single shot."); + Assert.True( + archivedReps.GetInt32() == reps, + $"Arm '{arm.Name}' in {archive.Name} claims Reps={archivedReps.GetInt32()}, but the run measured {reps} reps."); + } + } + finally + { + TryDeleteDirectory(tempDir); + } + } + + /// + /// The arm the harness reports as legacy variable-length must actually run variable-length + /// records. + /// + /// Red on the pre-fix binary: BuildConfig set AutoFixedWidthRecords = !fixedWidth, so the + /// non-forced arm kept the product's auto resolver — and that resolver turns any NEW table declaring a + /// PRIMARY KEY into fixed-width records (SqlParser.DDL.cs). The arm the fair-PK table calls + /// "legacy variable-length" therefore measured the *same* layout as the fixed-width arm (2026-10-11 run: + /// 223,728 vs 230,591 INSERT ops/s, ~3 % apart, against a 2,3x UPDATE separation in the plan's last + /// genuine variable-length reading), which makes both the label and the layout comparison false. + /// Green once the arm opts out of AutoFixedWidthRecords — how the 2026-09-03 fair-PK report + /// ("legacy variable-length (opt-out)") defines it. + /// + /// + [Fact] + public void PkLegacyArm_ResolvesToVariableLengthRecords_AsItsBannerClaims() + { + // Arrange + var root = FindRepositoryRoot(); + var executable = root is null ? null : FindHarnessExecutable(root); + if (root is null || executable is null) + { + Assert.Skip("SharpCoreDB.Benchmarks.Comparative is not built; build it in Release/net11.0 to run this guard."); + } + + var tempDir = Path.Combine(Path.GetTempPath(), $"SCDB_BenchLayout_{Guid.NewGuid():N}"); + Directory.CreateDirectory(tempDir); + var output = new List(); + try + { + using var process = StartHarness(executable!, tempDir, reps: 1, output); + var exited = process.WaitForExit(TimeSpan.FromMinutes(5)); + Assert.True(exited, $"The benchmark harness did not finish within 5 minutes:{Environment.NewLine}{Joined(output)}"); + + // Act — the harness prints the layout each arm actually resolved in its own [diag] line. + var legacy = ResolvedLayoutFor(output, "legacy variable-length"); + var fixedWidth = ResolvedLayoutFor(output, "fixed-width, plaintext"); + + // Assert + Assert.True(legacy.HasValue, $"No layout diagnostic for the legacy arm in the harness output:{Environment.NewLine}{Joined(output)}"); + Assert.True(fixedWidth.HasValue, $"No layout diagnostic for the fixed-width arm in the harness output:{Environment.NewLine}{Joined(output)}"); + Assert.False(legacy!.Value, "The arm labelled 'legacy variable-length' resolved to FIXED-WIDTH records, so that label is false and the fair-PK table compares the same layout twice."); + Assert.True(fixedWidth!.Value, "The arm labelled 'fixed-width' did not resolve to fixed-width records."); + } + finally + { + TryDeleteDirectory(tempDir); + } + } + + private static Process StartHarness(string executable, string tempDir, int reps, List output) + { + var info = new ProcessStartInfo(executable, "--pk") + { + WorkingDirectory = Path.GetDirectoryName(executable)!, + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + }; + info.Environment["SHARPCOREDB_BENCH_REPS"] = reps.ToString(CultureInfo.InvariantCulture); + info.Environment["SHARPCOREDB_WARMUP_REPS"] = "0"; // The protocol under test is the rep stamp, not the ramp. + info.Environment["SHARPCOREDB_BENCH_TEMP"] = tempDir; + + var process = new Process { StartInfo = info }; + process.OutputDataReceived += (_, e) => { if (e.Data is not null) { lock (output) { output.Add(e.Data); } } }; + process.ErrorDataReceived += (_, e) => { if (e.Data is not null) { lock (output) { output.Add(e.Data); } } }; + process.Start(); + process.BeginOutputReadLine(); + process.BeginErrorReadLine(); + return process; + } + + /// + /// The IsFixedWidthRecords= value the harness printed for the first arm run after the section + /// banner containing , or null when that diagnostic is absent. + /// + private static bool? ResolvedLayoutFor(List output, string bannerMarker) + { + List snapshot; + lock (output) + { + snapshot = new List(output); + } + + var banner = snapshot.FindIndex(line => line.Contains(bannerMarker, StringComparison.Ordinal)); + if (banner < 0) + { + return null; + } + + const string token = "IsFixedWidthRecords="; + for (var i = banner + 1; i < snapshot.Count; i++) + { + var at = snapshot[i].IndexOf(token, StringComparison.Ordinal); + if (at < 0) + { + continue; + } + + var value = snapshot[i][(at + token.Length)..].Trim(); + var end = value.IndexOf(' '); + if (end >= 0) + { + value = value[..end]; + } + + return bool.TryParse(value, out var parsed) ? parsed : null; + } + + return null; + } + + /// The newest pk_comparative_*.json written by the run this test just started. + private static FileInfo? NewestArchive(string resultsDir, DateTime startedAt) => + new DirectoryInfo(resultsDir) + .GetFiles("pk_comparative_*.json") + .Where(f => f.LastWriteTimeUtc >= startedAt.AddSeconds(-1)) + .OrderByDescending(f => f.LastWriteTimeUtc) + .FirstOrDefault(); + + private static string? FindRepositoryRoot() + { + var current = new DirectoryInfo(AppContext.BaseDirectory); + while (current is not null) + { + if (File.Exists(Path.Combine(current.FullName, "SharpCoreDB.sln"))) + { + return current.FullName; + } + + current = current.Parent; + } + + return null; + } + + /// + /// The harness executable, or null when the project has not been built (the CI case). + /// + /// Pinned to net11.0 and to the shallowest match on purpose. A plain recursive search for the + /// exe picks up leftovers from other target frameworks and RID/publish subdirectories: this box still + /// carries a bin\Release\net10.0\<rid>\ build of an older harness revision (its console + /// table has three arms, not four), and *that* one writes its archive to + /// bin\Release\net10.0\results\, because the harness resolves its results directory as "three + /// levels up from AppContext.BaseDirectory". The shallowest net11.0 exe resolves to the + /// project's own results\ folder, which is the tracked evidence location. + /// + /// + private static string? FindHarnessExecutable(string root) + { + var tfmDir = Path.Combine(root, "tests", "benchmarks", "SharpCoreDB.Benchmarks.Comparative", "bin", "Release", "net11.0"); + return Directory.Exists(tfmDir) + ? Directory.EnumerateFiles(tfmDir, "SharpCoreDB.Benchmarks.Comparative.exe", SearchOption.AllDirectories) + .OrderBy(path => path.Length) + .FirstOrDefault() + : null; + } + + /// Mirrors the harness's own ResultsDirectory(), so the guard reads where it wrote. + private static string HarnessResultsDirectory(string executable) => + Path.GetFullPath(Path.Combine(Path.GetDirectoryName(executable)!, "..", "..", "..", "results")); + + private static string Joined(List output) + { + lock (output) + { + return string.Join(Environment.NewLine, output); + } + } + + private static void TryDeleteDirectory(string path) + { + try + { + Directory.Delete(path, recursive: true); + } + catch (IOException) + { + // A leaked temp directory is not a test failure. + } + } +} diff --git a/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/Program.cs b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/Program.cs index 34ae0ae5..b2d03c2d 100644 --- a/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/Program.cs +++ b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/Program.cs @@ -1386,7 +1386,8 @@ static DatabaseConfig BuildConfig( SharpCoreDB.Interfaces.StorageEngineType engineType, bool fixedWidth = false, bool noEncrypt = true, - bool? atRestRecords = null) + bool? atRestRecords = null, + bool legacyVariableLength = false) { return new DatabaseConfig { @@ -1398,10 +1399,15 @@ static DatabaseConfig BuildConfig( // see docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md §3-1c/§3-1d. EnableAtRestRecordEncryption = atRestRecords ?? ProductAtRestDefault, StorageEngineType = engineType, - // The fair PK comparison intentionally isolates the record-layout variable: the legacy - // arm opts out of the AutoFixedWidthRecords default so it measures true variable-length - // records; the fixed-width arm forces FixedWidthRecordLayout. - AutoFixedWidthRecords = !fixedWidth, + // The fair PK comparison isolates the record-layout variable, and this is the only place it can + // be isolated: AutoFixedWidthRecords (product default TRUE since #359) turns every NEW table + // that declares a PRIMARY KEY into fixed-width records, so an arm that merely declines to force + // the layout still measures the fixed-width layout whatever its label says. `legacyVariableLength` + // is the opt-out the fair-PK legacy arm needs — without it the legacy arm read 223,728 INSERT / + // 562,562 UPDATE against the fixed-width arm's 230,591 / 582,384 (2026-10-11, ~3 % apart) while + // its label promises the layout the plan last measured 2,3x apart on UPDATE. `fixedWidth` forces + // the layout instead; every other caller keeps the product's auto resolver untouched. + AutoFixedWidthRecords = legacyVariableLength ? false : !fixedWidth, FixedWidthRecordLayout = fixedWidth, // §4b inline capacity — SHARPCOREDB_INLINE_BYTES. Unset keeps the PRODUCT default (24 since the // 2026-09-22 owner decision; it was 16 from §4b's landing on 2026-09-16), so the arms track it; @@ -3589,6 +3595,7 @@ static BenchmarkResult RunSharpCoreDBPk( string? defaultVariant = null, bool noEncrypt = true, bool? atRestRecords = null, + bool legacyVariableLength = false, bool profileUpdateArm = false, bool profileDeleteArm = false, bool profileInsertArm = false, @@ -3619,7 +3626,7 @@ static BenchmarkResult RunSharpCoreDBPk( } else { - config = BuildConfig(engineType, fixedWidth, noEncrypt, atRestRecords); + config = BuildConfig(engineType, fixedWidth, noEncrypt, atRestRecords, legacyVariableLength); } using var db = (SharpCoreDB.Database)factory.Create( @@ -3717,7 +3724,12 @@ data TEXT var pkFiles = TableFileSizes(db); Console.WriteLine($" [diag] pk INSERT shape: {(useArrayInsert ? "dictionary-free object[] columns=" + string.Join(",", PkColumns) : "List>")}"); - Console.WriteLine($" [diag] pk INSERT fixedWidth={fixedWidth} noEncrypt={noEncrypt} " + // Resolved values, never the parameters: this line describes the arm the reader is looking at, + // and in the useDefaultConfig branch the parameters carry their defaults while the CONFIG is the + // product's default posture — the encrypted stock arm printed `noEncrypt=True` ("no encryption") + // until 2026-10-11, which is the opposite of what that arm measures. + Console.WriteLine($" [diag] pk INSERT fixedWidth={config.FixedWidthRecordLayout} noEncrypt={config.NoEncryptMode} " + + $"autoFixedWidth={config.AutoFixedWidthRecords} " + $"atRest={config.EnableAtRestRecordEncryption} IsFixedWidthRecords={pkFixedWidth} inline={pkInline}"); Console.WriteLine($" [diag] pk INSERT engine allocated {pkEngineAlloc:N0} B " + $"({pkEngineAlloc / InsertCount:N0} B/row) · gen0 {pkEngineGen0} " @@ -3970,6 +3982,16 @@ static double MedianOf(IEnumerable xs, Func r.ReadTime), UpdateTime = MedianOf(runs, static r => r.UpdateTime), DeleteTime = MedianOf(runs, static r => r.DeleteTime), + // Harness defect (found 2026-10-11 while publishing the current numbers; no plan section owns + // it — `--pk` is the measurement path): this helper builds its own median object instead of + // going through MedianArm/RunLadderMedianOf, and it did not stamp the rep count, so every + // pk_comparative_*.json — the archive the plan publishes BESIDE the fair-PK table (§2 + // deliverable 1) so a reader can reproduce the protocol — claimed `"Reps": 1` while the run + // measured 3 reps after 8 discarded warm-ups. BenchmarkResult.Reps exists precisely to tell a + // median from a single-shot reading "without the console log", so the archive was telling the + // reader the opposite of what was measured. Fixed here and in the shared MedianOf overload; + // the guard is BenchmarkProtocolArchiveTests (Category=Manual), red on the pre-fix binary. + Reps = runs.Count, // S1: a median is a fresh object, so the comparator's regime has to be carried across // explicitly or the archived median loses the very thing S1 exists to record. SqlitePragmas = runs.Count > 0 ? runs[0].SqlitePragmas : null, @@ -4062,7 +4084,9 @@ static void RunPkComparison(SharpCoreDB.Interfaces.StorageEngineType engineType) Console.WriteLine($"(fair-PK arms: {ResolveWarmupReps()} discarded warm-up rep(s) each, then the measured reps — median time per phase is reported)"); Console.WriteLine("━━━ SharpCoreDB (SQL, PK, legacy variable-length, plaintext) ━━━"); - var scdb = RunPkMedian(() => RunSharpCoreDBPk(engineType)); + // The legacy arm opts out of the auto resolver so it measures what its banner promises + // (BuildConfig's `legacyVariableLength`); every other arm keeps the product default. + var scdb = RunPkMedian(() => RunSharpCoreDBPk(engineType, legacyVariableLength: true)); Console.WriteLine(); Console.WriteLine("━━━ SharpCoreDB (SQL, PK, fixed-width, plaintext) ━━━"); @@ -4085,15 +4109,22 @@ static void RunPkComparison(SharpCoreDB.Interfaces.StorageEngineType engineType) Console.WriteLine($"║ SharpCoreDB FW plaintext│ {scdbFw.InsertOpsPerSec,10:N0} │ {scdbFw.ReadOpsPerSec,8:N0} │ {scdbFw.UpdateOpsPerSec,8:N0} │ {scdbFw.DeleteOpsPerSec,8:N0} ║"); Console.WriteLine($"║ SharpCoreDB FW at-rest │ {scdbFwAtRest.InsertOpsPerSec,10:N0} │ {scdbFwAtRest.ReadOpsPerSec,8:N0} │ {scdbFwAtRest.UpdateOpsPerSec,8:N0} │ {scdbFwAtRest.DeleteOpsPerSec,8:N0} ║"); Console.WriteLine($"║ SQLite │ {sqlite.InsertOpsPerSec,10:N0} │ {sqlite.ReadOpsPerSec,8:N0} │ {sqlite.UpdateOpsPerSec,8:N0} │ {sqlite.DeleteOpsPerSec,8:N0} ║"); - Console.WriteLine($"\n UPDATE gap vs SQLite: legacy {sqlite.UpdateOpsPerSec / (double)scdb.UpdateOpsPerSec:F1}x " + // State the direction: without it "gap 0,5x" reads as "we are ahead" to a reader who assumes the + // ratio is ours-over-theirs, and the fair-PK table is the one artifact readers copy numbers from. + Console.WriteLine("\n Gap = SQLite's ops/s ÷ that arm's ops/s, so a value ABOVE 1,0x means SQLite is FASTER"); + Console.WriteLine(" and a value below 1,0x means the SharpCoreDB arm is faster."); + Console.WriteLine($" INSERT gap vs SQLite: legacy {sqlite.InsertOpsPerSec / (double)scdb.InsertOpsPerSec:F1}x " + + $"fixed-width {sqlite.InsertOpsPerSec / (double)scdbFw.InsertOpsPerSec:F1}x " + + $"fixed-width at-rest {sqlite.InsertOpsPerSec / (double)scdbFwAtRest.InsertOpsPerSec:F1}x"); + Console.WriteLine($" READ gap vs SQLite: legacy {sqlite.ReadOpsPerSec / (double)scdb.ReadOpsPerSec:F1}x " + + $"fixed-width {sqlite.ReadOpsPerSec / (double)scdbFw.ReadOpsPerSec:F1}x " + + $"fixed-width at-rest {sqlite.ReadOpsPerSec / (double)scdbFwAtRest.ReadOpsPerSec:F1}x"); + Console.WriteLine($" UPDATE gap vs SQLite: legacy {sqlite.UpdateOpsPerSec / (double)scdb.UpdateOpsPerSec:F1}x " + $"fixed-width {sqlite.UpdateOpsPerSec / (double)scdbFw.UpdateOpsPerSec:F1}x " + $"fixed-width at-rest {sqlite.UpdateOpsPerSec / (double)scdbFwAtRest.UpdateOpsPerSec:F1}x"); Console.WriteLine($" DELETE gap vs SQLite: legacy {sqlite.DeleteOpsPerSec / (double)scdb.DeleteOpsPerSec:F1}x " + $"fixed-width {sqlite.DeleteOpsPerSec / (double)scdbFw.DeleteOpsPerSec:F1}x " + $"fixed-width at-rest {sqlite.DeleteOpsPerSec / (double)scdbFwAtRest.DeleteOpsPerSec:F1}x"); - Console.WriteLine($" INSERT gap vs SQLite: legacy {sqlite.InsertOpsPerSec / (double)scdb.InsertOpsPerSec:F1}x " - + $"fixed-width {sqlite.InsertOpsPerSec / (double)scdbFw.InsertOpsPerSec:F1}x " - + $"fixed-width at-rest {sqlite.InsertOpsPerSec / (double)scdbFwAtRest.InsertOpsPerSec:F1}x"); Console.WriteLine($" At-rest tax (same arm shape): INSERT {scdbFw.InsertOpsPerSec / (double)scdbFwAtRest.InsertOpsPerSec:F2}x " + $"READ {scdbFw.ReadOpsPerSec / (double)scdbFwAtRest.ReadOpsPerSec:F2}x " + $"UPDATE {scdbFw.UpdateOpsPerSec / (double)scdbFwAtRest.UpdateOpsPerSec:F2}x " @@ -5186,6 +5217,11 @@ static int Median(IReadOnlyList runs, Func r.InsertOpsPerSec), ReadOpsPerSec = Median(runs, static r => r.ReadOpsPerSec), UpdateOpsPerSec = Median(runs, static r => r.UpdateOpsPerSec), diff --git a/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/comparative_20261011_125436.json b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/comparative_20261011_125436.json new file mode 100644 index 00000000..48f24146 --- /dev/null +++ b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/comparative_20261011_125436.json @@ -0,0 +1,72 @@ +{ + "SharpCoreDB (SQL)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.6910538, + "InsertOpsPerSec": 144706, + "ReadTime": 0.0950118, + "ReadOpsPerSec": 105250, + "UpdateTime": 0.0566345, + "UpdateOpsPerSec": 176570, + "DeleteTime": 0.0746144, + "DeleteOpsPerSec": 134022, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "SharpCoreDB (Direct)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.5355662, + "InsertOpsPerSec": 186718, + "ReadTime": 0.0522264, + "ReadOpsPerSec": 191474, + "UpdateTime": 0.0452299, + "UpdateOpsPerSec": 221092, + "DeleteTime": 0.030633, + "DeleteOpsPerSec": 326445, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "SharpCoreDB (StructRow)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.5323712, + "InsertOpsPerSec": 187838, + "ReadTime": 0.058933, + "ReadOpsPerSec": 169684, + "UpdateTime": 0, + "UpdateOpsPerSec": 0, + "DeleteTime": 0, + "DeleteOpsPerSec": 0, + "UpdateMeasured": false, + "DeleteMeasured": false + }, + "SQLite": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.5709774, + "InsertOpsPerSec": 175138, + "ReadTime": 0.091383, + "ReadOpsPerSec": 109429, + "UpdateTime": 0.013929, + "UpdateOpsPerSec": 717926, + "DeleteTime": 0.01025, + "DeleteOpsPerSec": 975609, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "LiteDB": { + "Reps": 1, + "SqlitePragmas": null, + "InsertTime": 1.4202476, + "InsertOpsPerSec": 70410, + "ReadTime": 0.6339357, + "ReadOpsPerSec": 15774, + "UpdateTime": 0.9058943, + "UpdateOpsPerSec": 11038, + "DeleteTime": 0.6720018, + "DeleteOpsPerSec": 14880, + "UpdateMeasured": true, + "DeleteMeasured": true + } +} \ No newline at end of file diff --git a/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_comparative_20261011_123404.json b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_comparative_20261011_123404.json new file mode 100644 index 00000000..6b5306ac --- /dev/null +++ b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_comparative_20261011_123404.json @@ -0,0 +1,58 @@ +{ + "SharpCoreDB (SQL, PK, legacy, plaintext)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.4290752, + "InsertOpsPerSec": 233059, + "ReadTime": 0.0744223, + "ReadOpsPerSec": 134368, + "UpdateTime": 0.0277771, + "UpdateOpsPerSec": 360008, + "DeleteTime": 0.0235335, + "DeleteOpsPerSec": 424926, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "SharpCoreDB (SQL, PK, fixed-width, plaintext)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.4412905, + "InsertOpsPerSec": 226608, + "ReadTime": 0.0706219, + "ReadOpsPerSec": 141599, + "UpdateTime": 0.0187298, + "UpdateOpsPerSec": 533908, + "DeleteTime": 0.014552, + "DeleteOpsPerSec": 687190, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "SharpCoreDB (SQL, PK, fixed-width, at-rest default)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.5204726, + "InsertOpsPerSec": 192133, + "ReadTime": 0.0934272, + "ReadOpsPerSec": 107035, + "UpdateTime": 0.0342727, + "UpdateOpsPerSec": 291777, + "DeleteTime": 0.020485, + "DeleteOpsPerSec": 488162, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "SQLite": { + "Reps": 3, + "SqlitePragmas": "journal_mode=wal, synchronous=1", + "InsertTime": 0.5636737, + "InsertOpsPerSec": 177407, + "ReadTime": 0.0892396, + "ReadOpsPerSec": 112057, + "UpdateTime": 0.0125697, + "UpdateOpsPerSec": 795563, + "DeleteTime": 0.0096551, + "DeleteOpsPerSec": 1035722, + "UpdateMeasured": true, + "DeleteMeasured": true + } +} \ No newline at end of file diff --git a/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_comparative_20261011_125346.json b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_comparative_20261011_125346.json new file mode 100644 index 00000000..e6110668 --- /dev/null +++ b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_comparative_20261011_125346.json @@ -0,0 +1,58 @@ +{ + "SharpCoreDB (SQL, PK, legacy, plaintext)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.4581757, + "InsertOpsPerSec": 218256, + "ReadTime": 0.0572881, + "ReadOpsPerSec": 174556, + "UpdateTime": 0.0269016, + "UpdateOpsPerSec": 371725, + "DeleteTime": 0.0214381, + "DeleteOpsPerSec": 466459, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "SharpCoreDB (SQL, PK, fixed-width, plaintext)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.4211281, + "InsertOpsPerSec": 237457, + "ReadTime": 0.0722035, + "ReadOpsPerSec": 138497, + "UpdateTime": 0.0159047, + "UpdateOpsPerSec": 628744, + "DeleteTime": 0.0139568, + "DeleteOpsPerSec": 716496, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "SharpCoreDB (SQL, PK, fixed-width, at-rest default)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0.5245267, + "InsertOpsPerSec": 190648, + "ReadTime": 0.0884405, + "ReadOpsPerSec": 113070, + "UpdateTime": 0.0303918, + "UpdateOpsPerSec": 329036, + "DeleteTime": 0.0219782, + "DeleteOpsPerSec": 454996, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "SQLite": { + "Reps": 3, + "SqlitePragmas": "journal_mode=wal, synchronous=1", + "InsertTime": 0.5467421, + "InsertOpsPerSec": 182901, + "ReadTime": 0.0886978, + "ReadOpsPerSec": 112742, + "UpdateTime": 0.0123272, + "UpdateOpsPerSec": 811214, + "DeleteTime": 0.0100101, + "DeleteOpsPerSec": 998991, + "UpdateMeasured": true, + "DeleteMeasured": true + } +} \ No newline at end of file diff --git a/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_default_20261011_125413.json b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_default_20261011_125413.json new file mode 100644 index 00000000..827f26db --- /dev/null +++ b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/pk_default_20261011_125413.json @@ -0,0 +1,30 @@ +{ + "SharpCoreDB (SQL, PK, default config)": { + "Reps": 3, + "SqlitePragmas": null, + "InsertTime": 0, + "InsertOpsPerSec": 204263, + "ReadTime": 0, + "ReadOpsPerSec": 134634, + "UpdateTime": 0, + "UpdateOpsPerSec": 398709, + "DeleteTime": 0, + "DeleteOpsPerSec": 509941, + "UpdateMeasured": true, + "DeleteMeasured": true + }, + "SQLite": { + "Reps": 3, + "SqlitePragmas": "journal_mode=wal, synchronous=1", + "InsertTime": 0, + "InsertOpsPerSec": 174469, + "ReadTime": 0, + "ReadOpsPerSec": 116916, + "UpdateTime": 0, + "UpdateOpsPerSec": 773323, + "DeleteTime": 0, + "DeleteOpsPerSec": 1085140, + "UpdateMeasured": true, + "DeleteMeasured": true + } +} \ No newline at end of file From 2663317478e2b29afeba5076e1566f50aa11d481 Mon Sep 17 00:00:00 2001 From: MPCoreDeveloper Date: Sun, 11 Oct 2026 15:57:54 +0200 Subject: [PATCH 2/6] docs(readme): the front page is a 2.1-line page, and the stale current-line docs with it The README was a 2.1-line page in its headers and a 2.0-line page in its instructions: the Quick Start installed 2.0.0.3 -- the parked net10 line -- in all three of its snippets, so a .NET 11 user following it got the wrong line entirely. Fixed, and the rest of the page brought onto the line it describes. README - Quick Start (core, Server.Core, Client) and the package block install 2.1.0. - What is new in v2.x -> What is new in v2.1.0: split into storage/record layout, SQLite-parity SQL semantics, performance/allocation, server and toolchain, with the v2.0-era work kept and labelled Carried from the v2.0.x line. Every claim was checked against the commits reachable from the v2.1.0 tag -- the 24-byte inline capacity (2026-09-23) and the 1 MiB .scdb growth default (2026-09-24) predate the publish and ship in it; the SELECT version() fix landed after the tag and is therefore not claimed as shipped. - The SIMD bullet claimed Vector batch filtering, which this line does not use: Query/SimdFilter.cs is intrinsics-only (AVX-512 -> AVX2 -> SSE2 -> scalar, int32/double, the 512-bit tier gated at 1.024 elements) as .github/SIMD_STANDARDS.md requires. - Core-suite count 1,995 (promotion record) -> 1,998 / 0 failed, re-measured on this line. - New Requirements and the two release lines table: TFM, language, SDK pin, pipeline, package version and status per line, plus SQLite parity, the SIMD standard and the durability rails. - The package block lists the complete packable set of 27 (was 19) and no longer warns that a stable 2.1.0 needs the prerelease filter. - Previous releases gains 2.0.0.3. The same defect (a current-line document pinning a superseded install), fixed where the sweep found it - docs/manual/quickstart.md, docs/manual/README.md, docs/scdb/PRODUCTION_GUIDE.md, docs/analytics/README.md, docs/cqrs/README.md, docs/graphrag/README.md: install pins 2.0.0 -> 2.1.0 (PRODUCTION_GUIDE also: should show 10.0 or higher -> 11.0). - docs/INDEX.md: v2.0 labels -> v2.1 (Performance Guide, the manual, the changelog reference [2.0.0.2] -> [2.1.0]). - docs/BENCHMARK_RESULTS.md: the executive summary called the engine a .NET 10 database. CHANGELOG - The [Unreleased] bucket now states where it ends: everything it collected up to the v2.1.0 tag (2026-10-10) shipped in [2.1.0]; only the entries dated 2026-10-11 are unreleased. - [2.1.0] engine behaviour is unchanged from 2.1.0-RC.3 is qualified to the promotion commit itself -- the engine work between the RC.3 tag and the v2.1.0 tag had already landed and is part of these packages. Validation: build 0 errors (287 warnings, the standing set); DocumentationConsistencyTests 3/3; core suite 1998 / 0 failed / 0 errors / 0 skipped on the CI filter in 90,1 s -- the count the README now publishes. No code, config, threshold or version moved. --- README.md | 136 +++++++++++++++++++++++++++++----- docs/BENCHMARK_RESULTS.md | 2 +- docs/CHANGELOG.md | 12 ++- docs/INDEX.md | 8 +- docs/analytics/README.md | 2 +- docs/cqrs/README.md | 2 +- docs/graphrag/README.md | 6 +- docs/manual/README.md | 6 +- docs/manual/quickstart.md | 4 +- docs/scdb/PRODUCTION_GUIDE.md | 4 +- 10 files changed, 143 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index 194720a5..7d94c93e 100644 --- a/README.md +++ b/README.md @@ -24,11 +24,12 @@ Use it when you need: - Fast embedded storage with **AES-256-GCM encryption** and ACID guarantees - A secure network database via **gRPC (HTTP/2 + HTTP/3)** - Built-in **vector search**, **advanced analytics**, and **GraphRAG/graph algorithms** -- A production-focused stack validated by the **core suite on the promoted tree (1,995 tests, 0 failed)**, - the **RC.3 sweep of 2,897 tests across 16 suites**, and **backward compatibility** +- A production-focused stack validated by the **core suite on this line (1,998 tests, 0 failed, + measured 2026-10-11)**, the **RC.3 sweep of 2,897 tests across 16 suites**, and **backward + compatibility** > **Current release: v2.1.0 (2026-10-10) — the .NET 11 / C# 15 line, on `master`.** **Published on -> NuGet.org** (all 27 packages) from `master`, `net11.0` only. Highlights: the fixed-width inline capacity +> NuGet.org** (all 27 packable packages) from `master`, `net11.0` only. Highlights: the fixed-width inline capacity > ships at 24 bytes by default on both storage paths (worth **+34 %** on the batched multi-row INSERT > shape and **0,63× → 0,87×** on the tracked fair-PK INSERT ratio), the overflow arena appends a row's payloads in one call instead of one > file open per value, and in-place UPDATE/DELETE with deferred index maintenance now run the @@ -42,7 +43,8 @@ Use it when you need: > *SQLite* arm allocating a command and two parameters per row — SQLite's own PK UPDATE/DELETE > references rose **3,05×** (288.108 → 878.557 / 385.116 → 1.172.704) once the comparator held one > prepared command, so that band is superseded, not better. Full notes: -> [`docs/2.1.0-RC.3_WHAT_CHANGED.md`](docs/2.1.0-RC.3_WHAT_CHANGED.md) — the RC.3 validation record; +> [`docs/2.1.0-RC.3_WHAT_CHANGED.md`](docs/2.1.0-RC.3_WHAT_CHANGED.md) — the RC.3 validation record +> (its own install snippets pin the RC.3 packages; the shipped line is `2.1.0`); > the promotion itself re-verified the core suite at **1995 / 0 failed** and is documented in > [`docs/net11/PROMOTION_TO_MAIN.md`](docs/net11/PROMOTION_TO_MAIN.md). @@ -69,6 +71,30 @@ Use it when you need: > moved to the standalone repo **[MPCoreDeveloper/SCDMS](https://github.com/MPCoreDeveloper/SCDMS)**. > This repository now contains only the engine, providers, and server. See [docs/SCDMS.md](docs/SCDMS.md). +## Requirements and the two release lines + +| | **v2.1.0 — this line (`master`)** | **v2.0.0.3 — parked line (`release/v2.0`)** | +|---|---|---| +| Target framework | `net11.0` only (every project under `src`) | `net10.0` | +| Language | C# 15 (`LangVersion=preview`, always on) | C# 14 | +| SDK | .NET 11 RC, pinned in `global.json` (`11.0.100-rc.1.26425.128`, `allowPrerelease`, `rollForward: latestPatch`) | .NET 10 (`10.0.x`) | +| Pipeline | `.github/workflows/ci.yml` | `.github/workflows/ci-net10.yml` | +| Packages | `2.1.0` — the 27 packable projects under `src` | `2.0.0.3` | +| Status | current release, receives new work | servicing only | + +- **Pick one line per solution.** The package references are line-specific: a `net10.0` project uses the + `2.0.0.x` packages, a `net11.0` project the `2.1.0` ones. Do not mix them in one solution. +- **The .NET 11 SDK is required to build** — the `global.json` pin is deliberate and `dotnet test` is not + this repository's runner (Microsoft.Testing.Platform is; run the built test executable, or see + [`docs/INDEX.md`](docs/INDEX.md)). +- **Runtimes shipped:** `win-x64`, `win-arm64`, `linux-x64`, `linux-arm64`, `osx-x64`, `osx-arm64`. +- **SQLite compatibility:** supported SQLite syntax and features are never fewer than SQLite's; extra + capability is fine. `INTEGER PRIMARY KEY` shapes are the like-for-like comparison in the numbers below. +- **SIMD standard:** `System.Runtime.Intrinsics` only (`AVX-512 → AVX2 → SSE2 → scalar`), never + `System.Numerics.Vector` — see [`.github/SIMD_STANDARDS.md`](.github/SIMD_STANDARDS.md). +- **Encryption and durability are never traded for speed:** AES-256-GCM at rest, WAL semantics and the + crash-consistency / reopen round-trip matrix are release gates. + ## Why SharpCoreDB? - **One stack, two deployment models**: embedded and server mode @@ -89,7 +115,7 @@ Use it when you need: --- -## What's new in v2.x (current engine line) +## What's new in v2.1.0 (current release) The v1.x benchmark gap (point reads/updates/deletes **16–52x behind SQLite** because every update appended a new row) is closed as an engine behaviour. The numbers below are re-measured on @@ -122,36 +148,91 @@ they understate SQLite and are superseded. Full tables and the API ladder: **[Performance Guide](docs/manual/performance.md)** · **[`docs/BENCHMARK_RESULTS.md`](docs/BENCHMARK_RESULTS.md)**. Roadmap: **[`docs/performance/V2_PERFORMANCE_PLAN.md`](docs/performance/V2_PERFORMANCE_PLAN.md)**. -Headline changes: -- ⚡ **Fixed-width record layout (default for new PK tables)** — constant-stride records with an +Headline changes, grouped by area. Everything below is in the shipped `2.1.0` packages — the +`2.1.0-RC.3` line plus the work that landed between that tag and `v2.1.0`. + +**Storage and record layout** + +- ⚡ **Fixed-width record layout is the default for new PK tables** — constant-stride records with an out-of-line `.ovf` overflow arena; UPDATE/DELETE are **in-place overwrites** with commit-time tombstone markers (no full-file rewrite) +- ⚡ **The inline capacity ships at 24 bytes on both storage paths** — multi-file and single-file + (`.scdb`) — worth **+34 %** on the batched multi-row INSERT shape and **0,63× → 0,87×** on the tracked + fair-PK INSERT ratio; 32 measures inside the noise at +20 % of the file, so 24 is the knee +- ⚡ **Existing databases upgrade their record layout themselves on the first writable open** — no + rewrite, no export/import, and no format break (the capacity is part of the persisted layout) +- ⚡ **Single-file (`.scdb`) mode now uses the same record layout as multi-file mode** (it never had) and + honours the configured inline capacity +- ⚙️ **`.scdb` growth is configurable** — `DatabaseConfig.SingleFileMinExtensionBytes`, minimum **1 MiB** + (was 10 MiB): every small database drops from 14,7 MB to **6,4 MB** +- 🗜️ **Block-level Brotli/GZip compression** for single-file (`.scdb`) storage — transparent, per-block, + applied before encryption / removed after decryption (#344) +- 🛡️ **Envelope encryption + full at-rest metadata encryption** — password-based per-file DEK + (PBKDF2-HMAC-SHA256), encrypted block registry / FSM / WAL, key & password rotation APIs (#341 follow-on) +- ⚙️ **Configurable metadata region sizing** — `FsmSizePages` / `BlockRegistrySizePages` / + `TableDirectorySizePages` remove the 512 MB single-file ceiling (#345) + +**SQL semantics (SQLite parity)** + +- ⚡ **A declared column type is honoured the way SQLite honours it** — `DOUBLE`/`FLOAT` are REAL + columns, `INT`/`SMALLINT`/`INT8` are INTEGER columns, resolved by a single type map +- ⚡ **`DELETE FROM t` with no `WHERE` deletes every row**, as SQLite does — it used to be refused +- ⚡ **`INSERT` with an explicit column list writes the named order** through every batch path, not + positionally +- ⚡ **`information_schema` is implemented in the engine**, so every protocol answers it +- ⚡ **The async entry points validate like their synchronous twins** — `ExecuteSQLAsync` validates, + `ExecuteBatchSQLAsync` composes inside a caller's transaction + +**Performance and allocation (this line)** + +- ⚡ **Index maintenance left the write path** — automatic hash indexes register **on demand**, the first + time an operation filters on a column, instead of one per column at `CREATE TABLE`; arm-B/C INSERT + reaches **1,60× / 1,26×** +- ⚡ **Single-key index operations no longer dispatch through reflection** (`IndexManager`, + `GenericHashIndex`, `Table.BTreeIndexing`) +- ⚡ **Batch INSERT's hash-index maintenance no longer rebuilds its key map per batch** — + `hash-index` 4.611.325 → 3.530.478 B per 10.000-row call +- ⚡ **The fixed-width codec allocates no `List<>` scratch per row** (706 → 547 B/row on the fair-PK + INSERT arm), and the encrypted UPDATE locate no longer copies every ciphertext frame before decrypting + it (431 → 279 B/record) +- ⚡ **Zstd blocks no longer round-trip through a stream** — 2.480 → 0 B of managed allocation per 4 KB block +- 🧪 **CI is split by line instead of by SDK** (`ci.yml` here, `ci-net10.yml` on the parked branch), and + the 18 % coverage floor is measured by a dedicated job rather than carried unmeasured + +**Server** + +- 🔧 **A plain client disconnect is no longer logged as an authentication failure** — a peer that aborts + its socket (TCP reset, Winsock 10053/10054) is now reported as what it is +- 🔧 **The server image is net11-aware** (`sdk:11.0` / `aspnet:11.0`) + +### Carried from the v2.0.x line + - ⚡ **Single-pass contiguous UPDATE/DELETE (B8/B9)** — strictly ascending `pk = literal` batches resolved as ONE contiguous range with decode verification (no per-key B-tree search) -- ⚡ **Reopen data-integrity hardening (2.0.0.2)** — empty-value overflow reload, single-file - fixed-width arena markers, and legacy 1.x delete-after-update purge (`ReopenRoundTripMatrixTests`) - ⚡ **Zero-allocation reads** — first-class `ExecuteQueryStruct(sql, params)` API + `VariableLengthSchema` cache - ⚡ **In-place UPDATE patches (WP11)** — single-row UPDATE overwrites changed fields at cached fixed column offsets (no deserialize → re-serialize round trip); growing records relocate across pages with correct PK/hash re-pointing - ⚡ **Unified DELETE core (WP12)** — one shared delete path with key-only hash index cleanup - ⚡ **Exact-size row serialization (WP13)** — one allocation per insert/update (no `ArrayPool.Rent` + `ToArray()` double allocation) - ⚡ **Zero-reparse SELECT fast path** — `SimpleSelectPlan` resolves simple point-lookup plans without re-lexing -- ⚡ **SIMD numeric WHERE filters** — `Vector` batch filtering for Integer/Long columns +- ⚡ **SIMD WHERE filters** — tiered `System.Runtime.Intrinsics` kernels (AVX-512 → AVX2 → SSE2 → scalar, + mask-based, with the 512-bit tier gated at 1.024 elements) for `int32` and `double` columns; + `System.Numerics.Vector` is deliberately **not** used, so codegen is deterministic under NativeAOT - ⚡ **Compiled regexes everywhere** on hot paths; regex-free `NormalizeSql`; cached DI - ⚡ **Removed hidden `D:\*.log` debug writes** that throttled every SELECT/execute/transaction/INSERT - 🛡️ **Native AOT readiness** — AOT-safe `TypeConverter`, `Option` reader, `[RequiresDynamicCode]` annotations, source-generated DTOs/JSON (`tools/SharpCoreDB.AotSmoke` publishes + runs, exit 0) -- ✅ validated by the **core suite on the promoted tree (1,995 tests, 0 failed)** — the **RC.3 sweep covered 2,897 tests across 16 suites** (CQRS 64, VectorSearch 248, EF Core 116, Provider.Sync 135, Search 58, HybridSearch 6, Functional.Linq2DB 24, …) -- 🛡️ **Envelope encryption + full at-rest metadata encryption** — password-based per-file DEK (PBKDF2-HMAC-SHA256), encrypted block registry / FSM / WAL, key & password rotation APIs (#341 follow-on) -- 🗜️ **Block-level Brotli/GZip compression** for single-file (`.scdb`) storage — transparent, per-block, applied before encryption / removed after decryption (#344) -- ⚙️ **Configurable metadata region sizing** — `FsmSizePages` / `BlockRegistrySizePages` / `TableDirectorySizePages` remove the 512 MB single-file ceiling; byte-based file extension (~10 MB regardless of `PageSize`) (#345) +- 🛡️ **Reopen data-integrity hardening (2.0.0.2)** — empty-value overflow reload, single-file + fixed-width arena markers, and legacy 1.x delete-after-update purge (`ReopenRoundTripMatrixTests`) +- ✅ **Validated by the core suite on this line (1,998 tests, 0 failed, measured 2026-10-11)** — the + **RC.3 sweep covered 2,897 tests across 16 suites** (CQRS 64, VectorSearch 248, EF Core 116, Provider.Sync 135, Search 58, HybridSearch 6, Functional.Linq2DB 24, …) ## Quick Start (under 30 seconds) ### 1) Embedded mode ```bash -dotnet add package SharpCoreDB --version 2.0.0.3 +dotnet add package SharpCoreDB --version 2.1.0 ``` ```csharp @@ -188,8 +269,8 @@ gRPC endpoint: `https://localhost:5001` Install client/server packages: ```bash -dotnet add package SharpCoreDB.Server.Core --version 2.0.0.3 -dotnet add package SharpCoreDB.Client --version 2.0.0.3 +dotnet add package SharpCoreDB.Server.Core --version 2.1.0 +dotnet add package SharpCoreDB.Client --version 2.1.0 ``` > The server host is a standalone executable (`src/SharpCoreDB.Server`), distributed as the @@ -200,8 +281,12 @@ dotnet add package SharpCoreDB.Client --version 2.0.0.3 --- -## Previous releases (v1.9.x) +## Previous releases +- **v2.0.0.3** — the net10.0 / C# 14 stable line, parked on `release/v2.0` and built by `ci-net10.yml` + (the complete package list is above). Engine foundation it carries: fixed-width record layout by default + for new PK tables, in-place UPDATE/DELETE with commit-time tombstones, and the reopen data-integrity + hardening batch. - **v1.9.6** — fixed the critical `WHERE col IN (...)` regression (Issue #339); SonarCloud quality-gate onboarding; 2,404 tests passing. - **v1.9.5** — full version synchronization across all packages; token-aware named-parameter binding fix (Issue #336); server parameter forwarding fix (Issue #337); standards-compliant Crockford Base32 ULID encoding; new `SharpCoreDB.Functional.Linq2DB` adapter. - **v1.9.1** — EF Core provider with Guid-keyed entity CRUD + reliable two-query relationship materialization; 22/22 EF Core integration tests; runnable EF Core demo; version alignment across 25+ packages. @@ -220,7 +305,10 @@ dotnet add package SharpCoreDB --version 2.1.0 # Server/client dotnet add package SharpCoreDB.Server.Core --version 2.1.0 +dotnet add package SharpCoreDB.Server.Protocol --version 2.1.0 dotnet add package SharpCoreDB.Client --version 2.1.0 +dotnet add package SharpCoreDB.Client.Protocol --version 2.1.0 +dotnet add package SharpCoreDB.Data.Provider --version 2.1.0 # Engines and extensions dotnet add package SharpCoreDB.Analytics --version 2.1.0 @@ -231,8 +319,11 @@ dotnet add package SharpCoreDB.Graph --version 2.1.0 dotnet add package SharpCoreDB.Graph.Advanced --version 2.1.0 dotnet add package SharpCoreDB.Distributed --version 2.1.0 dotnet add package SharpCoreDB.Provider.Sync --version 2.1.0 +dotnet add package SharpCoreDB.Provider.YesSql --version 2.1.0 dotnet add package SharpCoreDB.EntityFrameworkCore --version 2.1.0 dotnet add package SharpCoreDB.Extensions --version 2.1.0 +dotnet add package SharpCoreDB.Identity --version 2.1.0 +dotnet add package SharpCoreDB.Serilog.Sinks --version 2.1.0 # Optional architecture packages dotnet add package SharpCoreDB.EventSourcing --version 2.1.0 @@ -244,10 +335,15 @@ dotnet add package SharpCoreDB.Functional --version 2.1.0 dotnet add package SharpCoreDB.Functional.Dapper --version 2.1.0 dotnet add package SharpCoreDB.Functional.EntityFrameworkCore --version 2.1.0 dotnet add package SharpCoreDB.Functional.Linq2DB --version 2.1.0 + +# Hosting and tooling +dotnet add package SharpCoreDB.Aspire.Hosting --version 2.1.0 ``` -> Pre-release versions are **not** selected by default — request the version explicitly (as above) or -> tick *Include prerelease* in Visual Studio. +> The 27 commands above are the complete packable set of this line. **`2.1.0` is a stable release**, so +> NuGet selects it by default for a `net11.0` project; a `net10.0` project resolves to the `2.0.0.3` line +> instead. Pin the version explicitly (as above) when you want to be certain, and tick *Include +> prerelease* only if you are deliberately testing an RC build. > > The **server host is a standalone executable**, not a NuGet package: run > `dotnet run --project src/SharpCoreDB.Server -c Release` or use the diff --git a/docs/BENCHMARK_RESULTS.md b/docs/BENCHMARK_RESULTS.md index 417ceb79..af54a586 100644 --- a/docs/BENCHMARK_RESULTS.md +++ b/docs/BENCHMARK_RESULTS.md @@ -78,7 +78,7 @@ index maintenance); see the v2.1.0 section above for where the ratio stands toda ## Executive Summary -SharpCoreDB is a high-performance embedded database for .NET 10. This document presents comprehensive benchmark results comparing SharpCoreDB against **SQLite** and **LiteDB**. +SharpCoreDB is a high-performance embedded database for .NET 11. This document presents comprehensive benchmark results comparing SharpCoreDB against **SQLite** and **LiteDB**. ### Key Findings diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 7da00352..4517ad64 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +> **Where this bucket ends.** The `v2.1.0` tag was cut from this line's `HEAD` on 2026-10-10, so every +> entry this bucket had collected up to that tag — the net11.0 line's work since `2.0.0.3` — **shipped in +> `[2.1.0]`** below; the entries dated 2026-10-11 are the only still-unreleased ones (`git log v2.1.0..HEAD`). + ### Added - **The 18% coverage floor is measured and enforced by a dedicated CI job instead of being carried unmeasured.** There was no working coverage route in this repository: `dotnet test` is rejected because xUnit v3 runs on Microsoft.Testing.Platform, `SharpCoreDB.Tests.exe --coverage` answers `error: unknown option: --coverage` (and `--help` lists no coverage option) because the host is xUnit's own in-process runner, `Microsoft.Testing.Extensions.CodeCoverage` registers that switch with an MTP host only, and `coverlet.collector` is a VSTest data collector. `.github/workflows/ci.yml` now has a `coverage` job (ubuntu-latest, once) that measures the three suites `codecov.yml` counts through the standalone collector wrapped around the same test hosts the build job runs, and `tools/measure-coverage.ps1` binds it together: it builds, collects one Cobertura report per suite, merges them, prints a per-assembly table and fails below the floor (exit 0 at or above it, 1 below, 2 when a suite failed — verdict withheld — 3 when no measurement was possible). `dotnet-coverage` **18.11.2** is pinned in a new `.config/dotnet-tools.json`, so the measurement is reproducible from one version pin, and the job uploads the reports as an artifact and to Codecov. Measured on this branch: **45.53% of 60050 shipped lines** (core 52.96%, VectorSearch 83.66%, Graph 67.46%, EF Core 39.76%), with all three suites green — 1946 + 248 + 116 tests, 0 failed — against the 18% floor. @@ -19,6 +23,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **The published benchmark pages now carry the 2026-10-11 v2.1.0 numbers, and the fair-PK "ahead on UPDATE/DELETE" claim they used to carry is corrected rather than dropped — because it was measured against a crippled SQLite arm.** `README.md`, `docs/BENCHMARK_RESULTS.md`, `docs/manual/performance.md`, `docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md` and `docs/benchmarks/default-config-pk.md` are restamped to v2.1.0 / .NET 11 / `92794084` and re-tabulated from one **same-run** measurement set (`pk_default_20261011_125413.json`, `comparative_20261011_125436.json`): default posture fair-PK **INSERT 1,17× (1,03–1,27), READ 1,13× (1,12–1,15), UPDATE 0,53× (0,42–0,53), DELETE 0,47× (0,46–0,62)** against that run's own SQLite, and the record-layout A/B shows the fixed-width layout **1,2–1,5× ahead of the legacy variable-length layout** on UPDATE/DELETE at a **1,2–1,9×** per-record-encryption cost. Two published figures were wrong and are now stated with their cause: the inline-capacity default is **24** bytes, not the 16 the README still said (worth **+34 %** on the batched multi-row INSERT shape and fair-PK INSERT ratio **0,63× → 0,87×**), and the "fair-PK layout ahead of SQLite on UPDATE (0,8×) / DELETE (0,5×)" reading came from the window in which the harness left the ***SQLite*** arm allocating a command and two parameters per row — SQLite's own PK references are **3,05×** higher with the prepared comparator (288.108 → 878.557 / 385.116 → 1.172.704), so the honest reading is *behind* on those two cells and the earlier band is superseded, not better. Three harness defects were found while publishing and fixed in the same session: the `--pk` banner's "legacy variable-length" arm now genuinely opts out of the fixed-width layout (`BuildConfig(legacyVariableLength: true)` → `AutoFixedWidthRecords = false`; before the fix the two layout arms read ~3 % apart instead of the 1,3–2,3× the layout is worth), the `[diag]` line prints the **resolved** config instead of the requested parameters (the encrypted stock arm had printed `noEncrypt=True`), and `RunPkMedian` now stamps `Reps`, so a `pk_comparative_*.json` archive describes its protocol as measured instead of claiming `"Reps": 1` for a 3-rep median. Pinned by `BenchmarkProtocolArchiveTests` (`PkLegacyArm_ResolvesToVariableLengthRecords_AsItsBannerClaims`, `PkComparisonArchive_RecordsTheMeasuredRepCount_NotTheSingleShotDefault`; both **red before, green after**), Manual-tagged and skipping when the harness exe is absent. No engine code, config default, threshold or version moved. +- **The front page is a `2.1`-line page, and the last current-line documents that still sent a reader to the superseded install are corrected.** `README.md` now states the line it actually describes: the Quick Start installed `2.0.0.3` — the parked net10 line — in all three of its snippets (core, `SharpCoreDB.Server.Core`, `SharpCoreDB.Client`), so a `.NET 11` user following it got the wrong line entirely; the "What's new in v2.x" list mixed the v2.0-era foundation into the current release and is now split into the areas the `2.1.0` packages actually ship (storage and record layout, SQLite-parity SQL semantics, performance and allocation, server, toolchain), with the v2.0-era work kept and labelled **Carried from the v2.0.x line**; the SIMD bullet claimed `Vector` batch filtering, which this line does **not** use — `Query/SimdFilter.cs` is intrinsics-only (AVX-512 → AVX2 → SSE2 → scalar, `int32`/`double`, the 512-bit tier gated at 1.024 elements), as `.github/SIMD_STANDARDS.md` requires; the validated-by line pinned the core suite at the promotion's 1.995 against **1.998 / 0 failed** measured on 2026-10-11; a **Requirements and the two release lines** table was added (TFM, language, SDK pin, pipeline, package version and status per line, plus the SQLite-compatibility and SIMD standards); the package block now lists the complete packable set of **27** instead of 19 and no longer warns that a *stable* `2.1.0` needs the prerelease filter; and the previous-releases section gained `2.0.0.3`. The same defect — a current-line document pinning a superseded install — was fixed where the sweep found it: `docs/manual/quickstart.md` (`2.0.0` → `2.1.0`, and "the idiomatic v2.0 API surface"), `docs/manual/README.md` (title, description and version line), `docs/scdb/PRODUCTION_GUIDE.md` (`2.0.0` → `2.1.0` and "should show 10.0 or higher" → 11.0), `docs/analytics/README.md`, `docs/cqrs/README.md`, `docs/graphrag/README.md`, the `v2.0` labels in `docs/INDEX.md` (Performance Guide, manual, changelog reference) and `docs/BENCHMARK_RESULTS.md`'s executive summary, which still called the engine a `.NET 10` database. No code, config, threshold or version moved. + - **A Columnar table now registers an automatic hash index for a column the first time an operation filters on it, instead of registering one per column at `CREATE TABLE` — so a write stops maintaining indexes the workload never uses.** `DatabaseConfig.EnableHashIndexes` (default `true`) had always meant "auto-create a hash index per column": `SqlParser.DDL.cs` registered the primary key *and every other column*, and because `Table.EnsureAllRegisteredIndexesLoaded()` loads every **registered** index before a write on the append/Columnar engines, the five-column `docs` shape paid a hash remove+add per row on all five columns while the workload filtered on one — the measured 20.000 `index-maint` calls per 10.000 updates. Registration is now on demand: a user-declared primary key is still registered up front, every other column registers itself through the single gate `Table.EnsureAutoHashIndexRegistered`, which every lookup path now shares (the read path, the position-aware update locate, both delete cores, the parallel scan, both struct-scan fast paths and structured DML), and the hidden `_rowid` fallback is never registered because the PK B-tree already serves its point lookups. *Building* was already lazy (`Table.EnsureIndexLoaded`), so nothing else in the index lifecycle changed, `HashIndexAutoCreationGateTests` was rewritten to the new contract (primary key eagerly, others on the first filtered operation, none with the dial off) rather than loosened, and `SHARPCOREDB_HASH_INDEXES=0` still turns automatic indexes off entirely. **Measured on arms B and C (`--pk-default-batch` / `--docs-batch`, INSERT vs SQLite, median with the run's own per-rep range): arm B batch 0,95× → 1,60× (1,05–1,92) and arm B SQL 0,83× → 1,05× (0,94–1,26); arm C batch 1,26× (1,19–1,54) and arm C SQL 0,90× → 0,98× (0,98–1,29)** — with same-build `SHARPCOREDB_HASH_INDEXES=0` runs as the ceiling on the day (arm B 1,68× batch / 1,31× SQL, arm C 1,25× batch / 0,95× SQL), i.e. what is left of the margin is the eagerly registered declared-primary-key index. On the query side the cost is now visible instead of hidden: `--auto-index-benefit` reports the **first** filtered query separately, because that is the query that pays for building the index — 20.000 rows, 2.000 point queries per column, two runs: the **first** query costs 35,5–71,0 ms on `email`/`score`/`age` and the warm queries after it cost 4,7–14,1 µs (`email`, `score`) against 4,8–5,0 ms with the dial off (**354×–988×** and **3.106×–7.656×**), with the explicitly indexed control unchanged (63–68 µs first, 5,9–6,3 µs repeated) and identical row counts in both configurations. Core suite **1952 / 0 failed / 0 errors**; `--gate` **PASSED, 8/8 metrics `ok`**, all eight ahead of the recorded baseline (default INSERT 0,78×, default UPDATE 0,31×). - **The single-file growth minimum is now 1 MiB (was 10 MiB), which takes every small `.scdb` database from 14,7 MB to 6,4 MB.** The 10 MiB minimum was what made a 100-row `.scdb` file 14,7 MB: a fresh file is 1.037 pages and the extension is `max(requiredPages, currentSize / 2, minExtensionPages)`, where 10 MiB is 2.560 pages at a 4 KiB page size — so the minimum won the contest and the first extension landed on 3.597 pages = **14.733.312 B**, the same size at 1 row and at 2.000 rows (measured 2026-09-23, which is what proved it was a floor rather than growth). At the new 1 MiB default the minimum no longer binds (256 pages against the halving term's 518) and the same database lands on **6.369.280 B, −56,8 %**, measured at **1 / 100 / 200 / 500 / 2.000 / 20.000 / 200.000 / 1.000.000 rows** with **byte-identical allocation per row** and no measurable throughput difference — the single-file blocks are compressed, so even a million rows of the benchmark schema stay inside that first extension. Existing databases are unaffected: the value is read **per open**, never changes how a file is read, and the historical 14,7 MB floor is one configuration line away (`SingleFileMinExtensionBytes = 10 * 1024 * 1024`), pinned by `SingleFileFileGrowthTests.ExplicitHistoricalTenMiBMinimum_StillProducesTheOldFloor` so the old behaviour stays reproducible byte for byte. - **The inline capacity's default is now 24 bytes (was 16), and it is worth 0,63× → 0,87× on the fair-PK INSERT ratio.** `DatabaseConfig.FixedWidthInlineValueBytes` decides how many payload bytes a variable-length column may keep **inside** its fixed-width record slot instead of writing them to the overflow arena, and 16 was two bytes short of the most common short-value shape: on the benchmark schema the 18-byte `user99999@test.com` overflowed for every row whose index is ≥ 1.000, so **99.000 of 100.000 rows paid an arena write each**. At 24 every value in that schema fits inline and `arena-write`, `arena-append` and `encode-scratch` fall to **zero calls**, measured on the harness's own `[diag]` counter and on `--pk-profile-insert`. @@ -80,8 +86,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [2.1.0] - 2026-10-10 > The RC.3 line below, promoted to the main branch as **2.1.0**: `master` is the net11.0 / C# 15 preview -> line and the net10.0 / C# 14 line is parked on `release/v2.0`. Engine behaviour is unchanged from -> `2.1.0-RC.3` — what changed is the release identity and the metadata that ships with it. Procedure and +> line and the net10.0 / C# 14 line is parked on `release/v2.0`. **The promotion commit itself** changes +> the release identity and the metadata that ships with it, and nothing else — the engine work between the +> `v2.1.0-RC.3` tag and this one had already landed on the line and is recorded in the `[Unreleased]` bucket +> above, whose entries up to the `v2.1.0` tag are part of these packages. Procedure and > gates: [`net11/PROMOTION_TO_MAIN.md`](net11/PROMOTION_TO_MAIN.md). Verified on the promoted tree: > build 0 errors and **2,383 tests / 0 failed / 0 errors / 0 skipped** across the four CI suites > (core 1,995 · VectorSearch 248 · EF Core 116 · linq2db 24). **Published to NuGet.org on 2026-10-10**: diff --git a/docs/INDEX.md b/docs/INDEX.md index 61e34d49..d02a09d5 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -14,7 +14,7 @@ guide explaining when SharpCoreDB is fastest. - `manual/data-modeling.md` — types, ULID/rowid keys, constraints, collation - `manual/indexing.md` — hash, B-tree, expression, partial indexes - `manual/query.md` — SQL dialect, aggregates, window functions, joins, subqueries -- `manual/performance.md` — ⭐ **Performance Guide** (measured v2.0 numbers, API ladder, examples) +- `manual/performance.md` — ⭐ **Performance Guide** (measured v2.1 numbers, API ladder, examples) - `manual/simd-vector.md` — SIMD analytics, vector search, GraphRAG, time-series - `manual/server.md` — gRPC/REST server, security, multitenancy, observability - `manual/providers.md` — ADO.NET, EF Core, Dapper, linq2db, YesSql, Sync @@ -42,7 +42,7 @@ guide explaining when SharpCoreDB is fastest. - `CHANGELOG.md` - `CONTRIBUTING.md` - `UseCases.md` -- `manual/README.md` — the v2.0 manual (see Section 0) +- `manual/README.md` — the v2.1 manual (see Section 0) ## 2. Package Documentation (src) @@ -177,10 +177,10 @@ guide explaining when SharpCoreDB is fastest. 2026-09-17, and the remaining gaps - `2.0.0.2_WHAT_CHANGED.md` — ⭐ **release notes (2.0.0.2)**: the 2.x-vs-1.9.x major steps + the 2.0.0.2 hardening/perf batch -- `CHANGELOG.md` — full per-version change log (`[2.0.0.2]`, new `[Unreleased]` on top) +- `CHANGELOG.md` — full per-version change log (`[2.1.0]`, new `[Unreleased]` on top) - `manual/upgrade-and-downgrade.md` — compatibility matrix & downgrade policy - `benchmarks/default-config-pk.md` — default-config fair-PK benchmark + NoEncryptMode root cause -- `manual/performance.md` — ⭐ **Performance Guide**: when SharpCoreDB is fastest + v2.0 results +- `manual/performance.md` — ⭐ **Performance Guide**: when SharpCoreDB is fastest + v2.1 results - `performance/V2_PERFORMANCE_PLAN.md` — v2.x performance roadmap, root-cause analysis, .NET 11 plan - `benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md` — comparative report vs SQLite/LiteDB/BLite - `BENCHMARK_RESULTS.md` — micro-benchmark suite report (v1.x-era, with v2.0 update) diff --git a/docs/analytics/README.md b/docs/analytics/README.md index 0d70f712..a946e49e 100644 --- a/docs/analytics/README.md +++ b/docs/analytics/README.md @@ -32,7 +32,7 @@ The SharpCoreDB Analytics Engine provides high-performance data aggregation, win ### Installation ```bash -dotnet add package SharpCoreDB.Analytics --version 2.0.0 +dotnet add package SharpCoreDB.Analytics --version 2.1.0 ``` ### Basic Aggregation diff --git a/docs/cqrs/README.md b/docs/cqrs/README.md index f5381e33..cb17731b 100644 --- a/docs/cqrs/README.md +++ b/docs/cqrs/README.md @@ -54,7 +54,7 @@ Non-goals: ## 2. Installation ```bash -dotnet add package SharpCoreDB.CQRS --version 2.0.0 +dotnet add package SharpCoreDB.CQRS --version 2.1.0 ``` --- diff --git a/docs/graphrag/README.md b/docs/graphrag/README.md index 586071ee..d75be33c 100644 --- a/docs/graphrag/README.md +++ b/docs/graphrag/README.md @@ -49,9 +49,9 @@ For package-level mapping, see `../FEATURE_MATRIX.md`. ## Quick start ```bash -dotnet add package SharpCoreDB.Graph.Advanced --version 2.0.0 -dotnet add package SharpCoreDB.Graph --version 2.0.0 -dotnet add package SharpCoreDB.VectorSearch --version 2.0.0 +dotnet add package SharpCoreDB.Graph.Advanced --version 2.1.0 +dotnet add package SharpCoreDB.Graph --version 2.1.0 +dotnet add package SharpCoreDB.VectorSearch --version 2.1.0 ``` ```csharp diff --git a/docs/manual/README.md b/docs/manual/README.md index d4b690b9..e1d7be03 100644 --- a/docs/manual/README.md +++ b/docs/manual/README.md @@ -1,9 +1,9 @@ -# SharpCoreDB v2.0 Manual +# SharpCoreDB v2.1 Manual -> **SharpCoreDB v2.0 — performance-first.** An embedded, encrypted, SIMD-accelerated database +> **SharpCoreDB v2.1 — performance-first.** An embedded, encrypted, SIMD-accelerated database > engine and production gRPC server for .NET 11 / C# 15. > -> Version 2.0.0.0 · MIT License · [GitHub](https://github.com/MPCoreDeveloper/SharpCoreDB) +> Version 2.1.0 · MIT License · [GitHub](https://github.com/MPCoreDeveloper/SharpCoreDB) --- diff --git a/docs/manual/quickstart.md b/docs/manual/quickstart.md index 06ecb334..f105e9f7 100644 --- a/docs/manual/quickstart.md +++ b/docs/manual/quickstart.md @@ -1,14 +1,14 @@ # 1. Quick Start > ⚡ For maximum throughput, jump straight to the [Performance Guide](performance.md) after this -> page. The quick start covers the idiomatic v2.0 API surface. +> page. The quick start covers the idiomatic v2.1 API surface. --- ## 1.1 Install ```bash -dotnet add package SharpCoreDB --version 2.0.0 +dotnet add package SharpCoreDB --version 2.1.0 ``` The engine targets **.NET 11** and **C# 15**. No native dependencies, no external processes — diff --git a/docs/scdb/PRODUCTION_GUIDE.md b/docs/scdb/PRODUCTION_GUIDE.md index ebb560af..8cdc51dd 100644 --- a/docs/scdb/PRODUCTION_GUIDE.md +++ b/docs/scdb/PRODUCTION_GUIDE.md @@ -36,10 +36,10 @@ This guide covers deploying SharpCoreDB with the **SCDB (SharpCore DataBase)** s ```bash # .NET 11 Runtime (required) -dotnet --version # Should show 10.0 or higher +dotnet --version # Should show 11.0 or higher # NuGet Package -dotnet add package SharpCoreDB --version 2.0.0 +dotnet add package SharpCoreDB --version 2.1.0 ``` --- From 996d4148723655d8e6c1a7500cddcbbd593716eb Mon Sep 17 00:00:00 2001 From: MPCoreDeveloper Date: Sun, 11 Oct 2026 16:15:28 +0200 Subject: [PATCH 3/6] docs(perf): publish the NativeAOT arm the docs had been promising -- AOT at or behind the JIT on 12 of 16 cells Session 78 measured exactly this on 2026-10-09 and closed plan item S4 REJECTED; three maintained documents still sold the un-measured expectation and the file users read for benchmark status had no AOT arm at all. The measurement (reported, not re-run): the comparative harness published with -p:PublishAot=true (native win-x64 image, 14.023.680 B) and run as that binary against its own JIT build -- fresh processes, identical options, same box, all 16 cells. Per-cell AOT / JIT, median of the paired runs, in INSERT / READ / UPDATE / DELETE order: legacy plaintext 1,00 / 0,95 / 0,88 / 1,25; fixed-width plaintext 0,96 / 0,98 / 0,93 / 1,08; fixed-width at rest 0,99 / 0,90 / 0,89 / 0,92; SQLite control on the same rig 1,01 / 0,98 / 0,99 / 0,99. 12 of 16 cells strictly below 1,00, one at exactly 1,00; the control is what makes the 2-12 % READ/UPDATE deficits readable, and the only clear AOT lead (legacy DELETE 1,25) sits at n=2 processes with a 0,92-1,38 pair spread. So NativeAOT is a deployment option here (image size, startup, no JIT), not a speed lever. Fixed -- the same refuted expectation in four places: - README: the Native AOT readiness bullet now states what AOT is worth, not only that it works. - docs/manual/performance.md 7.10: the row listed "AOT interface-dispatch improvements -> faster interface-heavy storage paths under NativeAOT" among the v2.1 items; it now reads measured, no win. - docs/performance/V2_PERFORMANCE_PLAN.md 4.1: keeps the vendor expectation, marks it measured and not transferring on our workload. - docs/BENCHMARK_RESULTS.md: gains a NativeAOT arm section -- per-cell table, reproducible publish command, the two harness defects that nearly published the opposite verdict, and pointers to plan S4, worklog session 78 and the archived runs committed in 5dda8a20. docs/CHANGELOG.md [Unreleased] -> Fixed records the correction with the numbers. Validation: build 0 errors; DocumentationConsistencyTests 3/3; core suite 1998 / 0 failed / 0 errors / 0 skipped on the CI filter. Docs only -- no code, config, threshold, test or version moved. --- README.md | 5 +++- docs/BENCHMARK_RESULTS.md | 40 +++++++++++++++++++++++++ docs/CHANGELOG.md | 1 + docs/manual/performance.md | 2 +- docs/performance/V2_PERFORMANCE_PLAN.md | 2 +- 5 files changed, 47 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 7d94c93e..af2b84b8 100644 --- a/README.md +++ b/README.md @@ -221,7 +221,10 @@ Headline changes, grouped by area. Everything below is in the shipped `2.1.0` pa - ⚡ **Compiled regexes everywhere** on hot paths; regex-free `NormalizeSql`; cached DI - ⚡ **Removed hidden `D:\*.log` debug writes** that throttled every SELECT/execute/transaction/INSERT - 🛡️ **Native AOT readiness** — AOT-safe `TypeConverter`, `Option` reader, `[RequiresDynamicCode]` annotations, - source-generated DTOs/JSON (`tools/SharpCoreDB.AotSmoke` publishes + runs, exit 0) + source-generated DTOs/JSON (`tools/SharpCoreDB.AotSmoke` publishes + runs, exit 0). Measured for value as + well: the harness's **AOT ÷ JIT** A/B leaves native AOT **at or behind the JIT on 12 of 16 cells**, so AOT is + a deployment option (image size, startup, no JIT) and **not** a speed lever here — + [`docs/BENCHMARK_RESULTS.md`](docs/BENCHMARK_RESULTS.md) - 🛡️ **Reopen data-integrity hardening (2.0.0.2)** — empty-value overflow reload, single-file fixed-width arena markers, and legacy 1.x delete-after-update purge (`ReopenRoundTripMatrixTests`) - ✅ **Validated by the core suite on this line (1,998 tests, 0 failed, measured 2026-10-11)** — the diff --git a/docs/BENCHMARK_RESULTS.md b/docs/BENCHMARK_RESULTS.md index af54a586..9a3db33e 100644 --- a/docs/BENCHMARK_RESULTS.md +++ b/docs/BENCHMARK_RESULTS.md @@ -53,6 +53,46 @@ held one prepared command — 288.108 → 878.557 UPDATE, 385.116 → 1.172.704 > Full analysis + API ladder: [`docs/manual/performance.md`](manual/performance.md) · > comparative report: [`docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md`](benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md) +### NativeAOT arm (measured October 9, 2026) — publishes and runs, buys no throughput + +The AOT question is a real one for this engine, because every arm pays per-row `IStorageEngine` / +`IStorage` dispatch, so it was measured rather than assumed. The same harness was published with + +```bash +dotnet publish tests/benchmarks/SharpCoreDB.Benchmarks.Comparative -c Release -f net11.0 -r win-x64 -p:PublishAot=true +``` + +(native `win-x64` image, 14.023.680 bytes; run from the directory above `native\`, where `e_sqlite3.dll` +lives) and run **as that native binary** against its own JIT build — fresh processes, identical options, +same box, all 16 cells. Ratio is **AOT ÷ JIT**, median of the paired runs, in INSERT · READ · UPDATE · +DELETE order: + +| arm | AOT ÷ JIT | single-pair range | +|---|---|---| +| legacy plaintext (tuned `--pk`) | 1,00 · 0,95 · 0,88 · **1,25** | 0,82–1,38 | +| fixed-width plaintext | 0,96 · 0,98 · 0,93 · 1,08 | 0,81–1,34 | +| fixed-width at rest (encrypted) | 0,99 · 0,90 · 0,89 · 0,92 | 0,83–1,03 | +| **SQLite (control, same rig)** | 1,01 · 0,98 · 0,99 · 0,99 | 0,95–1,04 | + +**12 of the 16 cells sit strictly below 1,00 and one at exactly 1,00.** The control is what makes the +2–12 % READ/UPDATE deficits readable instead of box noise: AOT is marginally *behind* on SQLite's own +cells too, so the rig is not biased toward either runtime. The only cell where AOT leads by more than +that noise is the legacy arm's DELETE (1,25) — at n=2 AOT processes with a 0,92–1,38 pair spread, that is +not grounds for a recommendation. **So NativeAOT is a deployment option here (image size, startup, no +JIT) and not a speed lever**, and it needs the Visual Studio C++ workload on the build machine. + +Two harness defects surfaced during the measurement and are fixed in the harness (no `src/` change): +`--pk` sampled straight from process start, so arm 1's published reps *were* the JIT ramp — which made +AOT read **1,85× faster on UPDATE** — and the shared warm-up resolver now discards them; and the AOT +profile killed every run at the JSON archive write (`Reflection-based serialization has been disabled`), +pinned with `true` +in the harness project. Verdict, both defects and the raw per-rep runs: +[`docs/performance/BEAT_SQLITE_ALL_AXES_PLAN.md`](performance/BEAT_SQLITE_ALL_AXES_PLAN.md) §S4 (worklog +session 78); archived runs: +[`tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/`](../tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results) +(`pk_comparative_20261009_20*.json`, committed in `5dda8a20`). + + --- diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 4517ad64..2949712a 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -46,6 +46,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **The docs marketed a NativeAOT throughput win that measurement had already refuted, and the benchmark file carried no AOT arm at all.** Plan item S4 of the SQLite plan asked whether .NET 11's NativeAOT *shared dispatch helper* speeds up the per-row `IStorageEngine`/`IStorage` dispatch every cell pays for. Session 78 answered it by measurement: publishing the comparative harness with `-p:PublishAot=true` (native `win-x64` image, 14.023.680 B) and running it against its own JIT build — all 16 cells, fresh processes, identical options, same box — leaves AOT **at or behind the JIT on 12 of 16 cells** (one at exactly 1,00; the SQLite control on the same rig reads 0,98–1,01, which is what makes the 2–12 % READ/UPDATE deficits readable; the sole clear AOT lead is the legacy arm's DELETE at 1,25, at n=2 processes and a 0,92–1,38 pair spread), so S4 closed `REJECTED` with no AOT profile and no AOT recommendation. Three maintained documents still stated the un-measured expectation as a win — `docs/manual/performance.md` §7.10 listed *"AOT interface-dispatch improvements → faster interface-heavy storage paths under NativeAOT"* among the v2.1 items, `docs/performance/V2_PERFORMANCE_PLAN.md` §4.1 presented the helper as a speedup for `IStorage`/`IStorageEngine`/`ITable`, and the front page's AOT bullet read as a capability with no statement of what it is worth — and `docs/BENCHMARK_RESULTS.md`, the file users read for benchmark status, had no AOT number in it. All four now carry the measured result: the front page and the manual row state that AOT is a deployment option (image size, startup, no JIT) and not a speed lever here, `V2_PERFORMANCE_PLAN` §4.1 keeps the vendor expectation but marks it measured-and-not-transferring, and `BENCHMARK_RESULTS.md` gains a **NativeAOT arm** section with the per-cell AOT ÷ JIT table, the reproducible publish command, the two harness defects that nearly published the opposite verdict (the `--pk` warm-up ramp that made AOT look 1,85× on UPDATE, and the AOT profile failing on reflection-based JSON serialization, now pinned in the harness `.csproj`), and pointers to plan §S4, worklog session 78 and the archived runs. - **`SELECT version()` reported the pre-promotion product version *and* runtime to every client of the shipped 2.1.0 server.** The PostgreSQL handshake's `server_version` was a `const` literal, `SharpCoreDB 1.7.0 on .NET 10 (PostgreSQL protocol ...)`, so once 2.1.0 was published a client read a version two releases stale on a runtime the server no longer runs. It is now derived - the assembly's informational version plus `Environment.Version.Major` - so it tracks the build it ships in instead of being edited by hand, and it is pinned by `PgCatalogServiceTests.TryHandleCatalogQuery_Version_ReportsCurrentRuntimeAndProductVersion` (failed against the old literal with `Not found: "SharpCoreDB 2.1.0"`; passes after the change). Integration suite 19 / 0 failed. - **The 2.1.0 promotion left ".NET 10 / C# 14" as a *current requirement* in 70 files: front page (which also pinned the test count at 1,916 against 1,995 on this tree), `ROADMAP.md`, `CODING_STANDARDS.md`, around 12 shipped package READMEs (`NuGet.README.md` install snippets still pinning `2.1.0-rc.*`, the `Aspire` image tag `2.0.0.3` against `2.1.0`), around 14 `docs/**` requirement pages - including the load-bearing runtime requirement in `docs/server/INSTALLATION.md` - the publish-guide template, both `.runsettings` target frameworks, `PLATFORM_SUPPORT.md`, `SharpCoreDB.nuspec` (16 `net10.0` lib paths, tags, the `Microsoft.Data.Sqlite` floor), the `Examples/**` / `tests/**` guides and the decorative source labels - **589 `C# 14` labels across 260 `.cs` files and 26 stale `C# 14` claims in 11 `.md` files**, both measured in a follow-up sweep that verified the substitution line-for-line (in those 260 files every changed line *is* the label and nothing else, and the non-ASCII bytes on those lines are unchanged).** The promotion itself had refreshed the *build* metadata (props, packages, `.csproj`) and was validated by building, packing and publishing, so nothing in that route could see a claim that lives in prose. The sweep changed statements about the current line only: dated records (`STABILITY_REPORT.md`, the parked-line comment in `SharpCoreDB.csproj`, the `2.0.0.x` changelog sections, `ci-net10.yml`'s name), `#if NET10_0` regions and upstream package facts that merely mention a net10-era version are correct as written and were left alone - the boundary is recorded in `docs/net11/PROMOTION_TO_MAIN.md`, together with the note that the README text embedded in the already-published `2.1.0` packages only changes with the next version published. diff --git a/docs/manual/performance.md b/docs/manual/performance.md index a778d846..d4a27984 100644 --- a/docs/manual/performance.md +++ b/docs/manual/performance.md @@ -399,7 +399,7 @@ harness, and the two-run final ranges are recorded in §7.1. | **DELETE fast path — landed (v2.1)** | Deferred index maintenance (default on) cut random-key DELETE to **2.19×**/1.19× (plaintext/encrypted); the remaining gap is the tombstone write plus per-statement overhead | | **Single-statement SQL ladder** | The SQL layer still adds ~25 µs/row over the direct API on one-row statements; the remaining cost is spread across per-statement parsing and plan-cache work | | **.NET 11 / C# 15** | Runtime-native async, AVX-VNNI-512/SVE2, SIMD lane APIs, Zstandard, Decimal32/64/128 — free speedups in hot paths | -| **AOT interface-dispatch improvements** | Faster interface-heavy storage paths under NativeAOT | +| **AOT interface-dispatch improvements** — **measured 2026-10-09: no win** | The .NET 11 dispatch helper is *documented* to speed interface-heavy paths, but on this harness native AOT is **at or behind the JIT on 12 of 16 cells** (SQLite's own cells on the same rig read 0,98–1,01), so no AOT profile is recommended — publish AOT for deployment size and startup, not throughput. Per-cell table and both harness defects the measurement exposed: [`../BENCHMARK_RESULTS.md`](../BENCHMARK_RESULTS.md) §NativeAOT · [BEAT plan §S4](../performance/BEAT_SQLITE_ALL_AXES_PLAN.md) | Track progress in [`docs/performance/V2_PERFORMANCE_PLAN.md`](../performance/V2_PERFORMANCE_PLAN.md). diff --git a/docs/performance/V2_PERFORMANCE_PLAN.md b/docs/performance/V2_PERFORMANCE_PLAN.md index d23f9086..c3e31273 100644 --- a/docs/performance/V2_PERFORMANCE_PLAN.md +++ b/docs/performance/V2_PERFORMANCE_PLAN.md @@ -231,7 +231,7 @@ Full suite green: **1,644 tests, 0 failures** (16 skipped). ### 4.1 Runtime & JIT (automatic wins on `net11.0`) - **Runtime-native async (Runtime Async):** lower-overhead async, tail-merged suspension points, reduced code size, ExecutionContext-capture opt-out when no ambient state. Directly benefits `Execute*Async`, `InsertBatchAsync`, `ExecuteBatchSQLAsync`, and server paths. - **JIT:** bounds-check elimination, redundant checked-context removal, devirtualization, switch-expression folding, constant-folding of `SequenceEqual`, redundant branch elimination → free speedups in parser/materializer loops and index lookups. -- **NativeAOT:** faster interface dispatch (shared dispatch helper) → better throughput for interface-heavy code paths (`IStorage`, `IStorageEngine`, `ITable`). +- **NativeAOT:** faster interface dispatch (shared dispatch helper) → *documented* to give better throughput for interface-heavy code paths (`IStorage`, `IStorageEngine`, `ITable`). **Measured on our own workload 2026-10-09 and it does not transfer:** publishing the comparative harness with `-p:PublishAot=true` and running it against its own JIT build leaves AOT **at or behind the JIT on 12 of 16 cells**, so the expectation above is not a v2.1 win and no AOT profile ships. See [`BEAT_SQLITE_ALL_AXES_PLAN.md`](BEAT_SQLITE_ALL_AXES_PLAN.md) §S4 and `docs/BENCHMARK_RESULTS.md` §NativeAOT. ### 4.2 Hardware intrinsics - **AVX-VNNI-512** (x64) and **Arm SVE2** → vector search (HNSW) distance kernels and `SimdHelper` fallback/optimized paths. From 013fac2920f0a510b915b980528d2cfe7c54ae76 Mon Sep 17 00:00:00 2001 From: MPCoreDeveloper Date: Sun, 11 Oct 2026 17:17:21 +0200 Subject: [PATCH 4/6] docs(scdb): the SCDB pages now describe the shipped engine, and docs/ has no dangling links The production guide documented an engine that was never shipped: CHECKPOINT SQL, Database.BackupAsync, GetStatistics()/GetPerformanceMetrics(), EnableWalCheckpointing, WalCheckpointIntervalMs, WalDurabilityMode.Fsync, sidecar-*.wal incremental backups, and db.CreateUser/db.Login (which single-file mode rejects with NotSupportedException). Config samples passed DatabaseOptions where the Database constructor takes DatabaseConfig, and CorruptionDetector/RepairTool were handed a DI service provider instead of the SingleFileStorageProvider they require. Every option, method and type in docs/scdb/ was read out of src/; the guide is now split by type (DatabaseOptions vs DatabaseConfig), the WAL is described as a region inside the .scdb file (db.Flush(), no CHECKPOINT statement), monitoring uses GetStorageStatistics/GetQueryCacheStatistics/GetPageCacheStatistics/GetDatabaseStatistics, and the two unsourced ratio claims (7,765x, 1.37x) are gone in favour of pointers to the measured pages. docs/scdb/README.md's C# 15 samples no longer illustrate the unused ScdbFile prototype. 32 relative links across 11 files pointed at files that do not exist (SCDB_FILE_FORMAT_DESIGN.md, DATABASE_ARCHITECTURE.md, GraphTraversalPerformanceBenchmarks.cs, EFCORE_COLLATE_COMPLETE.md and others); every one now resolves, or names tests/benchmarks/ in text where no artefact is checked in. Red->green: DocumentationConsistencyTests.DocumentationMarkdownLinks_ShouldResolveToExistingFiles reported 32 dangling links before the sweep and 0 after, and its sensitivity was demonstrated by planting one link (__link_probe.md:3) and watching it fail on exactly that line. Docs suite 4/4; core suite 1999 / 0 failed / 0 errors / 0 skipped. Local commit on perf/autonomous-20261011, never pushed. --- ROADMAP.md | 5 + docs/CHANGELOG.md | 2 + docs/QUERY_PLAN_CACHE.md | 8 +- docs/analytics/README.md | 6 +- docs/benchmarks/BENCHMARK_METHOD.md | 5 +- .../ISSUE_56_ACCEPTANCE_CRITERIA.md | 12 +- docs/collation/COLLATION_GUIDE.md | 4 +- docs/features/README.md | 4 +- docs/graphrag/EF_CORE_COMPLETE_GUIDE.md | 4 +- docs/graphrag/LINQ_API_GUIDE.md | 6 +- docs/manual/query.md | 2 +- docs/migration/MIGRATION_GUIDE.md | 12 +- docs/migration/SQLITE_VECTORS_TO_SHARPCORE.md | 8 +- docs/performance/WORKLOG.md | 10 + docs/scdb/PRODUCTION_GUIDE.md | 361 +++++++++----- docs/scdb/README.md | 469 ++++++++---------- .../DocumentationConsistencyTests.cs | 51 ++ 17 files changed, 548 insertions(+), 421 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 9b128014..97959458 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -114,6 +114,11 @@ ## 🔶 In Progress +### Legacy `ScdbFile` prototype +- **Status:** public API in the shipped package, but unused and unfinished - `ReadBlock` returns an empty span, `WriteBlock` registers neither a block nor a WAL entry, `AllocatePages` only appends, and `Flush` writes nothing +- **What's needed:** decide its fate - finish it against `BlockRegistry` / `FreeSpaceManager` / `WalManager`, or deprecate and remove it in the next major (breaking change, so it is an owner call) +- **Why it is listed:** `docs/scdb/README.md` used to advertise it as the way to use `.scdb`; the docs now point at `SingleFileStorageProvider`, and the class must not go back to being invisible dead API + ### Visual Query Execution Plan Explorer - **Status:** Plan cache and optimizer internals exist; SCDMS UI not built yet - **What's needed:** Tree/graph view in SCDMS showing join types, cost estimates, and row counts per node — similar to pgAdmin's EXPLAIN visualizer diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 2949712a..acf1d9a5 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -66,6 +66,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **A block whose payload is legitimately empty (zero bytes) threw on both halves of a rewrite.** A single-file table whose values all fit inline has an *empty* overflow block, and `SingleFileStorageProvider.WriteBlockAsync` derived its page count from the payload length, so `FreeSpaceManager.AllocatePages(0)` threw `ArgumentOutOfRangeException` — and once that half was fixed, `FreePages(offset, 0)` threw the same way on the free half. Zero pages now skip the allocation (offset 0, and the defensive registry-area guard no longer applies to a block no reader dereferences, since every read honours the entry's length) and a zero-page free is a documented no-op, while a negative count still throws. This is the root cause behind the historical "the 16-byte default broke the single-file reopen matrix" note: that schema still overflowed at 16, so nothing had ever exercised an empty arena. The canary that proved it is `ReopenRoundTripMatrixTests.RoundTrip_ReopenWithEmptyValues_KeepsDataIntact(singlefile-fixedwidth)`, which failed until both halves were fixed; the full suite is **1924 / 0 failed / 16 skipped**. - **The server bootstrapped the tenant catalog and the database grants on throwaway repository instances, so the schema it created was not the schema it served.** `NetworkServer.InitializeServerAsync` built its own `TenantCatalogRepository` and `DatabaseGrantsRepository` bound to whatever `MasterDatabaseLocator` resolved, while the container handed the API, gRPC and binary-protocol endpoints *different* singletons — and with the grants repository a second `TenantSecurityAuditService`, hence a second audit store. On any configuration where the locator's pick differs from the database the endpoints serve, the tables were created in one database and read from another (`Tenants table not found in master database`) and tenant-security audit entries were split across two stores. Both repositories are now constructor-injected — the very instances the container serves — and `Program.cs` resolves the master database for both registrations through one local function, so a misconfiguration fails once with `MasterDatabaseLocator.DescribeUnresolved` instead of differing per copy. Red→green is pinned by `NetworkServerBootstrapTests` (2 facts; the container serves repositories bound to `testdb` while the locator resolves `master` for the same configuration): **2/2 failed** on the pre-change build, **2/2 pass** now, server integration suite **207 / 0 failed / 0 errors**, core suite **2049 / 0 failed / 0 errors / 16 skipped**. - **The SonarCloud quality gate on `master` failed on the promotion's own analysis (`58f297e9`, 2026-10-10 18:14) — `new_reliability_rating` 3 (C) and `new_security_rating` 4 (D) where A is required — and the cause was *4 MAJOR bugs and 13 vulnerabilities on new code*, not coverage.** The promotion made the whole 2.1 line "new code", so the gate was measuring the release; the two ratings were read from `qualitygates/project_status` and their causes from `issues/search?sinceLeakPeriod=true` (the platform's own API, not an inference), and all 17 are cleared in place. **Reliability:** the scalar double comparison in `SimdFilter.Matches` has to agree with the per-lane `Vector128/256/512.Equals` the vector bodies use, so its exact equality is annotated as deliberate — an epsilon there would make the scalar tail return a different match set than the SIMD body for the same input — and the medoid's `distance == bestDistance` tie-break in `DiskAnnIndex` is exact for the same reason, because it is what decides the selection; `Database.Load` now binds `config` with a pattern instead of testing the field and re-reading it; and the comparative harness's hand-rolled aggregation — which carried an emptiness test the analyzer could prove always true, since `reps` is 3 or a positive `SHARPCOREDB_BENCH_REPS` — now goes through the existing `MedianArm` helper, with the comparator's regime taken from a guarded `ArmRegime` helper used by all four call sites. **Security:** workflow-level `packages: write` is scoped to the publishing job in `ci.yml` and `ci-net10.yml` (S8233), the `codecov/codecov-action` step is pinned to a commit SHA rather than the moving `v5` tag (S7637), the SQLite reference pragmas and the fair arm's index DDL are validated and prebuilt instead of interpolated into SQL at the call site — the regime can come from outside the process, and `SHARPCOREDB_SQLITE_PRAGMAS` now rejects anything that is not a plain identifier/single token (S2077 x3) — the harness writes into a private, owner-only subdirectory of the temp path instead of the world-writable root (S5443; `SHARPCOREDB_BENCH_TEMP` still overrides it, and the run banner prints the resolved directory), the two deterministic RNGs in `DiskAnnIndex` are annotated rather than swapped for a cryptographic one that would destroy index reproducibility (S2245), `TableMetadataEntry.Reserved` is an `[InlineArray(22)]` region instead of an `unsafe fixed` buffer — byte-identical, and pinned by `ScdbStructureLayoutTests` (offsets 0x8E/0x92, a 22-byte region, a packed size of 0xA8 inside the 256-byte on-disk stride) (S6640) — and the benchmark-picking skill scripts draw from the platform CSPRNG while the category-hue hash moved from MD5 to SHA-256 (S2245/S4790, in the Python *and* the JS, which lives both in the asset and in `brain.py`'s HTML template). Verified: tests and benchmark projects build with **0 errors**, the layout guard is **3/3**, the core suite is **1998 / 0 failed / 0 errors / 0 skipped**, and the `// NOSONAR` mechanism the four deliberate cases rely on is proven on this project's own data — `Database.Batch.cs` reports `S3776` at lines 1050, 962, 1359, 1436 and 1749 and **not** at 710/1217, which are exactly its two NOSONAR-annotated lines. **The gate has since re-evaluated on the default branch: the fix was pushed as `b7bd563e` (2026-10-11 07:01:39Z), the GitHub App analysed it at 07:05:54Z, and the project's gate now reads `OK` on all five conditions — both ratings back to `1 (A)` — with the leak period's unresolved issues down 278 → 261, exactly the 17 (0 BUG, 0 VULNERABILITY, and `S8233`/`S7637`/`S2077`/`S5443`/`S2245`/`S6640`/`S1244` at 0 apiece, which is also the production proof that the four NOSONAR annotations held).** Coverage is not one of the gate's inputs because analysis here is automatic (GitHub App). +- **The `docs/scdb/` pages described an engine that was never shipped — every sample in the production guide named an option or a method that does not exist in the v2.1 code.** The configuration blocks passed `EnableWalCheckpointing`, `WalCheckpointIntervalMs`, `WalDurabilityMode.Fsync`, `EnablePerformanceMetrics`, `EnablePageCache` and `PageCacheCapacity` through `DatabaseOptions`; the real surface is split — `DatabaseOptions` (`StorageMode`, `PageSize`, `FsmSizePages`, `WalBufferSizePages`, `AutoVacuum`, `AutoVacuumMode`, `FragmentationThreshold`, `FileShareMode`, `UseUnbufferedIO`, `EnableMemoryMapping`) and `DatabaseConfig` (`WalDurabilityMode`, `WalBufferSize`, `EnableAdaptiveWalBatching`, `EnablePageCache`, `PageCacheCapacity`, `EnableQueryCache`, `QueryCacheSize`, `BufferPoolSize`, `WorkloadHint`, …) — `DurabilityMode` has exactly `FullSync` and `Async` (no `Fsync`), and the `Database` constructor takes a `DatabaseConfig`, so a `DatabaseOptions` argument does not compile. The backup section ran `db.ExecuteSQL("CHECKPOINT")` and `await db.BackupAsync(...)`: neither exists (there is no `CHECKPOINT` statement, and the WAL is a region inside the `.scdb` file flushed with `db.Flush()`), and its "incremental backup" copied sidecar `*.wal` files the format never writes. The monitoring and maintenance sections called `db.GetStatistics()` and `db.GetPerformanceMetrics()`; the real surface is `GetStorageStatistics()` (`TotalSize`, `UsedSpace`, `FreeSpace`, `FragmentationPercent`, `BlockCount`, `DirtyBlocks`, `PageCount`, `FreePages`, `WalSize`, `LastVacuum`), `GetQueryCacheStatistics()`, `GetPageCacheStatistics()` and `GetDatabaseStatistics()`, with per-stage write timings in `Diagnostics/WritePathProfiler.cs`. The security section hung access control on `db.CreateUser`/`db.Login`, which `DatabaseExtensions` rejects in single-file mode with `NotSupportedException` ("User management is not supported in single-file mode"), and the disaster-recovery samples handed `CorruptionDetector`/`RepairTool` the DI service provider where they require the `SingleFileStorageProvider` (the `SingleFileStorageProvider.Open` path, `RepairOptions`/`RepairAggressiveness`/`VacuumResult` names were kept because they are real). Two ratio claims with no run behind them ("7,765x faster analytics", "1.37x faster INSERTs") were deleted in favour of pointers to the measured pages, and the guide now states where each number comes from. +- **32 relative links in `docs/` pointed at files that do not exist, and two benchmark pages carried placeholder stubs whose link target was the literal word `link`.** `docs/migration/MIGRATION_GUIDE.md` pointed six times at `SCDB_FILE_FORMAT_DESIGN.md`/`SCDB_IMPLEMENTATION_STATUS.md`/`SCDB_PHASE1_IMPLEMENTATION.md` (the SCDB docs live in `docs/scdb/`), `docs/QUERY_PLAN_CACHE.md` at `DATABASE_ARCHITECTURE.md`/`CONFIGURATION.md`/`BEST_PRACTICES.md`, and `docs/analytics`, `docs/collation`, `docs/features`, `docs/graphrag`, `docs/manual`, `docs/migration` and `docs/benchmarks` at the rest (a 33rd, `docs/scdb/README.md` → `FILE_FORMAT_DESIGN.md`, was redirected earlier in the same session); the `REPORT_V1.md`/`REPORT_V2.md`/PR/`results` links in the issue-#56 closure template were unresolvable stubs. All of them now resolve to real files, or name `tests/benchmarks/` in text where no artefact is checked in. Red→green: `DocumentationConsistencyTests.DocumentationMarkdownLinks_ShouldResolveToExistingFiles` — the same algorithm reported **32 dangling links across 11 files** before the sweep and **0** after, and the guard's sensitivity was demonstrated in-session by planting one link (`__link_probe.md:3 -> ./does-not-exist.md`), watching it fail on exactly that line, and removing the probe. ### Performance diff --git a/docs/QUERY_PLAN_CACHE.md b/docs/QUERY_PLAN_CACHE.md index 08dca9cf..15312242 100644 --- a/docs/QUERY_PLAN_CACHE.md +++ b/docs/QUERY_PLAN_CACHE.md @@ -513,10 +513,10 @@ Query plan caching provides: ## Further Reading -- [Database Execution Layer](./DATABASE_ARCHITECTURE.md) -- [Performance Benchmarks](../README.md#performance-benchmarks) -- [Configuration Guide](./CONFIGURATION.md) -- [Best Practices](./BEST_PRACTICES.md) +- [Query Optimizer Architecture](./internals/OPTIMIZER_ARCHITECTURE.md) +- [Comparative Benchmarks](./benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md) +- [Optimizer Configuration & Tuning](./internals/OPTIMIZER_ARCHITECTURE.md#configuration--tuning) +- [Performance Guide: configuration knobs](./manual/performance.md#77-configuration-knobs-that-affect-performance) --- diff --git a/docs/analytics/README.md b/docs/analytics/README.md index a946e49e..b9cfb699 100644 --- a/docs/analytics/README.md +++ b/docs/analytics/README.md @@ -665,9 +665,9 @@ var gaps = await database.QueryAsync( ## See Also - [User Manual](../manual/README.md) - Complete feature guide -- [Vector Search](../vectors/README.md) - Embedding storage -- [Graph Algorithms](../graph/README.md) - Path finding -- [Performance Guide](../PERFORMANCE.md) - Optimization techniques +- [Vector Search](../Vectors/README.md) - Embedding storage +- [GraphRAG Documentation](../graphrag/README.md) - Graph traversal, path finding +- [Performance Guide](../manual/performance.md) - Optimization techniques --- diff --git a/docs/benchmarks/BENCHMARK_METHOD.md b/docs/benchmarks/BENCHMARK_METHOD.md index 8f43ff53..c9c6ba5a 100644 --- a/docs/benchmarks/BENCHMARK_METHOD.md +++ b/docs/benchmarks/BENCHMARK_METHOD.md @@ -509,8 +509,9 @@ This report follows [METHODOLOGY.md]. Results are reproducible. See [REPRODUCING --- -**Full raw data:** [results/2026-03-03-run-001/raw-data.json](link) -**Scripts:** [tests/benchmarks/](link) +**Full raw data:** `tests/benchmarks/` (one directory per run; no raw-data file is checked in for the +2026-03-03 run, so this line names the location rather than linking a missing artefact) +**Scripts:** `tests/benchmarks/README.md` and `tests/benchmarks/run-all.ps1` ``` ### 8.2 Raw Data Format (JSON) diff --git a/docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md b/docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md index e60f5260..4b619055 100644 --- a/docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md +++ b/docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md @@ -184,9 +184,9 @@ Issue #56 is **done** when: **Benchmark Suite: SharpCoreDB vs BLite vs Zvec** - Scenarios: 9 total (4 BLite, 5 Zvec) -- Raw data: Available in `tests/benchmarks/results/` -- Report v1: [REPORT_V1.md](link) -- Report v2: [REPORT_V2.md](link) +- Raw data: Available in `tests/benchmarks/` +- Report v1: `REPORT_V1.md` (attach the published link when closing the issue) +- Report v2: `REPORT_V2.md` (attach the published link when closing the issue) - Reproducibility: Scripts available, runnable **Key Findings:** @@ -195,9 +195,9 @@ Issue #56 is **done** when: - SharpCoreDB: [summary] **Links:** -- PR: [#XXX](link) -- Documentation: [README](link) -- Raw data: [results/](link) +- PR: `#XXX` (fill in the pull-request number) +- Documentation: [benchmarks README](../../tests/benchmarks/README.md) +- Raw data: `tests/benchmarks/` Benchmark suite is now public and reproducible. Community can validate. ``` diff --git a/docs/collation/COLLATION_GUIDE.md b/docs/collation/COLLATION_GUIDE.md index b0ffb737..73e42506 100644 --- a/docs/collation/COLLATION_GUIDE.md +++ b/docs/collation/COLLATION_GUIDE.md @@ -661,8 +661,8 @@ SELECT 'Café' = 'cafe' COLLATE UNICODE; -- TRUE (full normalization) ## See Also -- [EF Core Integration](../EFCORE_COLLATE_COMPLETE.md) -- [Collation Performance Tests](../benchmarks/Phase5_CollationQueryPerformanceBenchmark.cs) +- [EF Core Integration](../../src/SharpCoreDB.EntityFrameworkCore/README.md) +- [Collation Performance Benchmark](../../tests/SharpCoreDB.Benchmarks/Phase5_CollationQueryPerformanceBenchmark.cs) --- diff --git a/docs/features/README.md b/docs/features/README.md index ecf2f3d0..459b4490 100644 --- a/docs/features/README.md +++ b/docs/features/README.md @@ -62,7 +62,7 @@ See [Collation Guide](../collation/COLLATION_GUIDE.md) for complete documentatio - [API Reference](../Vectors/README.md#api-reference) - [Configuration](../Vectors/README.md#configuration) - [Examples](../Vectors/README.md#examples) -- [Implementation Complete Report](../Vectors/IMPLEMENTATION_COMPLETE.md) +- [Vector Search Technical Spec](../Vectors/TECHNICAL_SPEC.md) ### [Migration Guide: SQLite Vectors → SharpCoreDB](../migration/SQLITE_VECTORS_TO_SHARPCORE.md) **Status:** ✅ **Production Ready** @@ -193,7 +193,7 @@ See [Collation Guide](../collation/COLLATION_GUIDE.md) for complete documentatio ## Need Help? -- **Questions?** Check [FAQ](../FAQ.md) or open a GitHub issue +- **Questions?** Check the [documentation index](../INDEX.md) or open a GitHub issue - **Performance issues?** See [Tuning Guide](../Vectors/PERFORMANCE_TUNING.md) - **Troubleshooting?** See [Vector Troubleshooting](../Vectors/README.md#troubleshooting) - **API reference?** See [Vector API Docs](../Vectors/README.md#api-reference) diff --git a/docs/graphrag/EF_CORE_COMPLETE_GUIDE.md b/docs/graphrag/EF_CORE_COMPLETE_GUIDE.md index d831b40f..3ee7a7b8 100644 --- a/docs/graphrag/EF_CORE_COMPLETE_GUIDE.md +++ b/docs/graphrag/EF_CORE_COMPLETE_GUIDE.md @@ -688,5 +688,5 @@ The EF Core GraphRAG integration provides: For more information, see: - [LINQ_API_GUIDE.md](./LINQ_API_GUIDE.md) - API reference -- [EF_CORE_INTEGRATION_SUMMARY.md](./EF_CORE_INTEGRATION_SUMMARY.md) - Architecture overview -- [GRAPHRAG_IMPLEMENTATION_PLAN.md](./GRAPHRAG_IMPLEMENTATION_PLAN.md) - Implementation details +- [EF_CORE_DOCUMENTATION_INDEX.md](./EF_CORE_DOCUMENTATION_INDEX.md) - Index of the EF Core documentation set +- [GRAPH_RAG_SINGLE_SQL.md](./GRAPH_RAG_SINGLE_SQL.md) - single-query Graph RAG implementation diff --git a/docs/graphrag/LINQ_API_GUIDE.md b/docs/graphrag/LINQ_API_GUIDE.md index cc60e53f..86c53f1e 100644 --- a/docs/graphrag/LINQ_API_GUIDE.md +++ b/docs/graphrag/LINQ_API_GUIDE.md @@ -452,6 +452,6 @@ var relatedConcepts = await db.Concepts ## See Also -- [SharpCoreDB EF Core Provider](../SharpCoreDB.EntityFrameworkCore/README.md) -- [Graph Traversal Engine](../../SharpCoreDB.Graph/README.md) -- [Performance Benchmarks](./GraphTraversalPerformanceBenchmarks.cs) +- [SharpCoreDB EF Core Provider](../../src/SharpCoreDB.EntityFrameworkCore/README.md) +- [Graph Traversal Engine](../../src/SharpCoreDB.Graph/README.md) +- [Graph Traversal Tests](../../tests/SharpCoreDB.Tests/Graph/GraphTraversalEngineTests.cs) diff --git a/docs/manual/query.md b/docs/manual/query.md index 4287bc9d..8a7b2ebf 100644 --- a/docs/manual/query.md +++ b/docs/manual/query.md @@ -102,7 +102,7 @@ SharpCoreDB adds engine-specific keywords: `SELECT * FROM t WHERE key = @p` plans are resolved from the cache without re-lexing. > ⚡ **Guidance:** parameterize everything, reuse `IDatabase` instances, and keep working-set -> statements under the cache size. See [`docs/QUERY_PLAN_CACHE.md`](../docs/QUERY_PLAN_CACHE.md). +> statements under the cache size. See [`QUERY_PLAN_CACHE.md`](../QUERY_PLAN_CACHE.md). ## 6.7 `information_schema` metadata views diff --git a/docs/migration/MIGRATION_GUIDE.md b/docs/migration/MIGRATION_GUIDE.md index c1a51d93..5db3ff7c 100644 --- a/docs/migration/MIGRATION_GUIDE.md +++ b/docs/migration/MIGRATION_GUIDE.md @@ -557,9 +557,9 @@ if (validation.IsValid) ## 📖 See Also -- [SCDB File Format Design](./SCDB_FILE_FORMAT_DESIGN.md) -- [SCDB Implementation Status](./SCDB_IMPLEMENTATION_STATUS.md) -- [SCDB Phase 1 Implementation](./SCDB_PHASE1_IMPLEMENTATION.md) +- [SCDB File Format Design](../scdb/README.md) +- [SCDB Implementation Status](../scdb/README_INDEX.md) +- [SCDB Format: Implementation Status](../scdb/README.md#-implementation-status) --- @@ -1050,9 +1050,9 @@ if (validation.IsValid) ## 📖 See Also -- [SCDB File Format Design](./SCDB_FILE_FORMAT_DESIGN.md) -- [SCDB Implementation Status](./SCDB_IMPLEMENTATION_STATUS.md) -- [SCDB Phase 1 Implementation](./SCDB_PHASE1_IMPLEMENTATION.md) +- [SCDB File Format Design](../scdb/README.md) +- [SCDB Implementation Status](../scdb/README_INDEX.md) +- [Serialization and Storage Guide](../serialization/SERIALIZATION_AND_STORAGE_GUIDE.md) --- diff --git a/docs/migration/SQLITE_VECTORS_TO_SHARPCORE.md b/docs/migration/SQLITE_VECTORS_TO_SHARPCORE.md index 87fe7a6e..ae460f40 100644 --- a/docs/migration/SQLITE_VECTORS_TO_SHARPCORE.md +++ b/docs/migration/SQLITE_VECTORS_TO_SHARPCORE.md @@ -612,10 +612,10 @@ CREATE INDEX idx_embedding_flat ON documents(embedding) USING FLAT; ## Additional Resources -- [SharpCoreDB Vector Search Docs](./docs/VECTOR_SEARCH.md) -- [HNSW Algorithm Details](./docs/HNSW_ALGORITHM.md) -- [Performance Tuning Guide](./docs/PERFORMANCE_TUNING.md) -- [Examples: Vector Search](../Examples/VectorSearch/) +- [SharpCoreDB Vector Search Docs](../Vectors/README.md) +- [Vector Index and DiskANN Details](../Vectors/TECHNICAL_SPEC.md) +- [Performance Tuning Guide](../Vectors/PERFORMANCE_TUNING.md) +- [Example: GraphRAG with DiskANN](../Vectors/GRAPHRAG_WITH_DISKANN_EXAMPLE.md) --- diff --git a/docs/performance/WORKLOG.md b/docs/performance/WORKLOG.md index 66868538..67efc920 100644 --- a/docs/performance/WORKLOG.md +++ b/docs/performance/WORKLOG.md @@ -5100,3 +5100,13 @@ Nothing in the repository explains a stall: no `.git/*.lock`, no fsmonitor daemo **5. What was not touched.** No engine code, no config default, no threshold, no package or version metadata; the harness edits are confined to `BuildConfig`'s parameterisation, one `[diag]` string and the `Reps` stamp, and no other mode's `DatabaseConfig` moves (`legacyVariableLength` defaults to `false`, so every existing caller keeps the config it had). Nothing was weakened to publish a number: the encrypted posture is the *default* column in the README table, no arm's `NoEncryptMode` or durability was changed to win a cell, and the UPDATE/DELETE direction was reversed in the pages because the measurement says so, not because a target was met. No baseline was re-recorded (that needs the owner, §9 row 7). The comparison archives under `tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/` are untracked by convention and stay that way, except the four the published pages cite, which are committed with them so the numbers can be reproduced from the archive the plan asks for. + +### 2026-10-11 (session 82) — documentation correctness: the `docs/scdb/` pages were rewritten against the shipped v2.1 engine (the production guide had documented `CHECKPOINT` SQL, `Database.BackupAsync`, `GetPerformanceMetrics()` and its options on the wrong type), all 32 dangling relative links in `docs/` now resolve, and the sweep is pinned by a new core-suite test proven red→green + +- Session: 1 of 1 for this item — a documentation-correctness item rather than a plan §5 item, taken because the pages named an API that does not exist (the "a defect you find is yours to fix" duty applies to documentation that would send a user to a non-existent method); no §5 item was started, so the plan's queue is unchanged. +- Command(s): the link sweep (regex over `docs/**/*.md`, `Test-Path`/`File.Exists` against the resolved path) — **32 before, 0 after** · the API-surface reads: `Select-String` over `src/SharpCoreDB/DatabaseOptions.cs`, `DatabaseConfig.cs`, `Interfaces/IDatabase.cs`, `Storage/IStorageProvider.cs`, `Storage/Scdb/ScdbStructures.cs`, `Storage/Scdb/{CorruptionDetector,RepairTool,ScdbMigrator}.cs`, `Storage/SingleFileStorageProvider.cs`, `Storage/WalManager.cs` · `dotnet build tests/SharpCoreDB.Tests/SharpCoreDB.Tests.csproj -c Release -f net11.0` → 0 errors · the doc tests `tests/SharpCoreDB.Tests/bin/Release/net11.0/SharpCoreDB.Tests.exe -filterVSTest 'FullyQualifiedName~DocumentationConsistencyTests'` → **4 / 0 failed** · the guard's red proof: a planted `docs/__link_probe.md` → **1 failed, `__link_probe.md:3 -> ./does-not-exist.md`**, then the probe removed → 4 / 0 failed · the core suite `-filterVSTest 'Category!=Debug&Category!=Manual&Category!=Performance'` → **1999 / 0 failed / 0 errors / 0 skipped in 95,0 s**. +- Regime: docs-only plus one test file — no engine, configuration, threshold, package or version metadata touched. Release/JIT, `net11.0` / C# 15 preview, branch `perf/autonomous-20261011`. No benchmark was run and no number was produced: the guide's two unsourced ratio claims (7,765×, 1,37×) were **deleted** and replaced with pointers to the published pages, because re-measuring them was not this session's subject and inventing a figure would repeat the defect. +- Verdict: **KEPT.** Every option, method, type and path in `docs/scdb/PRODUCTION_GUIDE.md` and `docs/scdb/README.md` now names something that exists under `src/` — read from the source, not inferred — with the guide split by type (`DatabaseOptions` for storage/maintenance/file behaviour, `DatabaseConfig` for durability/caches/write path), the design-era fictions removed (`CHECKPOINT` SQL, `BackupAsync`, `GetStatistics()`, `GetPerformanceMetrics()`, `EnableWalCheckpointing`, `WalCheckpointIntervalMs`, `WalDurabilityMode.Fsync`, sidecar-`*.wal` incremental backup, single-file `CreateUser`/`Login`), and the residual numbers quoted only from `BENCHMARK_RESULTS.md`. Link integrity is now enforced by `DocumentationConsistencyTests.DocumentationMarkdownLinks_ShouldResolveToExistingFiles`, whose red state was demonstrated in the session rather than asserted. +- Commit: local commit on the feature branch `perf/autonomous-20261011` — **never on `master`**, never pushed (brief §9). +- NEXT: resume the plan's item list from session 81's `NEXT:` line (§5.1 INSERT, its item (i): the identity-remap fast path for `InsertBatch(table, rows, columns)` whose column order already is the table order — 112 B/row on the array arm). This session added no new perf item, so the queue's head is unchanged. + diff --git a/docs/scdb/PRODUCTION_GUIDE.md b/docs/scdb/PRODUCTION_GUIDE.md index 8cdc51dd..8a555a4d 100644 --- a/docs/scdb/PRODUCTION_GUIDE.md +++ b/docs/scdb/PRODUCTION_GUIDE.md @@ -1,22 +1,29 @@ # SharpCoreDB SCDB - Production Deployment Guide -**Version:** 2.0 -**Last Updated:** 2026-01-28 +**Version:** 2.1 +**Last Updated:** 2026-10-11 **Target:** Production environments --- +> Every option, method and number in this guide was checked against the shipped source. Quoted figures +> come from measured runs - see [`../BENCHMARK_RESULTS.md`](../BENCHMARK_RESULTS.md) and +> [`../performance/WORKLOG.md`](../performance/WORKLOG.md) - never from an estimate. The option surface +> is [`DatabaseOptions`](../../src/SharpCoreDB/DatabaseOptions.cs) and +> [`DatabaseConfig`](../../src/SharpCoreDB/DatabaseConfig.cs); if a setting is not there, it does not exist. + ## 🎯 Overview This guide covers deploying SharpCoreDB with the **SCDB (SharpCore DataBase)** single-file storage format in production environments. **SCDB Benefits:** -- ✅ Single `.scdb` file (easy backup/restore) -- ✅ WAL for ACID transactions -- ✅ Crash recovery -- ✅ Corruption detection & repair -- ✅ 7,765x faster analytics (SIMD) -- ✅ 1.37x faster INSERTs vs SQLite +- ✅ Single `.scdb` file (easy backup/restore) - one file, no sidecar WAL to keep in step +- ✅ WAL for ACID transactions - the WAL lives inside the `.scdb` file +- ✅ Crash recovery - replay on open, pinned by `CrashRecoveryTests` +- ✅ Corruption detection & repair - `CorruptionDetector` / `RepairTool` +- ✅ In-place writes without a per-row index rewrite +- ✅ Ratios against SQLite are measured per release and published with their method - see + [`../BENCHMARK_RESULTS.md`](../BENCHMARK_RESULTS.md); only the `--pk` arm is like-for-like --- @@ -35,13 +42,16 @@ This guide covers deploying SharpCoreDB with the **SCDB (SharpCore DataBase)** s ### Software Dependencies ```bash -# .NET 11 Runtime (required) +# .NET 11 Runtime (required by the 2.1 line on master) dotnet --version # Should show 11.0 or higher # NuGet Package dotnet add package SharpCoreDB --version 2.1.0 ``` +The 2.0.x servicing line on `release/v2.0` targets .NET 10 instead; match the runtime to the package +line you deploy. + --- ## 🚀 Quick Start @@ -50,7 +60,6 @@ dotnet add package SharpCoreDB --version 2.1.0 ```csharp using SharpCoreDB; -using SharpCoreDB.Storage; using Microsoft.Extensions.DependencyInjection; // Configure services @@ -58,22 +67,18 @@ var services = new ServiceCollection(); services.AddSingleton(); var provider = services.BuildServiceProvider(); -// Create SCDB database -var options = new DatabaseOptions -{ - StorageMode = StorageMode.SingleFile, // ✅ Use SCDB format - PageSize = 4096, // 4KB pages (default) - CreateImmediately = true, - EnableWalCheckpointing = true, // ✅ Auto-checkpoint - WalCheckpointIntervalMs = 30000, // Checkpoint every 30s -}; +// The .scdb extension selects single-file storage (DatabaseExtensions.DetectStorageMode), so the +// path is the only switch you need. The master password drives AES-256-GCM encryption. +// CreateSingleFileDefault() sets StorageMode, memory mapping and quick auto-VACUUM; DatabaseConfig +// is optional and stays null here, so the engine's own defaults apply. +var options = DatabaseOptions.CreateSingleFileDefault(); using var db = new Database( provider, - "./data/myapp.scdb", // ✅ Single file! + "./data/myapp.scdb", // ✅ Single file "your-master-password", isReadOnly: false, - config: options + config: options.DatabaseConfig ); // Use database @@ -81,6 +86,10 @@ db.ExecuteSQL("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)"); db.ExecuteSQL("INSERT INTO users VALUES (1, 'Alice')"); ``` +The same constructor serves both layouts: a path ending in `.scdb` (or an existing file) opens through +`SingleFileStorageProvider`, a directory path opens through `DirectoryStorageProvider`. See +[`README.md`](./README.md) for the file layout. + --- ## ⚙️ Configuration @@ -88,62 +97,88 @@ db.ExecuteSQL("INSERT INTO users VALUES (1, 'Alice')"); ### Production-Ready Configuration ```csharp -var config = new DatabaseOptions +// Storage-level options live on DatabaseOptions ... +var options = new DatabaseOptions { - // ========== SCDB Storage ========== + // ========== SCDB storage ========== StorageMode = StorageMode.SingleFile, PageSize = 4096, // 4KB pages (optimal for most workloads) CreateImmediately = true, - - // ========== WAL Configuration ========== - EnableWalCheckpointing = true, // ✅ CRITICAL: Enable auto-checkpoint - WalCheckpointIntervalMs = 30000, // Checkpoint every 30 seconds - WalMaxBatchSize = 100, // Batch up to 100 operations - WalMaxBatchDelayMs = 10, // Max 10ms batch delay - WalDurabilityMode = WalDurabilityMode.Fsync, // ✅ Maximum durability - - // ========== Performance ========== + FsmSizePages = 4, // free-space map pages + BlockRegistrySizePages = 4, + TableDirectorySizePages = 4, + WalBufferSizePages = 1024, // in-file WAL buffer, in pages + + // ========== Maintenance ========== + AutoVacuum = true, // VACUUM when fragmentation crosses the threshold + AutoVacuumMode = VacuumMode.Quick, + FragmentationThreshold = 25, // percent + + // ========== File behaviour ========== + FileShareMode = FileShare.ReadWrite, // one Database instance per file, per process + UseUnbufferedIO = false, // true bypasses the OS page cache + EnableMemoryMapping = true, +}; + +// ... and the durability, cache and write-path knobs live on DatabaseConfig. +var config = new DatabaseConfig +{ + // ========== Durability ========== + WalDurabilityMode = DurabilityMode.FullSync, // fsync on commit (the default) + WalBufferSize = 4 * 1024 * 1024, + EnableAdaptiveWalBatching = true, + + // ========== Caches ========== + EnablePageCache = true, + PageCacheCapacity = 1024, // pages (4 MB at 4 KB) EnableQueryCache = true, QueryCacheSize = 1000, - EnablePageCache = true, - PageCacheCapacity = 1024, // 4 MB cache (1024 * 4KB) - - // ========== Monitoring ========== - EnablePerformanceMetrics = true, + BufferPoolSize = 32 * 1024 * 1024, + + // ========== Write path ========== + UseGroupCommitWal = false, + EnableDeltaUpdates = false, + WorkloadHint = WorkloadHint.General, }; + +using var db = new Database(provider, "./data/myapp.scdb", "your-master-password", config: config); ``` ### Configuration by Workload #### High-Write Workload (OLTP) ```csharp -var config = new DatabaseOptions +var config = new DatabaseConfig { - PageSize = 4096, // Smaller pages for updates - WalMaxBatchSize = 50, // Smaller batches for lower latency - WalCheckpointIntervalMs = 60000, // Checkpoint every minute + EnableDeltaUpdates = true, // in-place deltas instead of row rewrites + UseGroupCommitWal = true, // batch commits + GroupCommitSize = 1000, + EnableAdaptiveWalBatching = true, PageCacheCapacity = 2048, // 8 MB cache + WorkloadHint = WorkloadHint.WriteHeavy, }; ``` #### High-Read Workload (Analytics) ```csharp -var config = new DatabaseOptions +var config = new DatabaseConfig { - PageSize = 8192, // Larger pages for scans - EnableQueryCache = true, - QueryCacheSize = 5000, // Large query cache PageCacheCapacity = 4096, // 32 MB cache + EnableQueryCache = true, + QueryCacheSize = 5000, // large query cache + UseMemoryMapping = true, + WorkloadHint = WorkloadHint.Analytics, }; ``` #### Balanced Workload ```csharp -var config = new DatabaseOptions +var config = new DatabaseConfig { - PageSize = 4096, // Standard pages - WalCheckpointIntervalMs = 30000, // 30s checkpoint PageCacheCapacity = 1024, // 4 MB cache + EnableAdaptiveWalBatching = true, + WalDurabilityMode = DurabilityMode.FullSync, + WorkloadHint = WorkloadHint.General, }; ``` @@ -155,25 +190,29 @@ var config = new DatabaseOptions #### Daily Backups ```csharp -// Simple file copy (database must be checkpointed first) -db.ExecuteSQL("CHECKPOINT"); // Flush WAL +// There is no separate WAL file to copy - the .scdb file is the whole database. Flush so the in-file +// WAL is applied, then copy the file. +db.Flush(); File.Copy("./data/myapp.scdb", $"./backups/myapp_{DateTime.UtcNow:yyyyMMdd}.scdb"); ``` -#### Hot Backup (No Downtime) +#### Consistent copy while the database is open ```csharp -// Use SCDB's built-in backup -await db.BackupAsync("./backups/myapp_hot.scdb"); +// Flush(), then copy. The provider has no BackupAsync() - a page that shows one is out of date. +// Reopening the copy is the only real verification that the backup is good: +db.Flush(); +File.Copy("./data/myapp.scdb", "./backups/myapp_hot.scdb", overwrite: true); + +using var restored = new Database(provider, "./backups/myapp_hot.scdb", "your-master-password"); +restored.ExecuteSQL("SELECT COUNT(*) FROM users"); ``` #### Incremental Backups ```csharp -// Copy WAL files separately -var walFiles = Directory.GetFiles("./data", "*.wal"); -foreach (var wal in walFiles) -{ - File.Copy(wal, Path.Combine("./wal_backups", Path.GetFileName(wal))); -} +// The WAL lives inside the .scdb file, so there is no sidecar *.wal to ship separately. Keep a +// rotation of full copies (the file is one unit) and shrink it with VACUUM: +var result = await db.VacuumAsync(VacuumMode.Incremental); +Console.WriteLine($"{result.FileSizeBefore:N0} -> {result.FileSizeAfter:N0} bytes, reclaimed {result.BytesReclaimed:N0}"); ``` ### Backup Retention Policy @@ -187,18 +226,20 @@ Monthly backups: Keep 12 months ### Disaster Recovery ```csharp +// `storage` is the SingleFileStorageProvider the database was opened with - CorruptionDetector and +// RepairTool take that provider, so corruption tooling applies to .scdb databases. // 1. Detect corruption -using var detector = new CorruptionDetector(provider, ValidationMode.Standard); +using var detector = new CorruptionDetector(storage, ValidationMode.Standard); var report = await detector.ValidateAsync(); if (report.IsCorrupted) { // 2. Attempt repair - using var repairTool = new RepairTool(report, provider); + using var repairTool = new RepairTool(report, storage); var result = await repairTool.RepairAsync(new RepairOptions { - CreateBackup = true, // ✅ Always backup first - AllowDataLoss = false, // Conservative repair + CreateBackup = true, // ✅ Always backup first (result.BackupPath records where) + AllowDataLoss = false, // conservative repair Aggressiveness = RepairAggressiveness.Conservative, }); @@ -210,6 +251,9 @@ if (report.IsCorrupted) } ``` +`RepairResult` reports `IssuesRepaired`, `IssuesRemaining`, `BackupPath`, `RepairTime` and `ErrorMessage`; +a conservative repair refuses issues the report marked unrepairable rather than guessing. + --- ## 📊 Monitoring @@ -219,24 +263,25 @@ if (report.IsCorrupted) ```csharp public class DatabaseHealthCheck : IHealthCheck { - private readonly Database _db; - + private readonly SingleFileStorageProvider _storage; + + public DatabaseHealthCheck(SingleFileStorageProvider storage) => _storage = storage; + public async Task CheckHealthAsync( HealthCheckContext context, CancellationToken cancellationToken = default) { try { - // Quick validation (header only, <1ms) - using var provider = GetStorageProvider(); - using var detector = new CorruptionDetector(provider, ValidationMode.Quick); + // Quick validation (header + WAL, no block scans) + using var detector = new CorruptionDetector(_storage, ValidationMode.Quick); var report = await detector.ValidateAsync(cancellationToken); - + if (report.IsCorrupted) { return HealthCheckResult.Unhealthy($"Corruption detected: {report.Summary}"); } - + return HealthCheckResult.Healthy("Database is healthy"); } catch (Exception ex) @@ -247,38 +292,63 @@ public class DatabaseHealthCheck : IHealthCheck } ``` +`CorruptionDetector` takes the **single-file provider** (`SingleFileStorageProvider`), not a DI service +provider - keep the instance the database was opened with and reuse it here. Validation modes: `Quick` +(header and WAL), `Standard` (registry and block checksums), `Deep` (full byte scan), `Paranoid` +(everything, re-verified). + ### Performance Metrics ```csharp -// Get statistics -var stats = db.GetStatistics(); +// Storage statistics (single-file mode) +StorageStatistics stats = db.GetStorageStatistics(); Console.WriteLine($"Total Size: {stats.TotalSize:N0} bytes"); -Console.WriteLine($"Block Count: {stats.BlockCount}"); -Console.WriteLine($"Fragmentation: {stats.FragmentationPercentage:F1}%"); -Console.WriteLine($"Free Space: {stats.FreeSpaceBytes:N0} bytes"); - -// Performance metrics -var metrics = db.GetPerformanceMetrics(); -Console.WriteLine($"Avg INSERT: {metrics.AvgInsertMs:F2}ms"); -Console.WriteLine($"Avg SELECT: {metrics.AvgSelectMs:F2}ms"); -Console.WriteLine($"Cache Hit Rate: {metrics.CacheHitRate:F1}%"); +Console.WriteLine($"Used: {stats.UsedSpace:N0} / Free: {stats.FreeSpace:N0} bytes"); +Console.WriteLine($"Pages: {stats.PageCount}, free pages: {stats.FreePages}"); +Console.WriteLine($"Blocks: {stats.BlockCount} ({stats.DirtyBlocks} dirty)"); +Console.WriteLine($"Fragmentation: {stats.FragmentationPercent:F1}%"); +Console.WriteLine($"WAL size: {stats.WalSize:N0} bytes"); +Console.WriteLine($"Last VACUUM: {stats.LastVacuum:u}"); + +// Cache statistics - the engine's live performance counters +var (qHits, qMisses, qHitRate, qCount) = db.GetQueryCacheStatistics(); +Console.WriteLine($"Query cache: {qHitRate:P1} hit rate over {qCount} entries"); + +var pageCache = db.GetPageCacheStatistics(); +Console.WriteLine($"Page cache: {pageCache.HitRate:P1} hit rate, {pageCache.CurrentSize}/{pageCache.Capacity} pages"); + +// Anything else the engine exposes as name/value pairs +foreach (var (key, value) in db.GetDatabaseStatistics()) +{ + Console.WriteLine($"{key}: {value}"); +} ``` +There is no `db.GetStatistics()` and no `db.GetPerformanceMetrics()`: storage numbers come from +`GetStorageStatistics()`, caches from `GetQueryCacheStatistics()` / `GetPageCacheStatistics()`. Per-stage +write timings come from the profiler in `src/SharpCoreDB/Diagnostics/WritePathProfiler.cs` +(the `WritePathProfilerTests` harness), and end-to-end ratios from the benchmark suite - not from a +database-level metrics API. Fragmentation is also exposed per extent by `FreeSpaceManager.GetDetailedStatistics()`. + ### Logging ```csharp -// Enable structured logging -var logger = LoggerFactory.Create(builder => -{ - builder.AddConsole(); - builder.AddFile("logs/sharpcoredb.log"); -}).CreateLogger(); +// Register logging on the same service provider the database is opened with +// (`Microsoft.Extensions.Logging` for AddLogging/AddConsole). +services.AddLogging(builder => builder.AddConsole()); +var provider = services.BuildServiceProvider(); -// Log critical operations +// Log signals that exist: logger.LogInformation("Database opened: {Path}", dbPath); -logger.LogWarning("Checkpoint took {Ms}ms", checkpointTime); -logger.LogError("Corruption detected: {Severity}", report.Severity); + +var stats = db.GetStorageStatistics(); +logger.LogWarning("Fragmentation {Percent:F1}%, WAL {WalSize} bytes", + stats.FragmentationPercent, stats.WalSize); + +using var detector = new CorruptionDetector(storage, ValidationMode.Quick); +var report = await detector.ValidateAsync(); +logger.LogError("Corruption detected: {Severity} - {Summary}", report.Severity, report.Summary); ``` --- @@ -301,48 +371,52 @@ tail -f logs/sharpcoredb.log #### Weekly Tasks ```csharp -// 1. Run VACUUM to defragment +// 1. Run VACUUM to defragment (SQL form; db.VacuumAsync(VacuumMode.Quick) is the API form) db.ExecuteSQL("VACUUM"); -// 2. Validate integrity (Standard mode, ~10ms/MB) -using var detector = new CorruptionDetector(provider, ValidationMode.Standard); +// 2. Validate integrity - the report carries BytesScanned/BlocksValidated/ValidationTime, +// so the cost is measured, not assumed +using var detector = new CorruptionDetector(storage, ValidationMode.Standard); var report = await detector.ValidateAsync(); Console.WriteLine(report.Summary); -// 3. Review performance metrics -var metrics = db.GetPerformanceMetrics(); -// Alert if INSERT time > 10ms or cache hit rate < 80% +// 3. Review the counters that exist +var stats = db.GetStorageStatistics(); +var (hits, misses, hitRate, count) = db.GetQueryCacheStatistics(); +// Alert on: FragmentationPercent growth, WalSize growth, query-cache hit-rate collapse ``` #### Monthly Tasks ```csharp -// 1. Deep validation (50ms/MB) -using var detector = new CorruptionDetector(provider, ValidationMode.Deep); +// 1. Deep validation (full byte scan - measure it on your data, do not assume a rate) +using var detector = new CorruptionDetector(storage, ValidationMode.Deep); var report = await detector.ValidateAsync(); -// 2. Analyze growth -var stats = db.GetStatistics(); -// Track totalSize, blockCount, fragmentation over time +// 2. Analyse growth +var stats = db.GetStorageStatistics(); +// Track stats.TotalSize, stats.BlockCount, stats.FragmentationPercent, stats.WalSize over time -// 3. Test restore from backup +// 3. Test restore from backup - and prove it by querying the copy File.Copy("./backups/latest.scdb", "./test/restore.scdb"); -// Verify can open and query +using var restored = new Database(provider, "./test/restore.scdb", "your-master-password"); +restored.ExecuteSQL("SELECT COUNT(*) FROM users"); ``` ### VACUUM Strategy ```csharp // When to VACUUM: -// - Fragmentation > 20% +// - FragmentationPercent above your threshold (AutoVacuum + FragmentationThreshold make it automatic) // - After large DELETE operations // - Monthly maintenance -var stats = db.GetStatistics(); -if (stats.FragmentationPercentage > 20) +var stats = db.GetStorageStatistics(); +if (stats.FragmentationPercent > 20) { Console.WriteLine("Running VACUUM..."); var result = await db.VacuumAsync(VacuumMode.Full); - Console.WriteLine($"Reclaimed {result.BytesReclaimed:N0} bytes"); + Console.WriteLine($"Reclaimed {result.BytesReclaimed:N0} bytes: " + + $"{result.FragmentationBefore:F1}% -> {result.FragmentationAfter:F1}%, {result.DurationMs}ms"); } ``` @@ -386,28 +460,43 @@ if (!result.Success) // 1. Create indexes db.ExecuteSQL("CREATE INDEX idx_users_email ON users(email)"); -// 2. Enable caches -var config = new DatabaseOptions +// 2. Enable caches (these live on DatabaseConfig, not DatabaseOptions) +var config = new DatabaseConfig { EnableQueryCache = true, + QueryCacheSize = 4096, EnablePageCache = true, - PageCacheCapacity = 2048, // Increase cache + PageCacheCapacity = 2048, // increase cache + UseMemoryMapping = true, }; ``` #### Issue: "WAL file growing too large" -**Cause:** Checkpointing disabled or too infrequent +**Cause:** A long write burst against a WAL region sized below it, or file growth from fragmentation — the WAL is applied by the provider and there is no checkpoint switch to disable **Solution:** ```csharp -// 1. Enable auto-checkpoint -var config = new DatabaseOptions +// The WAL is a region inside the .scdb file, not a sidecar file, and it is applied by the provider. +// There is no CHECKPOINT SQL statement and no EnableWalCheckpointing option. +// 1. Size the buffer for the burst and keep full durability +var config = new DatabaseConfig +{ + WalDurabilityMode = DurabilityMode.FullSync, + WalBufferSize = 4 * 1024 * 1024, + EnableAdaptiveWalBatching = true, +}; + +var options = new DatabaseOptions { - EnableWalCheckpointing = true, - WalCheckpointIntervalMs = 30000, // Checkpoint every 30s + StorageMode = StorageMode.SingleFile, + WalBufferSizePages = 2048, // in-file WAL region, in pages }; -// 2. Manual checkpoint -db.ExecuteSQL("CHECKPOINT"); +// 2. Flush pending work into the file (the API form) +db.Flush(); + +// 3. Shrink the file when fragmentation is the real cause +var result = await db.VacuumAsync(VacuumMode.Quick); +Console.WriteLine($"File after VACUUM: {result.FileSizeAfter:N0} bytes"); ``` --- @@ -417,12 +506,13 @@ db.ExecuteSQL("CHECKPOINT"); ### Encryption ```csharp -// Master password encryption (AES-256) +// Master password encryption (AES-256-GCM). The password is the key source: without it the file +// cannot be opened, and there is no plaintext mode for a password-protected database. var db = new Database( provider, "./data/secure.scdb", "strong-master-password-here", // ✅ Use strong password - config: new DatabaseOptions { /* ... */ } + config: new DatabaseConfig { /* DatabaseConfig, not DatabaseOptions */ } ); // Password requirements: @@ -434,17 +524,15 @@ var db = new Database( ### Access Control -```csharp -// User authentication -db.CreateUser("admin", "admin-password"); -db.CreateUser("readonly", "readonly-password"); +Single-file mode has **no in-database user management**: `CreateUser` / `Login` on +`DatabaseExtensions` throw `NotSupportedException` ("User management is not supported in single-file +mode"). Access control therefore sits at the process and deployment boundary: -// Login before operations -if (db.Login("admin", "admin-password")) -{ - // Perform admin operations -} -``` +- the encryption key comes from the master password, so whoever holds the password holds the data +- file-system ACLs on the `.scdb` file are the access control for local deployments +- for multi-user access, deploy the gRPC server, which serves multiple databases and enforces + HTTPS/TLS (minimum TLS 1.2) - never a plain-HTTP endpoint +- keep the password in a secrets manager, never in code or a checked-in settings file ### Network Security @@ -472,7 +560,7 @@ db.ExecuteSQL($"INSERT INTO users VALUES ({id}, '{userInput}')"); - [ ] **Indexes created** on frequently queried columns - [ ] **Page cache enabled** with adequate size - [ ] **Query cache enabled** for repeated queries -- [ ] **WAL checkpointing** configured appropriately +- [ ] **Durability and WAL sized** (`WalDurabilityMode`, `WalBufferSize`, `WalBufferSizePages`) - [ ] **Batch operations** used for bulk inserts - [ ] **VACUUM run** monthly or when fragmentation > 20% @@ -492,7 +580,9 @@ db.ExecuteBatchSQL(statements); sw.Stop(); Console.WriteLine($"1000 inserts: {sw.ElapsedMilliseconds}ms"); -// Target: <10ms (100k ops/sec) +// Compare against your OWN baseline, and against SQLite measured on this machine - never against a +// published number. Report ratios, not absolutes, and state the machine and posture (see +// ../BENCHMARK_RESULTS.md for the method this repository uses). ``` --- @@ -502,7 +592,7 @@ Console.WriteLine($"1000 inserts: {sw.ElapsedMilliseconds}ms"); ### DO ✅ 1. **Use `using` statements** to ensure Dispose() is called -2. **Enable WAL checkpointing** in production +2. **Keep `DurabilityMode.FullSync`** unless the data can be re-created after a crash 3. **Create backups** before VACUUM or repair 4. **Run validation** weekly (Standard mode) 5. **Monitor fragmentation** and VACUUM when needed @@ -513,7 +603,7 @@ Console.WriteLine($"1000 inserts: {sw.ElapsedMilliseconds}ms"); ### DON'T ❌ 1. **Don't** open multiple Database instances for same file -2. **Don't** disable WAL checkpointing in production +2. **Don't** drop to `DurabilityMode.Async` unless losing the last commits is acceptable to you 3. **Don't** skip backups before maintenance 4. **Don't** use string concatenation for SQL (SQL injection) 5. **Don't** run Deep validation on every request (too slow) @@ -559,7 +649,8 @@ Before going live: --- -**Last Updated:** 2026-01-28 -**Version:** 2.0 (SCDB Phase 5 Complete) +**Last Updated:** 2026-10-11 +**Version:** 2.1 (single-file format v2 - see [`README.md`](./README.md) and +[`../CHANGELOG.md`](../CHANGELOG.md)) 🎉 **You're ready for production!** diff --git a/docs/scdb/README.md b/docs/scdb/README.md index 983898ad..2e0f2104 100644 --- a/docs/scdb/README.md +++ b/docs/scdb/README.md @@ -2,80 +2,91 @@ ## Overview -The `.scdb` format is a **single-file, uncompressed, page-aligned storage format** for SharpCoreDB designed for maximum performance on modern SSDs. It consolidates the current multi-file architecture into a single random-access file optimized for zero-copy I/O and memory-mapping. +The `.scdb` format is SharpCoreDB's **single-file, page-aligned storage container** and the default +storage mode of the 2.1 line: one randomly-addressable file holding the block registry, an embedded +write-ahead log, the table directory and the data blocks. + +- **Encryption** — optional AES-256-GCM covering the whole file (block data, registry, free-space map + and WAL). New encrypted files use `EncryptionMode = 2`; see [`ENCRYPTION.md`](./ENCRYPTION.md). +- **Compression** — the container header is uncompressed (`ScdbFileHeader.CompressionMode` is 0 for + `.scdb`); columnar blocks written through the columnar adapter are Brotli-compressed by default. +- **Integrity** — a SHA-256 checksum of the file in the header, plus a SHA-256 checksum per block + registry entry. + +Pass a `.scdb` path to select single-file mode automatically; a directory path selects the classic +multi-file layout (`DirectoryStorageProvider`). Both are first-class in the 2.1 line. ## 📁 Files -- **[SCDB_FILE_FORMAT_DESIGN.md](./FILE_FORMAT_DESIGN.md)** - Complete design specification -- **[ScdbStructures.cs](../../src/SharpCoreDB/Storage/Scdb/ScdbStructures.cs)** - Binary format definitions (C# 15 structs) -- **[ScdbFile.cs](../../src/SharpCoreDB/Storage/Scdb/ScdbFile.cs)** - Implementation skeleton +- **[PRODUCTION_GUIDE.md](./PRODUCTION_GUIDE.md)** - configuration, backup, disaster recovery and tuning. +- **[ENCRYPTION.md](./ENCRYPTION.md)** - full at-rest encryption, key model, password/key rotation. +- **[../serialization/SERIALIZATION_AND_STORAGE_GUIDE.md](../serialization/SERIALIZATION_AND_STORAGE_GUIDE.md)** - record/column layout and free-space behaviour. +- **[ScdbStructures.cs](../../src/SharpCoreDB/Storage/Scdb/ScdbStructures.cs)** - the on-disk binary layout (`ScdbFileHeader`, `RegistryChunkHeader`, `BlockEntry`, `WalHeader`, `FreeSpaceMapHeader`). +- **[SingleFileStorageProvider.cs](../../src/SharpCoreDB/Storage/SingleFileStorageProvider.cs)** - the engine that owns the block registry, free-space map and WAL. + +> ⚠️ **`Storage/Scdb/ScdbFile.cs` is a low-level prototype, not the production path.** Its +> `ReadBlock` returns an empty span and its `WriteBlock` registers neither the block nor a WAL entry; +> nothing in the product or the test suite uses it. Production single-file access goes through +> `SingleFileStorageProvider` / `DatabaseOptions.CreateSingleFileDefault()` - see the Usage Examples +> section below. ## 🎯 Design Goals -1. ✅ **Zero Compression Overhead** - All data stored in native binary format -2. ✅ **SSD-Optimized** - 4KB-aligned pages for direct I/O -3. ✅ **Memory-Mappable** - Fixed offsets, no varints, direct struct access -4. ✅ **Fragmentation Avoidance** - Built-in Free Space Map (FSM) -5. ✅ **Extensible** - Reserved fields, versioned headers -6. ✅ **Crash-Safe** - Per-block checksums, embedded WAL -7. ✅ **Maintainable** - Self-describing format with block registry +1. **Page-aligned** - 4 KiB pages by default (`DatabaseOptions.PageSize`, `ScdbFileHeader.PageSize`). +2. **Memory-mappable** - fixed offsets, no varints, struct-parsed headers (`DatabaseOptions.EnableMemoryMapping`). +3. **Fragmentation-aware** - a free-space map tracks page allocation, so `VACUUM` runs incrementally instead of rewriting the file. +4. **Extensible** - versioned headers, reserved fields, feature flags. +5. **Crash-safe** - embedded WAL plus per-block checksums; recovery replays from the last checkpoint. +6. **Self-describing** - a namespaced block registry (`table:users:data`, `table:users:index:pk`) maps names to offsets. + +## 📊 Performance -## 📊 Performance Expectations +The format is what makes the measured numbers possible, and the measured numbers live in one place: +[`../BENCHMARK_RESULTS.md`](../BENCHMARK_RESULTS.md). The current v2.1.0 publication reports, on the +fair-`INTEGER PRIMARY KEY` arm and against SQLite in the same run: -| Metric | Current (multi-file) | SCDB (single-file) | Improvement | -|--------|---------------------|-------------------|-------------| -| **Startup Time** | 100ms (100 files) | 10ms (1 file) | 10x faster | -| **Write Throughput** | 50k ops/sec | 100k ops/sec | 2x faster | -| **Defragmentation** | 60s full rewrite | 600ms incremental | 100x faster | -| **Crash Recovery** | 500ms | 100ms | 5x faster | -| **File Handles** | 100+ | 1 | 100x fewer | +| Operation | Ratio (SharpCoreDB ÷ SQLite) | Spread across reps | +|-----------|------------------------------|--------------------| +| INSERT | **1,17×** (encrypted default posture) | 1,03–1,27 | +| UPDATE | **0,53×** | 0,42–0,53 | + +Quote the ratios, never lone absolutes: absolute ops/sec move between runs on the same machine, and +SQLite's own reference drifts. The raw per-rep JSON behind the publication is in +`tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/`. ## 🏗️ File Structure ``` -┌─────────────────────────────────────────────────────────┐ -│ SCDB FILE LAYOUT │ -├─────────────────────────────────────────────────────────┤ -│ [0x0000] File Header (512 bytes) │ -│ - Magic: "SCDB\x10\x00\x00\x00" │ -│ - Page size: 4096 (default, tunable) │ -│ - Block registry offset │ -│ - FSM offset (Free Space Map) │ -│ - WAL offset (Write-Ahead Log) │ -│ - Table directory offset │ -├─────────────────────────────────────────────────────────┤ -│ [0x0200] Block Registry (variable, page-aligned) │ -│ - Self-describing index of all blocks │ -│ - Namespaced block names: │ -│ * "table:users:data" │ -│ * "table:users:index:pk" │ -│ * "sys:fsm" │ -│ * "sys:wal" │ -├─────────────────────────────────────────────────────────┤ -│ [Aligned] Free Space Map (FSM) │ -│ - Two-level bitmap (inspired by PostgreSQL) │ -│ - L1: 1 bit per page (allocated/free) │ -│ - L2: Extent map for large allocations │ -├─────────────────────────────────────────────────────────┤ -│ [Aligned] Write-Ahead Log (WAL) │ -│ - Circular buffer of 4KB entries │ -│ - Transaction boundaries │ -│ - Redo/undo logs for crash recovery │ -├─────────────────────────────────────────────────────────┤ -│ [Aligned] Table Directory │ -│ - Table metadata │ -│ - Schema definitions │ -│ - Index pointers │ -├─────────────────────────────────────────────────────────┤ -│ [Aligned] Data Blocks (namespaced) │ -│ - Reuses existing PageBased format (8KB pages) │ -│ - Reuses existing Columnar format (.dat) │ -│ - Per-block SHA-256 checksums │ -└─────────────────────────────────────────────────────────┘ +[0x0000] File header - ScdbFileHeader, 512 bytes, fixed size + magic "SCDB\x10\x00\x00\x00" + format version (v2), page size (4096 default), + encryption mode + key id + nonce, file SHA-256 checksum, feature flags, + transaction state (last txn id, last checkpoint LSN), allocated pages, + RegistryRootOffset/Length, WalOffset/Length, TableDirOffset/Length +[Registry] Block registry - growable chain of page-aligned chunks (format v2, issue #345) + [RegistryChunkHeader (64 B, magic "BRGC")][BlockEntry * EntryCount], each chunk + linking to the next via NextChunkOffset/NextChunkLength + BlockEntry (96 B): Name[32] "namespace:identifier:subtype", BlockType, Offset, + Length, Checksum[32] (SHA-256 of the block data), Flags +[WAL] Embedded write-ahead log - WalHeader plus fixed-size entries with head/tail offsets; + checkpointed in place, so there is no sidecar .wal file to back up +[TableDir] Table directory (sys:tabledir) - table metadata, schemas, index pointers +[Blocks] Page-aligned data blocks - table data, indexes, columnar payloads (Brotli-compressed), + overflow chains; every block is page-aligned and carries its own SHA-256 checksum ``` +The metadata regions are **named blocks resolved through the registry**, not fixed offsets: the +free-space map (`sys:fsm`), the WAL (`sys:wal`), the table directory (`sys:tabledir`) and the +metadata payload (`sys:metadata`). Format v1 used fixed `FsmOffset`/`FsmLength` header fields; those +two slots are now `ReservedRegion0`/`ReservedRegion1`, which is why the free-space map can grow with +the file (see `SingleFileStorageProvider`'s dynamic-metadata path). + ## 🔧 C# 15 Features Used +> The snippets below illustrate the language features the implementation leans on; the authoritative +> declarations live in [`ScdbStructures.cs`](../../src/SharpCoreDB/Storage/Scdb/ScdbStructures.cs), +> [`ScdbFile.cs`](../../src/SharpCoreDB/Storage/Scdb/ScdbFile.cs) and +> [`SingleFileStorageProvider.cs`](../../src/SharpCoreDB/Storage/SingleFileStorageProvider.cs). + ### 1. **Readonly Structs with Fixed Buffers** ```csharp @@ -109,131 +120,128 @@ public bool IsDisposed { get => field; } // Implicit backing field ### 3. **Primary Constructors** ```csharp -public sealed class ScdbFile(string path, FileStream fs) : IDisposable +// Real example: src/SharpCoreDB/Storage/Columnar/ColumnCodec.cs +public sealed class ColumnCodec(ColumnFormat format) { - public string FilePath => path; // Direct parameter access - - public void Flush() => fs.Flush(flushToDisk: true); + private readonly ColumnFormat _format = format ?? throw new ArgumentNullException(nameof(format)); + + public byte[] EncodeColumn(string columnName, object?[] values) { /* ... */ } } ``` ### 4. **Pattern Matching with Switch Expressions** ```csharp -var fileAccess = mode switch +// Real example: src/SharpCoreDB/Storage/Overflow/StorageStrategy.cs +// (this StorageMode is the overflow strategy enum, not DatabaseOptions.StorageMode) +public static string GetDescription(StorageMode mode) => mode switch { - var m when m.HasFlag(ScdbOpenMode.ReadWrite) => FileAccess.ReadWrite, - var m when m.HasFlag(ScdbOpenMode.Write) => FileAccess.Write, - _ => FileAccess.Read + StorageMode.Inline => "Stored inline in data page (fastest)", + StorageMode.Overflow => "Stored in overflow page chain (medium)", + StorageMode.FileStream => "Stored in external file (unlimited size)", + _ => "Unknown storage mode", }; ``` ### 5. **ReadOnlySpan for Zero-Copy Parsing** ```csharp -public unsafe ReadOnlySpan ReadBlock(string name) +// Real example: src/SharpCoreDB/Storage/Scdb/ScdbStructures.cs - the header is reinterpreted in +// place: no allocation, no copy +[MethodImpl(MethodImplOptions.AggressiveInlining)] +public static unsafe ScdbFileHeader Parse(ReadOnlySpan data) { - // Memory-mapped I/O: zero allocations - using var accessor = _mmf.CreateViewAccessor(offset, length); - byte* ptr = null; - accessor.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr); - return new ReadOnlySpan(ptr, (int)length); + if (data.Length < HEADER_SIZE) + throw new InvalidDataException($"Header too small: {data.Length} < {HEADER_SIZE}"); + + fixed (byte* ptr = data) + { + return *(ScdbFileHeader*)ptr; + } } ``` ## 🚀 Usage Examples -### Opening a Database +### Opening a database ```csharp -// Create new .scdb file -using var db = ScdbFile.Open("mydb.scdb", ScdbOpenMode.Create | ScdbOpenMode.ReadWrite); +// A .scdb path selects single-file mode automatically (DatabaseExtensions.DetectStorageMode), +// so the ordinary construction path is all you need: +using var db = new Database(services, "./data/myapp.scdb", "your-master-password"); -// Open existing with memory-mapping (zero-copy reads) -using var db = ScdbFile.Open("mydb.scdb", ScdbOpenMode.ReadWrite | ScdbOpenMode.MemoryMapped); +db.ExecuteSQL("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)"); +db.ExecuteSQL("INSERT INTO users VALUES (1, 'Alice')"); ``` -### Writing Data +Tuned defaults come from `DatabaseOptions.CreateSingleFileDefault()`. When you need block-level access, +drive the provider directly - it is what owns the registry, the free-space map and the WAL: ```csharp -var tableData = SerializeTable(userTable); -db.WriteBlock("table:users:data", BlockType.TableData, tableData); +var options = DatabaseOptions.CreateSingleFileDefault(); // page size, WAL pages, FSM pages +using var provider = SingleFileStorageProvider.Open("./data/myapp.scdb", options); -var indexData = SerializeIndex(primaryKeyIndex); -db.WriteBlock("table:users:index:pk", BlockType.IndexData, indexData); - -db.Flush(); // Commit to disk +await provider.WriteBlockAsync("table:users:data", payload); // registered and WAL-logged +var block = await provider.ReadBlockAsync("table:users:data"); +await provider.CheckpointAsync(); // flush the WAL into the data blocks ``` -### Reading Data +### Block names and types -```csharp -// Zero-copy read via memory mapping -ReadOnlySpan data = db.ReadBlock("table:users:data"); +Blocks are namespaced strings; the type travels in the registry entry (`BlockType`): -// Deserialize directly from span -var table = DeserializeTable(data); -``` +| Block name | `BlockType` | Owner | +|------------|-------------|-------| +| `sys:fsm` | `FreeSpaceMap` | `FreeSpaceManager` | +| `sys:wal` | `WriteAheadLog` | `WalManager` | +| `sys:tabledir` | `TableDirectory` | `TableDirectoryManager` | +| `table::data` | `TableData` | table writer | +| `table::index:` | `IndexData` | index writer | +| `blob:*` / `temp:*` | `BlobData` / `TemporaryData` | overflow and temporary writers | -### Incremental VACUUM +### Maintenance ```csharp -// Background defragmentation (non-blocking) -await db.VacuumIncrementalAsync(maxPagesToMove: 1000); - -// Full VACUUM (creates new file) -await db.VacuumFullAsync(); +await db.VacuumAsync(VacuumMode.Quick); // reclaim obvious holes (the default) +await db.VacuumAsync(VacuumMode.Incremental); // page-by-page defragmentation +await db.VacuumAsync(VacuumMode.Full); // rewrite the file ``` -## 📈 Comparison with Other Formats +The `VACUUM` statement is parsed as well, so `db.ExecuteSQL("VACUUM")` works. The synchronous +`ScdbFile.VacuumIncremental(int maxPagesToMove = 1000)` / `ScdbFile.VacuumFull()` pair belongs to the +prototype described at the top of this page, not to the engine. -### SQLite +## 📈 How it compares -``` -✅ Single file -✅ Page-aligned -❌ No embedded WAL (separate file) -❌ No FSM (fragmentation issues) -❌ VACUUM requires full rewrite -``` - -### LiteDB - -``` -✅ Single file -❌ No page alignment (bad for SSDs) -❌ BSON parsing overhead -❌ No FSM -❌ No embedded WAL -``` +- **vs SQLite** - both are single-file, page-aligned engines. SQLite keeps its WAL in a sidecar + `-wal` file and its `VACUUM` is a full rewrite; `.scdb` keeps the WAL inside the database file and + reclaims space incrementally. On the fair-PK arm SQLite still leads on `UPDATE`/`DELETE` + throughput - the measured ratios and their caveats are in + [`../BENCHMARK_RESULTS.md`](../BENCHMARK_RESULTS.md). +- **vs LiteDB** - LiteDB is a document store (BSON with a query API); `.scdb` is a relational engine + with a SQL surface, so the difference is mostly the data model rather than the storage layout. +- **vs JSON-over-files** - the format is binary and page-aligned, so there is no parse cost per read + and no per-table file handle. -### SCDB - -``` -✅ Single file -✅ Page-aligned (SSD-optimized) -✅ Embedded WAL (atomic commits) -✅ FSM (incremental VACUUM) -✅ Zero-copy reads (memory-mapped) -✅ Per-block checksums -✅ Namespaced blocks -``` +This page deliberately limits competitor claims to what is measured or structurally verifiable; the +numbers and the method live in [`../BENCHMARK_RESULTS.md`](../BENCHMARK_RESULTS.md). ## 🔐 Security Features -### 1. **Per-Block Checksums** +### 1. **Checksums** ```csharp -// SHA-256 checksum stored in block entry -public readonly unsafe fixed byte Checksum[32]; +// Header: SHA-256 of the whole file, excluding this field +public unsafe fixed byte FileChecksum[32]; // ScdbFileHeader -// Validation on read -if (!blockEntry.ValidateChecksum(blockData)) -{ - throw new InvalidDataException("Block checksum mismatch"); -} +// Registry entry: SHA-256 of the block data +public unsafe fixed byte Checksum[32]; // BlockEntry ``` +`CorruptionDetector` validates at four levels - `Quick` (header only), `Standard` (header, blocks and +checksums), `Deep` (adds the WAL) and `Paranoid` (re-reads and verifies everything) - and `RepairTool` +acts on the resulting `CorruptionReport`, optionally taking a backup first. + ### 2. **Optional Encryption (full at-rest)** > ⚠️ **Format note:** `EncryptionMode = 1` (AES-256-GCM over block data only, issue #341) is @@ -254,142 +262,99 @@ password & key rotation, crash-safety, AOT/Sonar compliance notes and a full wor ### 3. **Transaction Integrity** ```csharp -// WAL ensures crash safety -db.BeginTransaction(); -db.WriteBlock("table:orders:data", ...); -db.CommitTransaction(); // Atomic via WAL checkpoint +// The provider owns the WAL: a checkpoint is the durability boundary, and recovery replays the +// entries written since the last checkpoint. +await provider.CheckpointAsync(); + +// After a crash, recovery runs through RecoveryManager (WAL replay from the last checkpoint LSN). ``` ## 🧪 Testing Strategy -### 1. **Unit Tests** - -```csharp -[Fact] -public void FileHeader_Parse_ValidatesCorrectly() -{ - var header = ScdbFileHeader.CreateDefault(); - Span buffer = stackalloc byte[512]; - header.WriteTo(buffer); - - var parsed = ScdbFileHeader.Parse(buffer); - - Assert.True(parsed.IsValid); - Assert.Equal(ScdbFileHeader.MAGIC, parsed.Magic); -} -``` - -### 2. **Integration Tests** +The format is covered by the suites that must stay green on every storage change: -```csharp -[Fact] -public void ScdbFile_WriteAndRead_RoundTrip() -{ - using var db = ScdbFile.Open("test.scdb", ScdbOpenMode.Create); - - var testData = Encoding.UTF8.GetBytes("Hello, SCDB!"); - db.WriteBlock("test:data", BlockType.TableData, testData); - - var readData = db.ReadBlock("test:data"); - - Assert.True(readData.SequenceEqual(testData)); -} -``` +| Suite | What it pins | +|-------|--------------| +| `ReopenRoundTripMatrixTests` | close/reopen round-trips across the column shapes and index kinds | +| `SingleFileDirectoryParityTests` | `.scdb` results equal directory-mode results for the same workload | +| `CrashRecoveryTests` | WAL replay and recovery after a simulated crash | +| `FixedWidthPatchTests`, `FixedWidthBulkUpdateTests`, `FixedWidthBulkDeleteTests` | in-place delta/patch paths | +| `FormatCompatPolicyTests` | format acceptance (v1 readable, v2 written) | -### 3. **Crash Recovery Tests** +Run them with the Microsoft.Testing.Platform executable (`dotnet test` does not work in this +repository - see `AGENTS.md`): -```csharp -[Fact] -public void ScdbFile_CrashRecovery_RestoresState() -{ - // Simulate crash during write - using (var db = ScdbFile.Open("crash.scdb", ScdbOpenMode.Create)) - { - db.BeginTransaction(); - db.WriteBlock("test:data", ...); - // Simulate crash (no commit) - } - - // Reopen and verify WAL recovery - using var db2 = ScdbFile.Open("crash.scdb"); - var data = db2.ReadBlock("test:data"); - - Assert.True(data.IsEmpty); // Transaction was rolled back -} +```powershell +tests/SharpCoreDB.Tests/bin/Release/net11.0/SharpCoreDB.Tests.exe -filterVSTest 'FullyQualifiedName~ReopenRoundTripMatrixTests' ``` ## 🛣️ Migration Path -### Option 1: Parallel Support (Recommended) - -```csharp -public interface IStorageProvider -{ - StorageMode Mode { get; } // MultiFile or SingleFile -} - -// Auto-detect by file extension -IStorageProvider storage = path.EndsWith(".scdb") - ? new ScdbStorageProvider(path) - : new MultiFileStorageProvider(path); -``` - -### Option 2: Migration Tool +### Option 1: Parallel support (recommended) -```bash -# Convert existing database to .scdb -sharpcoredb migrate --from mydb/ --to mydb.scdb +Both layouts implement one interface, `IStorageProvider` (`SingleFileStorageProvider`, +`DirectoryStorageProvider`), and `Database` picks one from the path it is given +(`DatabaseExtensions.DetectStorageMode`: a `.scdb` file - or any existing file - means single-file). +`StorageFormatSwitcher` covers the case where the active provider has to change at runtime. -# Progress: 10,000 records in ~30 seconds -# [█████████████████████████████] 100% (10,000 / 10,000) +```csharp +// Nothing to select by hand in the normal case: +// "./data/mydb.scdb" -> SingleFileStorageProvider +// "./data/mydb" -> DirectoryStorageProvider +using var db = new Database(services, "./data/mydb.scdb", "your-master-password"); ``` -### Option 3: Hybrid Mode +### Option 2: Migration API ```csharp -// Use .scdb for new tables, keep legacy format for old tables -var config = new DatabaseConfig -{ - DefaultStorageFormat = StorageFormat.Scdb, - AllowLegacyFormats = true -}; +// directory -> single file +var toSingleFile = await DatabaseMigrator.MigrateToSingleFileAsync( + sourceDirectoryPath: "./data/mydb", + targetScdbPath: "./data/mydb.scdb", + password: "your-master-password", + progress: new Progress(p => Console.WriteLine($"{p:P0}"))); + +// single file -> directory +var toDirectory = await DatabaseMigrator.MigrateToDirectoryAsync( + sourceScdbPath: "./data/mydb.scdb", + targetDirectoryPath: "./data/mydb-restored", + password: "your-master-password"); ``` -## 📚 Implementation Roadmap - -### Phase 1: Core Format (Week 1-2) - -- [x] Define binary structures (`ScdbStructures.cs`) -- [x] Implement file header read/write -- [ ] Implement block registry -- [ ] Basic read/write operations - -### Phase 2: FSM & Allocation (Week 3-4) +The same two operations exist on a live database (`db.MigrateToSingleFileAsync`, +`db.MigrateToDirectoryAsync`) and on `ScdbMigrator`. There is **no `sharpcoredb` CLI** in this +repository - the migration entry points are the APIs above. -- [ ] Free Space Map implementation -- [ ] Page allocation/deallocation -- [ ] Extent tracking for large allocations +### Option 3: Hybrid deployments -### Phase 3: WAL & Recovery (Week 5-6) +Storage mode is a per-database choice, not a per-table one, so a hybrid deployment keeps a directory +database and a `.scdb` database side by side and moves a table across when it is ready. -- [ ] WAL entry structure -- [ ] Transaction logging -- [ ] Crash recovery mechanism -- [ ] Checkpoint implementation +## 📚 Implementation Status -### Phase 4: Integration (Week 7-8) +The format this page describes is implemented and shipped; the phase checklist this section used to +carry was a design-time plan for the `ScdbFile` prototype, and the engine was built along a different +path. The mapping from that plan to shipped code: -- [ ] Integrate with existing `PageBasedEngine` -- [ ] Integrate with existing `ColumnarEngine` -- [ ] Migration tool (multi-file → .scdb) -- [ ] Performance benchmarks +| Planned item | Where it ships | +|--------------|----------------| +| Binary structures, file header read/write | `ScdbStructures.cs` (`ScdbFileHeader` 512 B, `RegistryChunkHeader` 64 B, `BlockEntry` 96 B, `WalHeader`, `FreeSpaceMapHeader`) | +| Block registry | `Storage/BlockRegistry.cs` - growable chunk chain rooted at `ScdbFileHeader.RegistryRootOffset` (format v2, issue #345) | +| Basic read/write operations | `SingleFileStorageProvider`, `SingleFileDatabase`, `SingleFileTable` | +| Free Space Map, page allocation, extent tracking | `Storage/FreeSpaceManager.cs` with `Storage/Scdb/ExtentAllocator.cs` (`DatabaseOptions.FsmSizePages`, `FsmStatistics`) | +| WAL entries, transaction logging, crash recovery, checkpoint | `Storage/WalManager.cs`, `Storage/Scdb/RecoveryManager.cs`, `SingleFileStorageProvider.CheckpointAsync` | +| Integrate with the page-based and columnar engines | `Storage/Scdb/PageBasedAdapter.cs`, `Storage/Scdb/ColumnarAdapter.cs` | +| Migration tool (multi-file → `.scdb`) | `Migration/DatabaseMigrator.cs`, `Storage/Scdb/ScdbMigrator.cs` | +| Error handling, corruption detection and repair | `Storage/Scdb/ScdbExceptions.cs`, `CorruptionDetector.cs`, `RepairTool.cs` | +| Edge-case testing, benchmarks | the canary suites listed under Testing Strategy, plus `tests/benchmarks/` | -### Phase 5: Production Hardening (Week 9-10) +Open work is tracked in [`../../ROADMAP.md`](../../ROADMAP.md) and +[`../PROJECT_STATUS.md`](../PROJECT_STATUS.md): this page documents the format, it is not a task list. -- [ ] Comprehensive error handling -- [ ] Edge case testing -- [ ] Corruption detection & repair -- [ ] Documentation & examples +**The one genuinely unfinished item** is the public `ScdbFile` prototype - its registry lookup, +free-space allocation, block registration and WAL logging are `NOTE:`-marked stubs, and nothing in the +product or the test suite uses it. It is listed in `ROADMAP.md` under **In Progress** so that it does +not disappear again. ## 🤝 Contributing @@ -423,10 +388,12 @@ MIT License - See LICENSE file for details. --- -**Status:** ✅ Design Complete | 🚧 Implementation In Progress +**Status:** shipped and maintained. New databases in the 2.1 line store as `.scdb`; the format is read +by every package in the family, and `FormatCompatPolicyTests` keeps v1 files readable while new files +are written as format v2. -**Next Steps:** -1. Review design document -2. Implement block registry -3. Add unit tests for struct serialization -4. Benchmark header read/write performance +**See also:** +1. [`PRODUCTION_GUIDE.md`](./PRODUCTION_GUIDE.md) - configuration, backup, recovery, tuning +2. [`ENCRYPTION.md`](./ENCRYPTION.md) - encryption model and key rotation +3. [`../serialization/SERIALIZATION_AND_STORAGE_GUIDE.md`](../serialization/SERIALIZATION_AND_STORAGE_GUIDE.md) - record and column layout +4. [`../BENCHMARK_RESULTS.md`](../BENCHMARK_RESULTS.md) - measured ratios and method diff --git a/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs b/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs index 3ad4423d..977b1ecd 100644 --- a/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs +++ b/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs @@ -1,5 +1,7 @@ namespace SharpCoreDB.Tests.Documentation; +using System.Text.RegularExpressions; + public sealed class DocumentationConsistencyTests { [Fact] @@ -58,6 +60,55 @@ public void ObsoleteDocumentationFiles_ShouldBeRemoved() } } + [Fact] + public void DocumentationMarkdownLinks_ShouldResolveToExistingFiles() + { + // Arrange: every relative link in docs/**, resolved from the file that contains it. + var docsRoot = GetRepositoryFilePath("docs"); + var failures = new List(); + var linkPattern = new Regex(@"\]\(([^)\s]+)\)", RegexOptions.Compiled); + + foreach (var file in Directory.EnumerateFiles(docsRoot, "*.md", SearchOption.AllDirectories)) + { + var lineNumber = 0; + + foreach (var line in File.ReadLines(file)) + { + lineNumber++; + + foreach (Match match in linkPattern.Matches(line)) + { + var target = match.Groups[1].Value; + + if (target.StartsWith('#') + || target.StartsWith("http://", StringComparison.OrdinalIgnoreCase) + || target.StartsWith("https://", StringComparison.OrdinalIgnoreCase) + || target.StartsWith("mailto:", StringComparison.OrdinalIgnoreCase)) + { + continue; + } + + var path = target.Split('#')[0]; + + if (string.IsNullOrWhiteSpace(path)) + { + continue; + } + + var resolved = Path.GetFullPath(Path.Combine(Path.GetDirectoryName(file)!, path.Replace('/', Path.DirectorySeparatorChar))); + + if (!File.Exists(resolved) && !Directory.Exists(resolved)) + { + failures.Add($"{Path.GetRelativePath(docsRoot, file)}:{lineNumber} -> {target}"); + } + } + } + } + + // Act / Assert + Assert.Empty(failures); + } + private static string GetRepositoryFilePath(params string[] segments) { var current = new DirectoryInfo(AppContext.BaseDirectory); From bdb4e094d4c1eecfb7a9acf190b51e260a5d8e1d Mon Sep 17 00:00:00 2001 From: MPCoreDeveloper Date: Sun, 11 Oct 2026 17:57:20 +0200 Subject: [PATCH 5/6] docs: audit the documentation set against the shipped 2.1 engine and pin the defect classes with tests Sweep: every backticked identifier, file path and package id in docs/** diffed against the src/** symbol sets, the project list and Test-Path, then each suspect read at its call site. Corrected (each verified in source, never inferred): - docs/server/CONFIGURATION_SCHEMA.md rewritten: the server registers JSON configuration sources only (no TOML/YAML parser exists in the repository), Databases is an array, and the real Logging/Performance/Security/ConnectionPool/TenantQuotas/Projections surface is documented with the Program.cs startup validation (TLS cannot be disabled, 32-char JWT secret, private key required unless the certificate is a .pfx). - docs/internals/OPTIMIZER_GUIDE.md: removed SubqueryOptimizer, JoinReorderer, OptimizerCache, OptimizerIntegration, QueryOptimizerExtensions, InvalidateOptimizationCache, FilterExecutor, PredicateAnalyzer and the unmeasured "Execution Impact" table; documents the real Optimization.QueryOptimizer + CostEstimator, the Database/Caching plan cache, the subquery classifier/planner/executor/cache, and the honest placeholders (JoinsReordered = hasJoins, SubqueriesEliminated = !hasSubquery, hash join disabled so joins are nested-loop). - docs/QUERY_PLAN_CACHE.md: GetPlanCacheStats()/ClearPlanCache() are internal (InternalsVisibleTo for tests/benchmarks only) - no longer shown as consumer code. - docs/manual/{migration,providers}.md: the FluentMigrator integration ships in SharpCoreDB.Extensions (AddSharpCoreDBFluentMigrator); there is no SharpCoreDB.FluentMigrator package. - docs/Vectors/TECHNICAL_SPEC.md: SharpCoreDB.EntityFrameworkCore has no vector support at all and no project references Microsoft.Extensions.AI. - docs/sharpcoredb-known-issues.md: verified item by item against 2.1 (items 2-5 fixed in the engine, 6 stays the default with the UseSqliteIntegerAffinity opt-in); the non-existent DurabilityMode.FlushToDisk is corrected; every item annotated. - Dead references removed: INDEX.md (4), serialization/README.md (2), graphrag/EF_CORE_DOCUMENTATION_INDEX.md (3), analytics/README.md, the issue-56 status table, Option.FromNullOrEmpty -> Prelude.Optional, stale v1.7.0 titles, and the never-created PERFORMANCE_VALIDATION_AND_ADVICE_CSHARP15_NET11.md pointer. Guards (DocumentationConsistencyTests, red->green by planting each defect): index paths resolve; no TOML sample or "Supported Formats: TOML" claim in the server guide; internal plan-cache APIs not presented as public; every known-issue item annotated; every install-snippet package id resolves to a project or central package version. Docs suite 9/0; core suite 2004/0 failed/0 skipped in 89,2 s; build 0 errors. --- docs/CHANGELOG.md | 3 + docs/FUNCTIONAL_NULL_SAFETY.md | 13 +- docs/INDEX.md | 16 +- docs/QUERY_PLAN_CACHE.md | 31 +- docs/README.md | 2 + docs/Vectors/TECHNICAL_SPEC.md | 23 +- docs/analytics/README.md | 4 +- .../ISSUE_56_ACCEPTANCE_CRITERIA.md | 66 +- docs/graphrag/EF_CORE_DOCUMENTATION_INDEX.md | 28 +- docs/internals/OPTIMIZER_GUIDE.md | 544 ++++++-------- docs/manual/migration.md | 6 +- docs/manual/providers.md | 7 +- .../INSERT_UPDATE_PERFORMANCE_PLAN.md | 2 +- docs/performance/WORKLOG.md | 10 + docs/serialization/README.md | 8 +- docs/server/CONFIGURATION_SCHEMA.md | 690 +++++++----------- docs/sharpcoredb-known-issues.md | 63 +- .../DocumentationConsistencyTests.cs | 116 +++ 18 files changed, 795 insertions(+), 837 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index acf1d9a5..1afb19c2 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -46,6 +46,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **The documentation set was audited against the shipped 2.1 engine, and every page that described an API, file or format the engine does not have was corrected.** The sweep diffed each backticked identifier, file path and package id in `docs/**` against the source (symbol sets, the project list, `Test-Path`) and then read every suspect site rather than inferring. What was wrong: `docs/server/CONFIGURATION_SCHEMA.md` advertised *"Supported Formats: TOML, JSON, YAML"* and built ~400 lines of underscore-keyed TOML (`server_name`, `tls_enabled`) for a server that registers **JSON configuration sources only** — no TOML or YAML parser exists anywhere in the repository — listed `sharpcoredb.toml`/`sharpcoredb.yaml` in the load order, documented `Logging`/`Performance`/`Security` property sets that do not exist, invented `[server.replication]`/`[server.monitoring]` sections no type binds (and a `SHARPCOREDB_*` environment prefix where the server uses `Server__…`), and named `sharpcoredb/server:1.7.0` against the real `ghcr.io/mpcoredeveloper/sharpcoredb-server`; it is **rewritten** from `ServerConfiguration`, `Program.cs` and the shipped `appsettings.json`, including the real startup validation (TLS cannot be disabled, 32-character JWT secret, private key required unless the certificate is a `.pfx`). `docs/internals/OPTIMIZER_GUIDE.md` documented components that do not exist — `SubqueryOptimizer` (EXISTS→semi/anti-join rewriting; `SemiJoin`/`AntiJoin` appear nowhere in the source), `JoinReorderer`, `OptimizerCache`, `QueryOptimizerExtensions`, `OptimizerIntegration.GetOrOptimize(...)`, `InvalidateOptimizationCache()`, `FilterExecutor`, `PredicateAnalyzer` — and a 2-5×/5-20×/50-1000× "Execution Impact" table; it now describes the real path (`Optimization.QueryOptimizer` + `CostEstimator`, wired from `Services/QueryCompiler.cs`), the real plan cache (`Database/Caching/Database.PlanCaching.cs` + `Services/QueryPlanCache.cs`), the real subquery subsystem (`SubqueryClassifier`/`Planner`/`Executor`/`Cache`) and the honest placeholders (`JoinsReordered = hasJoins`, `SubqueriesEliminated = !hasSubquery`, `HashJoinThreshold = int.MaxValue`, so joins are nested-loop). `docs/QUERY_PLAN_CACHE.md` presented `db.GetPlanCacheStats()` and `db.ClearPlanCache()` as consumer code; both are `internal` (`InternalsVisibleTo` reaches tests and benchmarks only), and the page now says so. `docs/manual/migration.md`/`providers.md` named a **`SharpCoreDB.FluentMigrator`** package that does not exist — the integration ships inside `SharpCoreDB.Extensions` as `AddSharpCoreDBFluentMigrator()` — and `docs/Vectors/TECHNICAL_SPEC.md` claimed `SharpCoreDB.EntityFrameworkCore` exposes `HasVectorProperty()` + `VectorDistance()` LINQ support and `Microsoft.Extensions.AI` compatibility, where the EF provider contains no vector support at all and no project references `Extensions.AI`. `docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md` carried "✅" marks for `REPORT_V1.md`, `REPORT_V2.md`, `REPRODUCING.md`, `INTERPRETING_RESULTS.md` and a committed `tests/benchmarks/results/` directory, none of which exist; it now states per-artefact status. `docs/sharpcoredb-known-issues.md` — a 1.9.3-era external report — was verified item by item against 2.1: items 2-5 are fixed (`Load()` pads all seven per-column lists, `SingleFileTable` keeps a PK hash index, `ExecuteQuery` flushes pending batch writes, `SqlQueryValidator.NormalizeKey` strips `@`/`:`), item 6 (`INTEGER` → Int32) remains the default with the documented `UseSqliteIntegerAffinity` opt-in, and the non-existent `DurabilityMode.FlushToDisk` is corrected. Dead references were also repaired in `docs/INDEX.md` (two `storage/*_v1.7.0.md`, two `migration/FLUENTMIGRATOR_*_v1.7.0.md`), `docs/serialization/README.md` (two `SERIALIZATION_DOCUMENTATION_*.md`), `docs/graphrag/EF_CORE_DOCUMENTATION_INDEX.md` (three never-written files), `docs/analytics/README.md` (a `docs/compatibility/…` page), and the `Option.FromNullOrEmpty` fiction in `docs/FUNCTIONAL_NULL_SAFETY.md` was replaced with the real `Prelude.Optional` (null-only — semantic nulls are handled by the functional read path), alongside stale `v1.7.0` titles and a pointer to a `PERFORMANCE_VALIDATION_AND_ADVICE_CSHARP15_NET11.md` that was never created. Five new `DocumentationConsistencyTests` guards pin each class of defect (INDEX paths resolve; no TOML samples or "Supported Formats: TOML" in the server guide; no internal plan-cache API shown as public; every known-issue item annotated; every install-snippet package id real), each demonstrated red by planting the defect and green after removing it. + + - **The docs marketed a NativeAOT throughput win that measurement had already refuted, and the benchmark file carried no AOT arm at all.** Plan item S4 of the SQLite plan asked whether .NET 11's NativeAOT *shared dispatch helper* speeds up the per-row `IStorageEngine`/`IStorage` dispatch every cell pays for. Session 78 answered it by measurement: publishing the comparative harness with `-p:PublishAot=true` (native `win-x64` image, 14.023.680 B) and running it against its own JIT build — all 16 cells, fresh processes, identical options, same box — leaves AOT **at or behind the JIT on 12 of 16 cells** (one at exactly 1,00; the SQLite control on the same rig reads 0,98–1,01, which is what makes the 2–12 % READ/UPDATE deficits readable; the sole clear AOT lead is the legacy arm's DELETE at 1,25, at n=2 processes and a 0,92–1,38 pair spread), so S4 closed `REJECTED` with no AOT profile and no AOT recommendation. Three maintained documents still stated the un-measured expectation as a win — `docs/manual/performance.md` §7.10 listed *"AOT interface-dispatch improvements → faster interface-heavy storage paths under NativeAOT"* among the v2.1 items, `docs/performance/V2_PERFORMANCE_PLAN.md` §4.1 presented the helper as a speedup for `IStorage`/`IStorageEngine`/`ITable`, and the front page's AOT bullet read as a capability with no statement of what it is worth — and `docs/BENCHMARK_RESULTS.md`, the file users read for benchmark status, had no AOT number in it. All four now carry the measured result: the front page and the manual row state that AOT is a deployment option (image size, startup, no JIT) and not a speed lever here, `V2_PERFORMANCE_PLAN` §4.1 keeps the vendor expectation but marks it measured-and-not-transferring, and `BENCHMARK_RESULTS.md` gains a **NativeAOT arm** section with the per-cell AOT ÷ JIT table, the reproducible publish command, the two harness defects that nearly published the opposite verdict (the `--pk` warm-up ramp that made AOT look 1,85× on UPDATE, and the AOT profile failing on reflection-based JSON serialization, now pinned in the harness `.csproj`), and pointers to plan §S4, worklog session 78 and the archived runs. - **`SELECT version()` reported the pre-promotion product version *and* runtime to every client of the shipped 2.1.0 server.** The PostgreSQL handshake's `server_version` was a `const` literal, `SharpCoreDB 1.7.0 on .NET 10 (PostgreSQL protocol ...)`, so once 2.1.0 was published a client read a version two releases stale on a runtime the server no longer runs. It is now derived - the assembly's informational version plus `Environment.Version.Major` - so it tracks the build it ships in instead of being edited by hand, and it is pinned by `PgCatalogServiceTests.TryHandleCatalogQuery_Version_ReportsCurrentRuntimeAndProductVersion` (failed against the old literal with `Not found: "SharpCoreDB 2.1.0"`; passes after the change). Integration suite 19 / 0 failed. - **The 2.1.0 promotion left ".NET 10 / C# 14" as a *current requirement* in 70 files: front page (which also pinned the test count at 1,916 against 1,995 on this tree), `ROADMAP.md`, `CODING_STANDARDS.md`, around 12 shipped package READMEs (`NuGet.README.md` install snippets still pinning `2.1.0-rc.*`, the `Aspire` image tag `2.0.0.3` against `2.1.0`), around 14 `docs/**` requirement pages - including the load-bearing runtime requirement in `docs/server/INSTALLATION.md` - the publish-guide template, both `.runsettings` target frameworks, `PLATFORM_SUPPORT.md`, `SharpCoreDB.nuspec` (16 `net10.0` lib paths, tags, the `Microsoft.Data.Sqlite` floor), the `Examples/**` / `tests/**` guides and the decorative source labels - **589 `C# 14` labels across 260 `.cs` files and 26 stale `C# 14` claims in 11 `.md` files**, both measured in a follow-up sweep that verified the substitution line-for-line (in those 260 files every changed line *is* the label and nothing else, and the non-ASCII bytes on those lines are unchanged).** The promotion itself had refreshed the *build* metadata (props, packages, `.csproj`) and was validated by building, packing and publishing, so nothing in that route could see a claim that lives in prose. The sweep changed statements about the current line only: dated records (`STABILITY_REPORT.md`, the parked-line comment in `SharpCoreDB.csproj`, the `2.0.0.x` changelog sections, `ci-net10.yml`'s name), `#if NET10_0` regions and upstream package facts that merely mention a net10-era version are correct as written and were left alone - the boundary is recorded in `docs/net11/PROMOTION_TO_MAIN.md`, together with the note that the README text embedded in the already-published `2.1.0` packages only changes with the next version published. diff --git a/docs/FUNCTIONAL_NULL_SAFETY.md b/docs/FUNCTIONAL_NULL_SAFETY.md index 36d5b93f..9cdfe0f9 100644 --- a/docs/FUNCTIONAL_NULL_SAFETY.md +++ b/docs/FUNCTIONAL_NULL_SAFETY.md @@ -1,4 +1,4 @@ -# Functional Null Safety in SharpCoreDB v1.7.0 +# Functional Null Safety in SharpCoreDB 2.1 ## Why This Exists @@ -104,13 +104,18 @@ if (rows != null && rows.Count > 0) // Compiler ensures you handle absence. Zero exceptions possible. var email = (await fdb.GetByIdAsync(Tables.Users, userId)) .Map(u => u.Email) - .Bind(Option.FromNullOrEmpty) // built-in helper, no ternary - .IfNone(Defaults.NoEmail); // named constant, not a magic string + .Bind(Prelude.Optional) // built-in helper: null → None + .IfNone(Defaults.NoEmail); // caller-defined constant, not a magic string ``` > **Best practice:** Always use strongly-typed table references (e.g., `Tables.Users`) and named > constants (e.g., `Defaults.NoEmail`) instead of raw strings. `Option` composes cleanly with > either approach — but strong typing eliminates the "magic string" class of bugs entirely. +> +> **Which absences are covered where:** `Prelude.Optional` (and `Option`) covers `null` only. The +> functional read path additionally treats *semantic nulls* — empty string and the literal `"NULL"` — +> as absent when materializing rows (`FunctionalDb.cs`), so an empty `Email` surfaces as `None` +> without an extra check in caller code. #### Error Handling: `Fin` vs Try/Catch @@ -164,7 +169,7 @@ Database operations have inherently unpredictable data. Foreign keys reference d --- -## Functional SQL Syntax (v1.7.0) +## Functional SQL Syntax (SharpCoreDB 2.1) SharpCoreDB.Functional now supports a functional SQL layer that maps directly to `Option` semantics. diff --git a/docs/INDEX.md b/docs/INDEX.md index d02a09d5..e826b615 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -115,9 +115,9 @@ guide explaining when SharpCoreDB is fastest. - `scdb/PRODUCTION_GUIDE.md` - `serialization/README.md` - `serialization/SERIALIZATION_AND_STORAGE_GUIDE.md` -- `storage/QUICK_REFERENCE_v1.7.0.md` -- `storage/METADATA_IMPROVEMENTS_v1.7.0.md` -- `storage/SINGLE_FILE_SQL_LIMITATIONS.md` +- `storage/STORAGE_MODE_GUIDANCE.md` — which storage mode to pick, and why +- `storage/SINGLE_FILE_SQL_LIMITATIONS.md` — what `.scdb` single-file SQL does and does not support +- `serialization/SERIALIZATION_FAQ.md` — common serialization questions ## 6. Engine Implementation Notes @@ -160,10 +160,10 @@ guide explaining when SharpCoreDB is fastest. - `distributed/README.md` - `sync/README.md` -- `migration/README.md` -- `migration/MIGRATION_GUIDE.md` -- `migration/FLUENTMIGRATOR_EMBEDDED_MODE_v1.7.0.md` -- `migration/FLUENTMIGRATOR_SERVER_MODE_v1.7.0.md` +- `migration/README.md` — migration scenarios overview + the FluentMigrator FAQ (embedded, + in-process server and remote gRPC paths, syntax modes) +- `migration/MIGRATION_GUIDE.md` — step-by-step guides per source engine +- `../Examples/FluentMigrator/` — working FluentMigrator + SingleFileDatabase demo ## 9. Functional Programming & Null Safety @@ -206,3 +206,5 @@ guide explaining when SharpCoreDB is fastest. - Superseded duplicates are removed instead of linked alongside canonical files. - All documentation is in English only. + + diff --git a/docs/QUERY_PLAN_CACHE.md b/docs/QUERY_PLAN_CACHE.md index 15312242..097b8c59 100644 --- a/docs/QUERY_PLAN_CACHE.md +++ b/docs/QUERY_PLAN_CACHE.md @@ -26,9 +26,11 @@ db.ExecuteSQL("INSERT INTO users (name) VALUES (?)", db.ExecuteSQL("INSERT INTO users (name) VALUES (?)", new Dictionary { ["@p0"] = "Bob" }); -// Cache statistics -var (hits, misses, hitRate, count) = db.GetPlanCacheStats(); -// hits: 1, misses: 1, hitRate: 0.5 (50%), count: 1 +// Cache statistics — the in-engine accessor is internal +// Database.GetPlanCacheStats() → (Hits, Misses, HitRate, Count) +// It is visible to the SharpCoreDB.Tests / SharpCoreDB.Benchmarks assemblies only +// (InternalsVisibleTo), not to application code. See "Monitoring & Diagnostics" +// below for the public alternative. ``` --- @@ -273,8 +275,18 @@ Result: [Entry4, Entry1, Entry2] ### Get Cache Statistics +`Database.GetPlanCacheStats()` returns `(long Hits, long Misses, double HitRate, int Count)` but it is +**`internal`** (`src/SharpCoreDB/Database/Caching/Database.PlanCaching.cs`), so application code cannot +call it — the `SharpCoreDB.Tests` and `SharpCoreDB.Benchmarks` assemblies reach it through +`InternalsVisibleTo`, which is how the suite asserts cache behaviour. + +From outside the assembly, use the public surface of the cache class +(`src/SharpCoreDB/Services/QueryPlanCache.cs`): + ```csharp -var (hits, misses, hitRate, count) = db.GetPlanCacheStats(); +var cache = new QueryPlanCache(capacity: 2048); + +var (hits, misses, hitRate, count) = cache.GetStatistics(); Console.WriteLine($"Cache Hits: {hits}"); Console.WriteLine($"Cache Misses: {misses}"); @@ -293,11 +305,15 @@ Console.WriteLine($"Cached Plans: {count}"); ### Clear Cache Manually +`Database.ClearPlanCache()` is **`internal`** as well — the engine calls it during `Dispose` and on +explicit reset, so you rarely need it at all. With your own cache instance the call is public: + ```csharp -// Not typically needed (automatic disposal) -db.ClearPlanCache(); +cache.Clear(); ``` +Simply disposing the database clears the engine's own plan cache. + --- ## Best Practices @@ -373,7 +389,8 @@ db.ClearPlanCache(); **Solutions:** 1. Reduce `CompiledPlanCacheCapacity` (default: 2048) 2. Disable caching for specific operations if needed -3. Monitor cache size with `GetPlanCacheStats().Count` +3. Monitor cache size with `QueryPlanCache.GetStatistics().Count` — the engine's own + `GetPlanCacheStats()` is `internal` and not callable from application code ### Issue: Thread Contention diff --git a/docs/README.md b/docs/README.md index 15c9ca56..3f122506 100644 --- a/docs/README.md +++ b/docs/README.md @@ -51,3 +51,5 @@ Obsolete phase-status, kickoff, duplicate, and superseded planning docs are remo + + diff --git a/docs/Vectors/TECHNICAL_SPEC.md b/docs/Vectors/TECHNICAL_SPEC.md index bf807965..204c32ef 100644 --- a/docs/Vectors/TECHNICAL_SPEC.md +++ b/docs/Vectors/TECHNICAL_SPEC.md @@ -749,9 +749,26 @@ When SharpCoreDB scales to RDBMS mode: ### Integration Points -- `Microsoft.Extensions.AI.IEmbeddingGenerator>` compatibility -- `SharpCoreDB.EntityFrameworkCore` — `HasVectorProperty()` + `VectorDistance()` LINQ support -- `SharpCoreDB.Data.Provider` — ADO.NET `DbType` mapping for vectors +> **Corrected 2026-10-10 (verified against 2.1.0).** The three bullets that used to stand here were +> aspirational, not shipped: `SharpCoreDB.EntityFrameworkCore` contains **no vector support at all** (a +> search of that project for `vector`/`embedding` returns nothing — no `HasVectorProperty()`, no +> `VectorDistance()`, no vector type mapping), and no project references `Microsoft.Extensions.AI` +> (`Directory.Packages.props` has no `Extensions.AI` entry), so an `IEmbeddingGenerator` integration is +> not part of the package. + +What ships today: + +- **`SharpCoreDB.VectorSearch`** (optional package) — HNSW / DiskANN / Flat indexes, quantization, + distance metrics and hybrid fusion; see [`README.md`](README.md) and + [`DISKANN_SQL_DDL.md`](DISKANN_SQL_DDL.md). +- **SQL DDL for vector indexes** (`CREATE INDEX … USING HNSW|DISKANN …`), parsed in + `Services/SqlParser.DDL.cs`. Ranking and distance work happen through the package's C# search API — + there is no SQL `VECTOR_DISTANCE()` / `COSINE_DISTANCE()` function (none exists in the parser or the + expression evaluator). +- **`SharpCoreDB.Data.Provider`** — ADO.NET access; vector values travel as their stored + representation. +- **Embeddings** are produced by whatever generator the application chooses; SharpCoreDB consumes the + resulting float vectors and bundles no generator abstraction. --- diff --git a/docs/analytics/README.md b/docs/analytics/README.md index b9cfb699..f9b246b6 100644 --- a/docs/analytics/README.md +++ b/docs/analytics/README.md @@ -354,7 +354,9 @@ var filtered = await database.QueryAsync( ); ``` -Current support and known gaps are tracked in `docs/compatibility/SQLITE_POSTGRESQL_AGGREGATE_SYNTAX_v1.9.0.md`. +Support for aggregate `FILTER` clauses and the known gaps against the SQLite/PostgreSQL aggregate +syntax are covered in [`../manual/query.md`](../manual/query.md) (the SQL dialect and aggregates +chapter). There is no separate compatibility-tracking page in this repository. ### HAVING Filters grouped results (WHERE applies before GROUP BY, HAVING after). diff --git a/docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md b/docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md index 4b619055..5ceb1d9f 100644 --- a/docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md +++ b/docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md @@ -5,6 +5,12 @@ **Status:** Ready for Implementation (M1 Locked) **Definition of Done:** All items below must be completed and verified +> **Audit note (2026-10-10):** this is the original plan of record. The harness, runner scripts, +> methodology and scenario documents were produced; the two report files, the committed raw data, +> `REPRODUCING.md` and `INTERPRETING_RESULTS.md` named below were **not**. Every "✅" this file used to +> carry is answered with its real artefact in [Evidence status](#evidence-status-audited-2026-10-10-against-the-210-tree) +> at the end — the unchecked boxes above are therefore accurate as written. + --- ## Phase 1: Benchmark v1 (Weeks 3-6, Milestone M2) @@ -154,40 +160,37 @@ --- -## Evidence Submission (for Closure) - -Issue #56 is **done** when: - -1. ✅ Pull request merged with all benchmark code and results -2. ✅ Raw data published: - - `tests/benchmarks/results/` directory with all runs - - `raw-data.json` for each run - - CSV exports available -3. ✅ Reports published: - - Report v1 (`docs/benchmarks/REPORT_V1.md`) - - Report v2 (`docs/benchmarks/REPORT_V2.md`) -4. ✅ Documentation published: - - Methodology (`BENCHMARK_METHOD.md`) - - Scenarios (`BENCHMARK_SCENARIOS_FINAL.md`) - - Reproducibility guide (`REPRODUCING.md`) - - Interpretation guide (`INTERPRETING_RESULTS.md`) -5. ✅ Scripts available: - - Benchmark harness (`tests/benchmarks/SharpCoreDB.Benchmarks.csproj`) - - Runner scripts (`run-all.ps1`, `run-all.sh`) - - Report generator -6. ✅ CI integration verified (results published automatically) - -**Closure Comment Template:** +## Evidence status (audited 2026-10-10 against the 2.1.0 tree) + +The ✅ markers this section used to carry asserted artefacts that were never produced. Here is what each +line actually resolves to: + +| Evidence | Status | Where it really is | +|----------|--------|--------------------| +| Benchmark code in the repository | ✅ produced | `tests/SharpCoreDB.Benchmarks/` and `tests/SharpCoreDB.Benchmarks.Comparative/` | +| Runner scripts | ✅ produced | `tests/benchmarks/run-all.ps1`, `run-all.sh`, `BenchmarkConfig.json` | +| Raw data committed under `tests/benchmarks/results/` | ❌ not in the repository | that directory does not exist; run output is local and is not committed | +| `raw-data.json` per run, CSV exports | ❌ not published | produced by a run, never checked in | +| Report v1 — `docs/benchmarks/REPORT_V1.md` | ❌ never written | use `SHARPCOREDB_COMPARATIVE_BENCHMARKS.md`, `ZVEC_BENCHMARKS_COMPLETE.md` | +| Report v2 — `docs/benchmarks/REPORT_V2.md` | ❌ never written | later results live in `V198_V20_V21_PERFORMANCE_COMPARISON.md`, `SQLITE_LITEDB_COMPARISON_2026-09-16.md` | +| Methodology — `BENCHMARK_METHOD.md` | ✅ produced | `docs/benchmarks/BENCHMARK_METHOD.md` | +| Scenarios — `BENCHMARK_SCENARIOS_FINAL.md` | ✅ produced | `docs/benchmarks/BENCHMARK_SCENARIOS_FINAL.md` | +| Reproducibility guide — `REPRODUCING.md` | ❌ never written | the run steps are in `tests/benchmarks/README.md` | +| Interpretation guide — `INTERPRETING_RESULTS.md` | ❌ never written | interpretation notes live inside the report documents | +| CI publishes results automatically | ❌ not implemented | `benchmarks.yml` runs the suite; it does not attach results to a release | + +**Closure Comment Template** (keeping only claims that hold): + ```markdown -## Issue #56 Completed ✅ +## Issue #56 — benchmark suite published **Benchmark Suite: SharpCoreDB vs BLite vs Zvec** -- Scenarios: 9 total (4 BLite, 5 Zvec) -- Raw data: Available in `tests/benchmarks/` -- Report v1: `REPORT_V1.md` (attach the published link when closing the issue) -- Report v2: `REPORT_V2.md` (attach the published link when closing the issue) -- Reproducibility: Scripts available, runnable +- Scenarios: 9 total (4 BLite, 5 Zvec) — definitions in `BENCHMARK_SCENARIOS_FINAL.md` +- Harness: `tests/SharpCoreDB.Benchmarks/`, runnable via `tests/benchmarks/run-all.ps1` / `run-all.sh` +- Reports: `SHARPCOREDB_COMPARATIVE_BENCHMARKS.md`, `ZVEC_BENCHMARKS_COMPLETE.md` +- Methodology: `BENCHMARK_METHOD.md` +- Raw per-run JSON: not committed — reproduce locally with the runner scripts **Key Findings:** - BLite: [summary] @@ -197,9 +200,6 @@ Issue #56 is **done** when: **Links:** - PR: `#XXX` (fill in the pull-request number) - Documentation: [benchmarks README](../../tests/benchmarks/README.md) -- Raw data: `tests/benchmarks/` - -Benchmark suite is now public and reproducible. Community can validate. ``` --- diff --git a/docs/graphrag/EF_CORE_DOCUMENTATION_INDEX.md b/docs/graphrag/EF_CORE_DOCUMENTATION_INDEX.md index 2c8879e2..d209d2f5 100644 --- a/docs/graphrag/EF_CORE_DOCUMENTATION_INDEX.md +++ b/docs/graphrag/EF_CORE_DOCUMENTATION_INDEX.md @@ -1,7 +1,7 @@ # GraphRAG EF Core Integration - Documentation Index **Status:** ✅ Phase 2 complete (Phase 3 prototype) -**Last Updated:** 2025-02-15 +**Last Updated:** 2026-10-10 (links audited against the 2.1.0 tree) **Test Status:** Run the MTP test host to validate locally — `dotnet test` is not supported in this repo (see `docs/CONTRIBUTING.md`) --- @@ -11,11 +11,14 @@ | Document | Purpose | Audience | |----------|---------|----------| | [LINQ_API_GUIDE.md](#linq-api-guide) | LINQ API reference | Developers | -| [EF_CORE_COMPLETE_GUIDE.md](#ef-core-complete-guide) | Usage guide | Developers | -| [EF_CORE_INTEGRATION_SUMMARY.md](#ef-core-integration-summary) | Architecture summary | Tech Leads | +| [EF_CORE_COMPLETE_GUIDE.md](#ef-core-complete-guide) | Usage guide + architecture | Developers, Tech Leads | | [EF_CORE_TEST_DOCUMENTATION.md](#ef-core-test-documentation) | Test suite documentation | QA Engineers | -| [TEST_EXECUTION_REPORT.md](#test-execution-report) | Test execution template | Project Managers | -| [COMPLETE_DELIVERY_SUMMARY.md](#delivery-summary) | Current status summary | Stakeholders | +| [`00_START_HERE.md`](00_START_HERE.md) | Entry point and current status | Everyone | + +> **Audit note (2026-10-10):** this index used to list three documents that were never written — +> `EF_CORE_INTEGRATION_SUMMARY.md`, `TEST_EXECUTION_REPORT.md` and `COMPLETE_DELIVERY_SUMMARY.md`. +> Their content lives in the guides below (architecture in `EF_CORE_COMPLETE_GUIDE.md`, the test +> inventory in `EF_CORE_TEST_DOCUMENTATION.md`, current status in `00_START_HERE.md` and `README.md`). --- @@ -46,7 +49,7 @@ --- ## EF Core Integration Summary -**File:** `docs/graphrag/EF_CORE_INTEGRATION_SUMMARY.md` +**Where:** the architecture section of `docs/graphrag/EF_CORE_COMPLETE_GUIDE.md` **Contents:** - Implementation overview @@ -65,17 +68,8 @@ --- -## Test Execution Report -**File:** `docs/graphrag/TEST_EXECUTION_REPORT.md` - -**Contents:** -- Template for capturing test results -- Execution notes - ---- - -## Delivery Summary -**File:** `docs/graphrag/COMPLETE_DELIVERY_SUMMARY.md` +## Delivery Status +**Where:** [`00_START_HERE.md`](00_START_HERE.md) and [`README.md`](README.md) **Contents:** - Current implementation status diff --git a/docs/internals/OPTIMIZER_GUIDE.md b/docs/internals/OPTIMIZER_GUIDE.md index a7e1f950..7f1264f9 100644 --- a/docs/internals/OPTIMIZER_GUIDE.md +++ b/docs/internals/OPTIMIZER_GUIDE.md @@ -1,15 +1,23 @@ -# Query Optimizer - Complete Implementation Guide +# Query Optimizer - Implementation Guide + +> **Verified against SharpCoreDB 2.1.0.** The optimizer on the engine's query path is +> `SharpCoreDB.Optimization.QueryOptimizer` (`src/SharpCoreDB/Optimization/QueryOptimizer.cs`), +> constructed with `Optimization.CostEstimator` from `Services/QueryCompiler.cs`. The `Planning/` +> types (`QueryOptimizer`, `CardinalityEstimator`, `PredicatePushdown`) and `Query/CostBasedOptimizer.cs` +> are real, compiling components that **no shipped call site constructs** — building blocks, not the +> active path. Sections that used to describe components which do not exist at all +> (`SubqueryOptimizer`, `JoinReorderer`, `OptimizerCache`, `QueryOptimizerExtensions`, +> `OptimizerIntegration`, `InvalidateOptimizationCache()`) are corrected below and marked as such. ## Overview -A lightweight, cost-based query optimizer for SharpCoreDB with logical/physical plan separation, predicate pushdown, join reordering, and subquery elimination. +A lightweight, cost-based query optimizer for SharpCoreDB with logical/physical plan separation. **Key Characteristics**: -- ✅ Cost-based optimization -- ✅ Struct-based plan nodes (zero allocation) -- ✅ No LINQ, no async -- ✅ Seamless QueryCache integration -- ✅ Reusable physical plans +- ✅ Cost-based plan selection +- ✅ `readonly struct` plan nodes, no LINQ and no async on the plan path +- ✅ Plans cached per compiled query (`Database/Caching/Database.PlanCaching.cs`) +- ⚠️ Join reordering and subquery elimination are **not** implemented on this path — see §4 and §6 ## Architecture @@ -59,426 +67,282 @@ QueryOptimizer.Core ## Components -### 1. QueryOptimizer.Core +### 1. `Optimization.QueryOptimizer` -**Main Entry Point**: `QueryOptimizer.Optimize(SelectNode query)` +**Main entry point**: `QueryOptimizer.Optimize(SelectNode query)` → `PhysicalPlan` +**Constructed as**: `new QueryOptimizer(new CostEstimator(stats))` in `Services/QueryCompiler.cs` -**Output**: `PhysicalPlan` (execution steps in order) +**Key types** (all in `src/SharpCoreDB/Optimization/QueryOptimizer.cs`): +- `LogicalPlan` (`readonly struct`) — `Query`, `PredicatesPushed`, `JoinsReordered`, + `SubqueriesEliminated`, `EstimatedRows`, `Cacheable` +- `PhysicalPlan` (`readonly struct`) — `Logical`, `Steps` (`PhysicalStep[]`), `EstimatedCost` +- `PhysicalStep` — one execution step: `Type`, `TableName`, `Alias` +- `PhysicalStepType` — `TableScan`, `Filter`, `Aggregate`, `Sort`, `Project` -**Key Classes**: -- `LogicalPlan`: Logical operators organized by phase -- `PhysicalPlan`: Ordered execution steps -- `PhysicalStep`: Individual execution operation -- `LogicalOperator` hierarchy: Scan, Join, Filter, Aggregate, Sort +**What it actually does**: walks the parsed `SelectNode` and emits one physical step per clause it +recognises (scan, filter, aggregate, sort, project), pricing the plan through `CostEstimator`. Two +`LogicalPlan` flags describe the query's *shape*, not the result of a rewrite: +`JoinsReordered = hasJoins` and `SubqueriesEliminated = !hasSubquery` (the assignments in `Optimize` +carry the comment "placeholder flag" / "trivially eliminated"). **Example**: ```csharp -var optimizer = new QueryOptimizer( - costEstimator, - predicatePushdown, - subqueryOptimizer, - joinReorderer); - +var optimizer = new QueryOptimizer(new CostEstimator(tableStats)); var physicalPlan = optimizer.Optimize(selectNode); -// physicalPlan.Steps contains: -// 1. TableScan(A) -// 2. Filter(id > 5) -// 3. HashJoin(B on A.id = B.id) -// 4. Aggregate(GROUP BY dept) -// 5. Sort(by salary DESC) -// 6. LimitOffset(10, 0) -// 7. Project(id, name, salary) +Console.WriteLine($"cost={physicalPlan.EstimatedCost} steps={physicalPlan.Steps.Length}"); +foreach (var step in physicalPlan.Steps) +{ + Console.WriteLine($"{step.Type}: {step.TableName ?? "N/A"}"); +} +// Steps follow the statement's clause order, e.g. +// TableScan(orders) → Filter(id > 5) → Aggregate → Sort → Project ``` -### 2. CostEstimator +### 2. `Optimization.CostEstimator` -**Purpose**: Estimates execution cost and cardinality +**Purpose**: prices a plan and predicts its output size. -**Cost Model**: -``` -TableScan: 1.0 × row_count -IndexScan: 1.0 + log₂(row_count) -HashJoin: left_cost + right_cost + build(smaller) + probe(larger) -NestedLoopJoin: left_cost + (right_cost × left_rows) -Filter: input_cost × selectivity (default 0.1 = 10%) -Aggregate: input_cost × 0.1 (groups ≈ 10% of input rows) -Sort: input_cost + n × log₂(n) -``` +**Real API** (`src/SharpCoreDB/Optimization/CostEstimator.cs`): -**Cardinality Estimation**: -``` -TableScan: exact row count from statistics -Filter: estimated_rows × selectivity -Join(INNER): (left_rows × right_rows) × 0.5 / max(distinct_L, distinct_R) -Join(LEFT): left_rows -Join(FULL): left_rows + right_rows -``` - -**Usage**: ```csharp -var estimator = new CostEstimator(tableStatistics); +var estimator = new CostEstimator(statistics); // IReadOnlyDictionary -var scanCost = estimator.EstimateCost( - new ScanOperator { TableName = "orders" }); +EstimatedCost scan = estimator.EstimateScanCost("orders"); +EstimatedCost filter = estimator.EstimateFilterCost(inputCost); +EstimatedCost join = estimator.EstimateJoinCost(leftCost, rightCost, /* … */); +EstimatedCost agg = estimator.EstimateAggregateCost(inputCost); +EstimatedCost sort = estimator.EstimateSortCost(inputCost); -Console.WriteLine($"Cost: {scanCost.Cost}, Rows: {scanCost.OutputRows}"); -// Output: Cost: 10000.0, Rows: 10000 +Console.WriteLine($"cost={scan.Cost} rows={scan.OutputRows}"); // EstimatedCost { double Cost; long OutputRows; } ``` -### 3. PredicatePushdown +The coefficients that turn each operator into `Cost`/`OutputRows` are in the source file — read them +there rather than trusting a copied table; they are the numbers the planner compares. -**Purpose**: Move filters closer to data sources +### 3. Predicate pushdown -**Transformations**: +**Purpose**: apply filters as close to the storage layer as possible, and evaluate the checklist of +predicates most-selective-first. -``` -BEFORE: AFTER: -Scan(A) → Join(B) → Filter(a.x=5) Filter(a.x=5) → Scan(A) → Join(B) -``` +> **Correction:** this is a *building block*, not a wired-in rewrite. `Planning/PredicatePushdown.cs` +> exposes static helpers and is referenced by no shipped call site; on the compiled path the optimizer +> only records the *intent* — `LogicalPlan.PredicatesPushed = hasWhere || hasHaving`, commented +> "pushdown planned" in `Optimization/QueryOptimizer.cs`. The compiled plan carries the actual filter +> delegates (`whereFilter`, `whereFilterIndexed`) built in `Services/QueryCompiler.cs`. -**AND-Predicate Splitting**: -``` -WHERE a.x = 5 AND a.y > 10 AND b.z = 'X' -→ Push [a.x = 5, a.y > 10] to table A -→ Keep [b.z = 'X'] for join filter -``` +**Real API** (`src/SharpCoreDB/Planning/PredicatePushdown.cs`, all `static`): -**Benefits**: -- Reduces rows processed by joins -- Fewer comparisons in nested loops -- Better memory efficiency - -**Example**: ```csharp -var pushdown = new PredicatePushdown(); -logicalPlan = pushdown.PushdownPredicates(logicalPlan); +// Order predicates most-selective-first before evaluating them. +List ordered = PredicatePushdown.OptimizePredicateOrder(predicates); -// Original: Scan(orders) → Filter(amount > 1000 AND status = 'shipped') -// Optimized: Filter(amount > 1000 AND status = 'shipped') → Scan(orders) -``` +// Can this predicate be evaluated at this level (are its columns available)? +bool pushable = PredicatePushdown.CanPushDown(predicate, availableColumns); -### 4. SubqueryOptimizer +// Evaluate pushed predicates against columnar data and return the surviving row indices. +// Integrates with the SIMD filter path (column encoding aware). +int[] rows = PredicatePushdown.ExecutePushedPredicates( + predicates, totalRows, getColumnData, getColumnEncoding); -**Purpose**: Convert subqueries to joins (better optimization) +// Rewrite a predicate for columnar access; combine AND-connected predicates. +PredicateInfo rewritten = PredicatePushdown.RewriteForColumnar(predicate, /* … */); +List combined = PredicatePushdown.CombinePredicates(predicates); +``` -**Conversions**: +The `PushdownPredicates(logicalPlan)` / `new PredicatePushdown()` shape shown in earlier revisions of +this page does not exist. -**EXISTS → Semi-Join**: -```sql -WHERE EXISTS (SELECT 1 FROM B WHERE B.id = A.id) -→ INNER JOIN B ON B.id = A.id (with DISTINCT) -``` +### 4. Subqueries — classified, planned, executed and cached (not rewritten) -**NOT EXISTS → Anti-Join**: -```sql -WHERE NOT EXISTS (SELECT 1 FROM B WHERE B.id = A.id) -→ LEFT JOIN B ON B.id = A.id WHERE B.id IS NULL -``` +> **Correction:** there is **no** `SubqueryOptimizer` and no `EliminateSubqueries(...)`; the types +> `SemiJoin` and `AntiJoin` do not exist anywhere in the source. SharpCoreDB does **not** rewrite +> `EXISTS` / `NOT EXISTS` / `IN (SELECT)` into semi-joins or anti-joins. `LogicalPlan.SubqueriesEliminated` +> is `!hasSubquery` — a shape flag, commented "non-subquery sources are trivially eliminated". -**IN (SELECT) → Hash Semi-Join**: -```sql -WHERE A.id IN (SELECT B.id FROM B WHERE B.active = 1) -→ INNER JOIN B ON A.id = B.id WHERE B.active = 1 (DISTINCT on A.id) -``` +What does exist (`src/SharpCoreDB/Execution/`): -**Scalar Subquery Elimination**: -```csharp -// Scalar subqueries detected by SubqueryClassifier -// Non-correlated: Cached by SubqueryCache (see Execution module) -// Correlated: Can be inlined into join filters -``` +| Type | Role | +|------|------| +| `SubqueryClassifier` | classifies the subquery (shape/type) | +| `SubqueryPlanner` | turns it into an execution plan | +| `SubqueryExecutor` | executes it | +| `SubqueryCache` | caches the **result**, with per-table invalidation | -**Example**: ```csharp -var subqueryOpt = new SubqueryOptimizer(); -logicalPlan = subqueryOpt.EliminateSubqueries(logicalPlan); - -// BEFORE: Scan(orders) → Filter(EXISTS subquery) -// AFTER: Scan(orders) → SemiJoin(customers, on order.cust_id = customer.id) +var cached = cache.GetOrExecute(/* … */); // result cache +cache.Invalidate("orders"); // drop entries for one table +cache.Clear(); +var (hits, misses, hitRate, count) = cache.GetStatistics(); ``` -### 5. JoinReorderer +Non-correlated subqueries therefore benefit from result caching rather than from a plan rewrite; +correlated ones are executed as the plan dictates. -**Purpose**: Reorder joins to minimize intermediate result size +### 5. Join ordering -**Strategy**: Small-to-large ordering -``` -Query: A ⋈ B ⋈ C -Stats: |A| = 10M, |B| = 1K, |C| = 5M - -Bad Order: (A ⋈ B) ⋈ C = (10M × 1K) ⋈ C = 10B rows ⋈ 5M (huge!) -Good Order: (B ⋈ A) ⋈ C = (1K × 10M) ⋈ C = 10M rows ⋈ 5M (manageable) -``` +> **Correction:** there is no `JoinReorderer` type and no `ReorderJoins(...)` method. The compiled +> path does **not** reorder joins: `LogicalPlan.JoinsReordered = hasJoins`, commented "placeholder +> flag". Joins execute through `Execution/JoinExecutor.cs`. -**Cost Estimation**: -- Tries different orderings -- Estimates total cost including intermediate results -- Chooses minimum cost order +`OptimizeJoinOrder(...)` does exist, on two components that no shipped call site constructs: -**Limitations**: -- Current: Greedy ordering (fast) -- Future: Full Selinger DP (optimal but slower) - -**Example**: ```csharp -var reorderer = new JoinReorderer(); -logicalPlan = reorderer.ReorderJoins(logicalPlan, costEstimator); +// Planning/QueryOptimizer.cs — statistics-driven, accepts TableInfo{Name, RowCount} +List order = plannerOptimizer.OptimizeJoinOrder(tables, joinConditions); -// BEFORE: orders ⋈ customers ⋈ products -// AFTER: customers ⋈ orders ⋈ products (customers likely smallest) +// Query/CostBasedOptimizer.cs — table-name list + registered TableStatistics +List order2 = costBasedOptimizer.OptimizeJoinOrder(tableNames, joins); ``` -### 6. QueryOptimizerExtensions +**Execution reality** (`src/SharpCoreDB/Execution/JoinExecutor.cs`): the join executor selects between a +hash join and a nested-loop join on row count, but the hash-join threshold is currently +`HashJoinThreshold = int.MaxValue` with the comment *"Disable hash join until key hashing is +implemented"* — so in practice joins run as **nested-loop** joins today. + +### 6. Plan caching (there is no `OptimizerCache`) -**Purpose**: Integrate optimizer with QueryCache +> **Correction:** `OptimizerCache`, `QueryOptimizerExtensions`, `OptimizerIntegration.GetOrOptimize(...)` +> and `InvalidateOptimizationCache()` do not exist. Plan caching is done by the compiled-plan cache. -**Caching Strategy**: ``` -QueryCache OptimizerCache -┌──────────────┐ ┌──────────────┐ -│ Parsed AST │ │ Physical Plan │ -│ + Parts │ │ + Cost est. │ -└──────────────┘ └──────────────┘ - ↑ ↑ - └──── Same key ──────┘ +Database.ExecuteSQL / ExecuteQuery + │ + ▼ +Database/Caching/Database.PlanCaching.cs ← GetOrAddPlan / TryGetCachedPlan / + │ NormalizeSqlForCaching / BuildCacheKey + ▼ +Services/QueryPlanCache.cs ← ConcurrentDictionary + │ (+ LRU eviction, capacity configurable) + ▼ +CacheEntry { CachedPlan, CompiledPlan, OptimizedPlan, OptimizedCost, CachedAtUtc, AccessCount } ``` -**Cache Invalidation**: -- On table statistics change -- On INSERT/UPDATE/DELETE (table modifications) -- Manual clear via `InvalidateOptimizationCache()` +**Configuration** (`DatabaseConfig`, defaults shown): -**Integration Points**: ```csharp -// In QueryCache -public void GetOrAdd(string sql, SelectNode query) +var config = new DatabaseConfig { - // Get parsed AST from cache (existing) - var parsed = cache.GetParsed(sql); - - // Get optimized plan from optimizer cache (new) - var physical = OptimizerIntegration.GetOrOptimize( - sql, parsed, costEstimator, ...); - - // Execute physical plan - return ExecutePhysicalPlan(physical); -} + EnableCompiledPlanCache = true, // default + CompiledPlanCacheCapacity = 2048, // default; LRU eviction past this + NormalizeSqlForPlanCache = true, // collapse whitespace so equal queries share an entry +}; ``` -## Performance Characteristics +**Cache key** = normalised SQL + parameter shape + command type (`SqlCommandType`), so an `INSERT` and +a `SELECT` over the same text never share an entry. -### Optimization Time +**Diagnostics and clearing are `internal`** — `Database.GetPlanCacheStats()` (`(Hits, Misses, HitRate, +Count)`) and `Database.ClearPlanCache()` are visible to the `SharpCoreDB.Tests` / +`SharpCoreDB.Benchmarks` assemblies through `InternalsVisibleTo`, but **not callable from application +code**. Applications that host their own `Services.QueryPlanCache` instance can use its public +`GetStatistics()` / `Clear()`. -``` -Query Component Time Complexity -───────────────────────────────────── -Logical Planning O(n) where n = operators -Predicate Pushdown O(n) where n = predicates -Subquery Elimination O(m) where m = subqueries -Join Reordering O(k! / 2) worst case, O(k) greedy where k = joins -Physical Planning O(n) where n = logical operators +## Cost model characteristics -Total: O(n + k! / 2) [bounded by join count k] +The compiled path's planner is deliberately light: `BuildLogicalPlan` reads the AST's *shape* (a few +flags and `EstimatedRows` from `CostEstimator.EstimateScanCost`), `EstimateCost` accumulates a heuristic +cost for each recognised clause, and `BuildPhysicalSteps` emits the step array. There is no +combinatorial search — no join-order enumeration and no subquery rewrite — so planning here is +O(clauses). -Typical: <1ms for 5-10 table queries -``` +> Earlier revisions of this page published an "Execution Impact" table (2-5× pushdown, 5-20× join +> reordering, 50-1000× combined). Those multipliers were never measured against this code path and have +> been removed. For measured, reproducible numbers — including the planner's contribution — +> use the benchmark harness and the reports indexed in +> [`docs/benchmarks/BENCHMARK_METHOD.md`](../benchmarks/BENCHMARK_METHOD.md) and +> [`docs/manual/performance.md`](../manual/performance.md). -### Memory Usage +## Usage examples -``` -Logical Plan: O(n) nodes -Physical Plan: O(n) steps -Cost Estimates: O(k) join combinations -Total: ~KB for typical queries -``` - -### Execution Impact - -``` -Optimization Result Execution Speedup -────────────────────────────────────────────── -Predicate Pushdown 2-5x (fewer rows) -Join Reordering (bad→good order) 5-20x (less memory) -Subquery→Join Conversion 10-100x (cache reuse) -Combined 50-1000x possible -``` - -## Usage Examples - -### Basic Optimization +### Planning a statement ```csharp -// Setup components -var costEstimator = new CostEstimator(tableStatistics); -var predicatePushdown = new PredicatePushdown(); -var subqueryOptimizer = new SubqueryOptimizer(); -var joinReorderer = new JoinReorderer(); - -// Create optimizer -var optimizer = new QueryOptimizer( - costEstimator, - predicatePushdown, - subqueryOptimizer, - joinReorderer); - -// Optimize parsed AST -var physicalPlan = optimizer.Optimize(selectNode); +var estimator = new CostEstimator(tableStatistics); // IReadOnlyDictionary +var optimizer = new QueryOptimizer(estimator); -// Inspect plan -foreach (var step in physicalPlan.Steps) +PhysicalPlan plan = optimizer.Optimize(selectNode); +double cost = plan.EstimatedCost; +LogicalPlan logical = plan.Logical; // shape flags + EstimatedRows + +foreach (var step in plan.Steps) { Console.WriteLine($"{step.Type}: {step.TableName ?? "N/A"}"); } ``` -### Cached Optimization - -```csharp -// First execution: optimize -var plan1 = OptimizerIntegration.GetOrOptimize( - "SELECT * FROM orders WHERE id > 100", - selectNode, costEstimator, ...); - -// Second execution: retrieve from cache -var plan2 = OptimizerIntegration.GetOrOptimize( - "SELECT * FROM orders WHERE id > 100", - selectNode, costEstimator, ...); - -// plan1 and plan2 are identical, cache hit! -``` - -### Cost Estimation +### Estimating clause costs ```csharp var estimator = new CostEstimator(statistics); -// Estimate scan cost -var scanOp = new ScanOperator { TableName = "orders" }; -var scanCost = estimator.EstimateCost(scanOp); +EstimatedCost scan = estimator.EstimateScanCost("orders"); +EstimatedCost filtered = estimator.EstimateFilterCost(scan); -// Estimate join cost -var joinOp = new JoinOperator -{ - JoinType = JoinNode.JoinType.Inner, - Left = scanOp, - Right = new ScanOperator { TableName = "customers" } -}; -var joinCost = estimator.EstimateCost(joinOp); - -Console.WriteLine($"Scan: {scanCost.Cost}, Join: {joinCost.Cost}"); +Console.WriteLine($"scan cost={scan.Cost} rows={scan.OutputRows} | " + + $"filter cost={filtered.Cost} rows={filtered.OutputRows}"); ``` -### Predicate Analysis +### Where plan caching happens -```csharp -var columns = PredicateAnalyzer.GetReferencedColumns(whereCondition); -// Returns: {"orders.id", "orders.status", "customers.country"} +Callers do not wire caching up. `Database.ExecuteSQL` / `ExecuteQuery` go through +`Database.PlanCaching.cs`, which normalises the SQL and asks `Services.QueryPlanCache` for an existing +entry before compiling a new plan. Configuration and the (internal) diagnostics are in §6. -var isSimple = PredicateAnalyzer.IsSimpleEquality( - condition, out var col, out var value); -// Returns: true, col = "orders.id", value = 5 -``` - -## Integration with Execution - -### PhysicalPlan → Execution Steps +## From plan to execution -Physical plan maps directly to execution: +`PhysicalPlan.Steps` describes the statement's clause order; it is not a second execution engine: ``` -PhysicalStep(TableScan, "orders") - ↓ (executed by TableScanExecutor) -Dictionary[] rows from orders - -PhysicalStep(Filter, "id > 100") - ↓ (executed by FilterExecutor) -Filtered rows - -PhysicalStep(HashJoin, customers) - ↓ (executed by JoinExecutor) -Joined rows - -... etc -``` - -### Executor Integration - -```csharp -// In QueryExecutor -public List> Execute(PhysicalPlan plan) -{ - var results = ExecuteSource(); // First step - - foreach (var step in plan.Steps.Skip(1)) - { - results = step.Type switch - { - PhysicalStepType.Filter => ExecuteFilter(step, results), - PhysicalStepType.HashJoin => ExecuteJoin(step, results), - PhysicalStepType.Aggregate => ExecuteAggregate(step, results), - PhysicalStepType.Sort => ExecuteSort(step, results), - ... etc - }; - } - - return results; -} +PhysicalStep(TableScan, "orders") → rows from the table +PhysicalStep(Filter, "id > 100") → predicate evaluation (compiled whereFilter / §3 helper) +PhysicalStep(Aggregate) → grouping and aggregation +PhysicalStep(Sort) → ORDER BY +PhysicalStep(Project) → projection / selected columns ``` -## Optimization Rules - -### When is Predicate Pushdown Applied? - -✅ **YES**: -- Simple equality: `a.id = 5` → push to scan -- AND-connected: `a.id = 5 AND a.status = 'X'` → push both -- Type-safe: Can determine column belongs to table - -❌ **NO**: -- Functions: `UPPER(a.name) = 'JOHN'` (can't push) -- Multi-table: `a.id = b.id` (join predicate, keep at join) -- Correlated subqueries: (keep at join level) - -### When is Subquery→Join Conversion Applied? - -✅ **YES**: -- EXISTS with simple join condition -- NOT EXISTS with simple join condition -- IN (SELECT col) with single column result - -❌ **NO**: -- Scalar subqueries (cached instead) -- Correlated subqueries without join equality -- Subqueries with aggregates (complex) +Executors that exist under `src/SharpCoreDB/Execution/`: `JoinExecutor` (with +`JoinExecutor.Diagnostics` and `JoinConditionEvaluator`) for joins, `AggregationOptimizer` for +aggregates, and the subquery trio from §4. **`FilterExecutor` and `TableScanExecutor` are not types in +this codebase** — earlier revisions named them — and `PhysicalStepType` has no `HashJoin` member, because +joins are not a plan-step type here. -### Join Order Heuristics +## Optimization rules — what is actually applied -✅ **Reorder**: -- Multiple independent joins -- Different table sizes (significant difference) -- No complex join conditions +| Rule | Status on the engine's query path | Where | +|------|-----------------------------------|-------| +| Cost-based plan shape | ✅ applied — heuristic cost per recognised clause | `Optimization/QueryOptimizer.EstimateCost` | +| Predicate pushdown (as a rewrite) | ❌ not applied — `LogicalPlan.PredicatesPushed` is `hasWhere \|\| hasHaving` ("pushdown planned") | `Optimization/QueryOptimizer.BuildLogicalPlan` | +| Predicate evaluation helper (`CanPushDown`, most-selective-first ordering, SIMD-aware row filtering) | ✅ available, not wired into the compiled path | `Planning/PredicatePushdown` (static) | +| Join reordering | ❌ not applied — `LogicalPlan.JoinsReordered = hasJoins` ("placeholder flag") | `Optimization/QueryOptimizer.BuildLogicalPlan` | +| Join-order algorithms | ✅ exist on unused components | `Planning.QueryOptimizer.OptimizeJoinOrder`, `Query.CostBasedOptimizer.OptimizeJoinOrder` | +| Subquery → semi/anti-join conversion | ❌ does not exist (`SemiJoin`/`AntiJoin` absent from the source) | — | +| Subquery result caching with per-table invalidation | ✅ applied | `Execution/SubqueryCache` | +| Compiled-plan caching (normalised SQL + parameter shape + command type) | ✅ applied, on by default | `Database/Caching/Database.PlanCaching.cs` | +| Hash join selection | ❌ disabled — threshold is `int.MaxValue` ("until key hashing is implemented") | `Execution/JoinExecutor.cs` | -❌ **Don't Reorder**: -- OUTER JOINs (may change semantics) -- Cross joins (already optimal) -- Single join (nothing to reorder) +## Roadmap -## Future Enhancements +What is *not* on the path yet, in rough priority order: -1. **Selectivity Estimation**: Use column statistics for better filter estimates -2. **Index-Aware Optimization**: Choose IndexScan when beneficial -3. **Lateral Joins**: PostgreSQL-style lateral join optimization -4. **Partition Pruning**: Skip unneeded partitions -5. **Parallel Execution**: Multi-threaded operator execution -6. **Query Rewriting**: Algebraic equivalences (e.g., (a ⋈ b) ⋈ c ≡ a ⋈ (b ⋈ c)) -7. **Approximate Selectivity**: ML-based predicate selectivity prediction -8. **Plan Statistics**: Track plan performance, adapt future optimizations +1. **Join reordering** — the flags exist; the search does not. `OptimizeJoinOrder` implementations are + already written and tested on two unused components. +2. **Subquery→join rewriting** — needed before `SubqueriesEliminated` can mean anything beyond + "the source is not a subquery". +3. **Hash joins** — requires key hashing in `JoinExecutor`; today's joins are nested-loop. +4. **Index-aware plan choice on the compiled path** (`Query/CostBasedOptimizer.ChooseBestIndex` exists, + unused). +5. Deeper selectivity statistics, partition pruning, and parallel operator execution — none implemented. ## Notes -- **HOT PATH**: All hot paths (optimize, estimate cost, pushdown) use zero-allocation algorithms -- **Cache-Friendly**: Struct-based nodes in PhysicalPlan enable plan reuse -- **Extensible**: Easy to add new optimizer rules by creating new transformation classes -- **Production-Ready**: Handles edge cases, null checks, error conditions -- **.NET 11 + C# 15**: Uses latest features (init properties, required, patterns) +- **Plan shape**: `LogicalPlan` and `PhysicalPlan` are `readonly struct`s, and `PhysicalStep` is a small + struct array — the plan path is allocation-light, but "zero allocation" would overstate it: each + `Optimize` call builds a `PhysicalStep[]`. +- **Not a search engine**: no Selinger-style dynamic programming, no plan enumeration beyond the single + costed plan this class builds. +- **Honesty over multipliers**: this page deliberately carries no per-rule speedup table; the numbers + that are published for this engine are measured end-to-end and live in `docs/benchmarks/` and + `docs/manual/performance.md`. +- **.NET 11 / C# 15**: the implementation uses `required` members, `init` properties and pattern + matching, consistent with the rest of the codebase. diff --git a/docs/manual/migration.md b/docs/manual/migration.md index 4bcf5a9b..5cfa0878 100644 --- a/docs/manual/migration.md +++ b/docs/manual/migration.md @@ -35,8 +35,10 @@ ## 12.4 FluentMigrator -Add versioned migrations to any SharpCoreDB database with the `SharpCoreDB.FluentMigrator` -package — see [Providers](providers.md#106-migrations-sharpcoredbfluentmigrator). +Add versioned migrations to any SharpCoreDB database with the FluentMigrator integration that ships +in the **`SharpCoreDB.Extensions`** package (`AddSharpCoreDBFluentMigrator()` / +`AddSharpCoreDBFluentMigratorGrpc()` — there is no separate `SharpCoreDB.FluentMigrator` package). +See [Providers](providers.md) §10.6 and the [FluentMigrator FAQ](../migration/README.md). ## 12.5 Check list diff --git a/docs/manual/providers.md b/docs/manual/providers.md index 16d11e46..1230a5b1 100644 --- a/docs/manual/providers.md +++ b/docs/manual/providers.md @@ -67,9 +67,12 @@ See [`src/SharpCoreDB.Functional.Linq2DB/README.md`](../../src/SharpCoreDB.Funct Bidirectional cloud/edge data sync provider built on Dotmim.Sync. Docs: [`docs/sync/README.md`](../sync/README.md) · [`docs/sync/CHANGELOG.md`](../sync/CHANGELOG.md) -## 10.6 Migrations (`SharpCoreDB.FluentMigrator`) +## 10.6 Migrations (FluentMigrator via `SharpCoreDB.Extensions`) -FluentMigrator integration for embedded and server modes: +FluentMigrator integration for embedded and server modes ships in the **`SharpCoreDB.Extensions`** +package (`AddSharpCoreDBFluentMigrator()` / `AddSharpCoreDBFluentMigratorGrpc()`); there is no +separate `SharpCoreDB.FluentMigrator` package. Scenarios, syntax modes and overrides are documented +in [`docs/migration/README.md`](../migration/README.md): ```csharp [Maintenance(MigrationStage.BeforeAll)] diff --git a/docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md b/docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md index a3567a1e..4aeaea2b 100644 --- a/docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md +++ b/docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md @@ -1,7 +1,7 @@ # SharpCoreDB — Insert & Update Performance Plan **Status:** draft for owner review · **Date:** 2026-09-17 · **Branch:** `release/v2.1.0.0-RC.3` -**Second-opinion addendum (Grok/xAI, 2026-09-17):** Deep codebase investigation completed. Detailed implementation hints, exact file+method references, profiler citations and prioritized next steps added at the end of this document (section 12). This serves as living reference for any AI continuing performance work. See also `PERFORMANCE_VALIDATION_AND_ADVICE_CSHARP15_NET11.md` (created alongside). +**Second-opinion addendum (Grok/xAI, 2026-09-17):** Deep codebase investigation completed. Detailed implementation hints, exact file+method references, profiler citations and prioritized next steps added at the end of this document (section 12). This serves as living reference for any AI continuing performance work. The mission brief and the append-only session log for that work are [`AUTONOMOUS_AGENT_BRIEF.md`](AUTONOMOUS_AGENT_BRIEF.md) and [`WORKLOG.md`](WORKLOG.md). **Companion docs:** [`V2_PERFORMANCE_PLAN.md`](V2_PERFORMANCE_PLAN.md) (the living v2.x roadmap), [`../benchmarks/V198_V20_V21_PERFORMANCE_COMPARISON.md`](../benchmarks/V198_V20_V21_PERFORMANCE_COMPARISON.md), [`../benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md`](../benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md), diff --git a/docs/performance/WORKLOG.md b/docs/performance/WORKLOG.md index 67efc920..094df0a3 100644 --- a/docs/performance/WORKLOG.md +++ b/docs/performance/WORKLOG.md @@ -5110,3 +5110,13 @@ Nothing in the repository explains a stall: no `.git/*.lock`, no fsmonitor daemo - Commit: local commit on the feature branch `perf/autonomous-20261011` — **never on `master`**, never pushed (brief §9). - NEXT: resume the plan's item list from session 81's `NEXT:` line (§5.1 INSERT, its item (i): the identity-remap fast path for `InsertBatch(table, rows, columns)` whose column order already is the table order — 112 B/row on the array arm). This session added no new perf item, so the queue's head is unchanged. + +### 2026-10-10 (session 83) — documentation correctness pass 2: every page checked against the shipped 2.1 engine — the server guide's TOML format, the optimizer guide's non-existent components, the plan-cache guide's internal APIs, a `SharpCoreDB.FluentMigrator` package that does not exist, and five new core-suite guards + +- Session: 1 of 1 for this item — a documentation-correctness item, not a plan §5 item; taken because the pages named a configuration format, a package, an optimizer component and a statistics API that do not exist, and the "a defect you find is yours to fix" duty covers documentation that sends a reader to a non-existent method or file. No §5 item was started, so the plan's queue is unchanged. +- Command(s): `pwsh artifacts/doc-audit-v2.ps1` (scratch harness — extracts every backticked identifier, path and package id from `docs/**/*.md` and diffs it against symbol sets built from `src/**/*.cs`, the project list and `Test-Path`) → raw buckets PKG 22 / TEST 0 / PATH 91 / API 29, dominated by the append-only `WORKLOG.md` / `CHANGELOG.md` evidence, with every residual non-historical finding then read at its site · source reads: `Server.Core/Configuration/ServerConfiguration.cs`, `Server/Program.cs`, `Server/appsettings.json`, `Optimization/QueryOptimizer.cs`, `Execution/{JoinExecutor,SubqueryCache}.cs`, `Database/Caching/Database.PlanCaching.cs`, `Database/Execution/Database.Execution.cs`, `Services/{QueryCompiler,SqlQueryValidator}.cs`, `Database/Core/Database.Core.cs`, `SingleFileTable.cs`, `Extensions/SharpCoreDBFluentMigratorExtensions.cs`, `SharpCoreDB.Functional/{Prelude,Fin}.cs`, plus `VectorSearch`/`EntityFrameworkCore` greps · `dotnet build tests/SharpCoreDB.Tests/SharpCoreDB.Tests.csproj -c Release -f net11.0` → **0 errors** (287 warnings, the standing baseline) · the doc tests `tests/SharpCoreDB.Tests/bin/Release/net11.0/SharpCoreDB.Tests.exe -filterVSTest 'FullyQualifiedName~DocumentationConsistencyTests'` → **9 / 0 failed** · the guards' red proof: a planted `` `storage/GONE_v9.md` `` line in `docs/INDEX.md` and `dotnet add package SharpCoreDB.FluentMigrator` in `docs/README.md` → **2 failed, naming exactly those two findings**, then both probes removed → **9 / 0 failed** · the core suite `-filterVSTest 'Category!=Debug&Category!=Manual&Category!=Performance'` → **2004 / 0 failed / 0 errors / 0 skipped in 89,2 s** (1999 before, +5 = the new guards). +- Regime: docs plus one test file — no engine, configuration, threshold, package or version metadata touched. Nothing was measured, so no number is produced or quoted; the removed optimizer "speedup" table was deleted rather than restated. Release/JIT, `net11.0` / C# 15 preview, branch `perf/autonomous-20261011`. The scratch harness and its reports live under `artifacts/` and are not committed. +- Verdict: **KEPT.** Every corrected page now names something that exists under `src/`, read from the source rather than inferred: `docs/server/CONFIGURATION_SCHEMA.md` is rewritten to the real JSON-only configuration model (the old page claimed "Supported Formats: TOML, JSON, YAML", shipped ~400 lines of underscore-keyed TOML and invented `[server.replication]`/`[server.monitoring]` sections against a server that registers `Microsoft.Extensions.Configuration.Json` only), `docs/internals/OPTIMIZER_GUIDE.md` no longer documents `SubqueryOptimizer`/`JoinReorderer`/`OptimizerCache`/`OptimizerIntegration`/`InvalidateOptimizationCache()`/`FilterExecutor` and states the real placeholders (`JoinsReordered = hasJoins`, `SubqueriesEliminated = !hasSubquery`, `HashJoinThreshold = int.MaxValue`), `docs/QUERY_PLAN_CACHE.md` no longer presents `internal` APIs as consumer code, the `SharpCoreDB.FluentMigrator` package reference is corrected to the real `SharpCoreDB.Extensions` integration, the EF-Core vector LINQ claims are corrected to "not shipped", the issue-#56 ✅ marks are replaced with per-artefact status, and the 1.9.3-era known-issues report carries a verified status per item (2-5 fixed in the engine, 1 and 6 resolved by a flip/opt-in). Two false leads were rejected by reading the code before editing: `SharpCoreDB.FluentMigrator` (the *package* is fictional, the *integration* is real — `AddSharpCoreDBFluentMigrator` ships in `SharpCoreDB.Extensions`, so deleting the section would have destroyed correct documentation) and `Optimization.QueryOptimizer` (the guide's architecture matches this real class, so the fix was correction, not rewrite — the "never act on an inferred reading" trap the brief lists). +- Commit: local commit on the feature branch `perf/autonomous-20261011` — **never on `master`**, never pushed (brief §9). +- NEXT: resume the plan's item list from session 81's `NEXT:` line (§5.1 INSERT, its item (i): the identity-remap fast path for `InsertBatch(table, rows, columns)` whose column order already is the table order — 112 B/row on the array arm). This session added no perf item, so the queue's head is unchanged. + diff --git a/docs/serialization/README.md b/docs/serialization/README.md index 0a863d67..255f2360 100644 --- a/docs/serialization/README.md +++ b/docs/serialization/README.md @@ -233,9 +233,11 @@ All documentation is: - `src/SharpCoreDB/Storage/SingleFileStorageProvider.cs` - Storage provider - `src/SharpCoreDB/Storage/Scdb/ScdbStructures.cs` - File header definitions -**In documentation root:** -- `SERIALIZATION_DOCUMENTATION_COMPLETE.md` - Status report -- `SERIALIZATION_DOCUMENTATION_SUMMARY.md` - Executive summary +**In this directory:** +- `BINARY_FORMAT_VISUAL_REFERENCE.md` - byte-level layout diagrams +- `SERIALIZATION_FAQ.md` - common questions about the binary format +- `SERIALIZATION_AND_STORAGE_GUIDE.md` - the long-form guide +- `../storage/STORAGE_MODE_GUIDANCE.md` - which storage mode to use --- diff --git a/docs/server/CONFIGURATION_SCHEMA.md b/docs/server/CONFIGURATION_SCHEMA.md index af2e116e..dab69fe4 100644 --- a/docs/server/CONFIGURATION_SCHEMA.md +++ b/docs/server/CONFIGURATION_SCHEMA.md @@ -1,452 +1,327 @@ -# SharpCoreDB Server Configuration Schema (v1.7.0) +# SharpCoreDB Server Configuration Schema (2.1.0) -**Supported Formats:** TOML, JSON, YAML -**Default File:** `appsettings.json` -**Environment Variables:** `SHARPCOREDB_*` prefix +**Format:** JSON, through ASP.NET Core configuration (`appsettings.json` by default) +**Root section:** `Server` +**Bound type:** `ServerConfiguration` — `src/SharpCoreDB.Server.Core/Configuration/ServerConfiguration.cs` +**Environment variables:** nested with a double underscore, e.g. `Server__Security__JwtSecretKey` + +> **There is no TOML and no YAML support.** The server registers JSON configuration sources only +> (`Microsoft.Extensions.Configuration.Json`); no TOML or YAML parser is referenced by the server or any +> project it compiles. A `sharpcoredb.toml` or `sharpcoredb.yaml` file is **not read**, and underscorized +> keys (`server_name`, `tls_enabled`, …) never bind to anything. Earlier revisions of this page showed +> TOML samples built on those keys — they described a format the server cannot consume. Use JSON. + +Keys that are not properties of the bound type are ignored by the binder, so a typo fails silently: if a +setting appears to have no effect, check its spelling against the tables below first. --- -## Basic Server Configuration - -### TOML Format -```toml -[server] -server_name = "SharpCoreDB-01" -bind_address = "0.0.0.0" -grpc_port = 5001 -enable_grpc = true -enable_grpc_http3 = true -https_api_port = 8443 -enable_https_api = true -max_connections = 1000 -connection_timeout_seconds = 300 -default_database = "master" - -[server.security] -tls_enabled = true -minimum_tls_version = "Tls12" -tls_certificate_path = "/etc/sharpcoredb/certs/server.crt" -tls_private_key_path = "/etc/sharpcoredb/certs/server.key" -jwt_secret_key = "your-256-bit-secret-here" -jwt_expiration_hours = 24 -enable_api_keys = true - -[server.databases.master] -name = "master" -database_path = "/data/master.db" -storage_mode = "SingleFile" -encryption_enabled = false -connection_pool_size = 50 -is_system_database = true -is_read_only = false - -[server.databases.app] -name = "app" -database_path = "/data/app.db" -storage_mode = "Directory" -encryption_enabled = true -encryption_key_file = "/etc/sharpcoredb/keys/app.key" -connection_pool_size = 200 -is_system_database = false -is_read_only = false - -[server.system_databases] -enabled = true -master_database_name = "master" -model_database_name = "model" -temp_database_name = "temp" - -[server.logging] -level = "Information" -file_path = "/var/log/sharpcoredb/server.log" -max_file_size_mb = 100 -max_files = 30 -structured_logging = true - -[server.performance] -query_cache_size_mb = 256 -connection_pool_max_idle_time_seconds = 300 -max_concurrent_queries = 500 -memory_limit_mb = 4096 -cpu_limit_cores = 4 -``` +## 1. How configuration is loaded + +`Program.cs` builds the host with `WebApplication.CreateBuilder(args)`, which registers the standard +providers, and adds one extra JSON file when `--appsettings` is supplied: + +| Order | Source | Notes | +|-------|--------|-------| +| 1 | `appsettings.json` | content root; the shipped sample is `src/SharpCoreDB.Server/appsettings.json` | +| 2 | `appsettings.{Environment}.json` | e.g. `appsettings.Production.json` | +| 3 | `--appsettings ` / `--appsettings=` | added with `optional: false` — a missing file **fails the process** instead of falling back silently | +| 4 | Environment variables | `Server__…` nested form | +| 5 | Command-line arguments | standard `--key=value` | + +Later sources override earlier ones. The section is materialised with +`builder.Configuration.GetSection("Server").Get()`; a missing section throws +`InvalidOperationException("Server configuration is missing")`. + +--- + +## 2. Complete example + +This is the structure of the shipped `src/SharpCoreDB.Server/appsettings.json` with development values +reduced to one database: -### JSON Format ```json { - "server": { - "serverName": "SharpCoreDB-01", - "bindAddress": "0.0.0.0", - "grpcPort": 5001, - "enableGrpc": true, - "enableGrpcHttp3": true, - "httpsApiPort": 8443, - "enableHttpsApi": true, - "maxConnections": 1000, - "connectionTimeoutSeconds": 300, - "defaultDatabase": "master", - - "security": { - "tlsEnabled": true, - "minimumTlsVersion": "Tls12", - "tlsCertificatePath": "/etc/sharpcoredb/certs/server.crt", - "tlsPrivateKeyPath": "/etc/sharpcoredb/certs/server.key", - "jwtSecretKey": "your-256-bit-secret-here", - "jwtExpirationHours": 24, - "enableApiKeys": true - }, - - "databases": { - "master": { - "name": "master", - "databasePath": "/data/master.db", - "storageMode": "SingleFile", - "encryptionEnabled": false, - "connectionPoolSize": 50, - "isSystemDatabase": true, - "isReadOnly": false - }, - "app": { - "name": "app", - "databasePath": "/data/app.db", - "storageMode": "Directory", - "encryptionEnabled": true, - "encryptionKeyFile": "/etc/sharpcoredb/keys/app.key", - "connectionPoolSize": 200, - "isSystemDatabase": false, - "isReadOnly": false + "Logging": { + "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning", "Grpc": "Information" } + }, + "AllowedHosts": "*", + "Server": { + "ServerName": "SharpCoreDB Server (Development)", + "BindAddress": "0.0.0.0", + "GrpcPort": 5001, + "EnableGrpc": true, + "EnableHttpsApi": true, + "HttpsApiPort": 8443, + "EnableWebAdmin": false, + "WebAdminPath": "/admin", + "EnableWebSocket": true, + "WebSocketPath": "/ws", + "WebSocketMaxMessageSize": 4194304, + "WebSocketKeepAliveSeconds": 30, + "MaxConnections": 1000, + "ConnectionTimeoutSeconds": 300, + "DefaultDatabase": "master", + "Databases": [ + { + "Name": "appdb", + "DatabasePath": "./data/user/appdb.scdb", + "StorageMode": "SingleFile", + "EncryptionEnabled": true, + "EncryptionKeyFile": "./secrets/appdb.key", + "ConnectionPoolSize": 50, + "IsSystemDatabase": false, + "IsReadOnly": false } + ], + "SystemDatabases": { + "Enabled": true, + "MasterDatabaseName": "master", + "ModelDatabaseName": "model", + "MsdbDatabaseName": "msdb", + "TempDbDatabaseName": "tempdb" }, - - "systemDatabases": { - "enabled": true, - "masterDatabaseName": "master", - "modelDatabaseName": "model", - "tempDatabaseName": "temp" + "Security": { + "TlsEnabled": true, + "TlsCertificatePath": "./certs/server.pfx", + "TlsPrivateKeyPath": null, + "MinimumTlsVersion": "Tls12", + "JwtSecretKey": "CHANGE-ME-in-production-must-be-at-least-32-characters-long", + "JwtExpirationHours": 24, + "EnableApiKeys": true, + "Users": [ + { "Username": "admin", "PasswordHash": "", "Role": "admin" } + ] }, - - "logging": { - "level": "Information", - "filePath": "/var/log/sharpcoredb/server.log", - "maxFileSizeMb": 100, - "maxFiles": 30, - "structuredLogging": true + "Logging": { + "Level": "Information", + "FilePath": "./logs/sharpcoredb-server.log", + "MaxFileSizeMb": 100, + "MaxFiles": 10 }, - - "performance": { - "queryCacheSizeMb": 256, - "connectionPoolMaxIdleTimeSeconds": 300, - "maxConcurrentQueries": 500, - "memoryLimitMb": 4096, - "cpuLimitCores": 4 + "Performance": { + "QueryCacheSizeMb": 256, + "MaxConcurrentQueries": 500 } } } -``` --- -## Configuration Sections - -### Server Section - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `serverName` | string | "SharpCoreDB" | Server display name | -| `bindAddress` | string | "0.0.0.0" | IP address to bind to | -| `grpcPort` | int | 5001 | gRPC HTTPS port | -| `enableGrpc` | bool | true | Enable gRPC protocol | -| `enableGrpcHttp3` | bool | false | Enable gRPC over HTTP/3 | -| `httpsApiPort` | int | 8443 | HTTPS REST API port | -| `enableHttpsApi` | bool | true | Enable HTTPS REST API | -| `maxConnections` | int | 1000 | Maximum concurrent connections | -| `connectionTimeoutSeconds` | int | 300 | Connection timeout | -| `defaultDatabase` | string | "master" | Default database name | - -### Security Section - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `tlsEnabled` | bool | true | Require TLS encryption | -| `minimumTlsVersion` | enum | "Tls12" | Minimum TLS version (Tls12, Tls13) | -| `tlsCertificatePath` | string | required | Path to TLS certificate (.crt/.pem) | -| `tlsPrivateKeyPath` | string | required | Path to TLS private key | -| `jwtSecretKey` | string | required | JWT signing secret (256-bit) | -| `jwtExpirationHours` | int | 24 | JWT token expiration | -| `enableApiKeys` | bool | true | Enable API key authentication | - -### Database Configuration - -Each database is configured as a named section: - -```toml -[server.databases.myapp] -name = "myapp" -database_path = "/data/myapp.db" -storage_mode = "SingleFile" -encryption_enabled = true -encryption_key_file = "/keys/myapp.key" -connection_pool_size = 100 -is_system_database = false -is_read_only = false -``` - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `name` | string | required | Logical database name | -| `databasePath` | string | required | File system path to database | -| `storageMode` | enum | "SingleFile" | SingleFile, Directory, Columnar | -| `encryptionEnabled` | bool | false | Enable AES-256-GCM encryption | -| `encryptionKeyFile` | string | null | Path to encryption key file | -| `connectionPoolSize` | int | 50 | Connection pool size | -| `isSystemDatabase` | bool | false | Marks as system database | -| `isReadOnly` | bool | false | Read-only access | - -### System Databases - -```toml -[server.system_databases] -enabled = true -master_database_name = "master" -model_database_name = "model" -temp_database_name = "temp" -``` - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `enabled` | bool | true | Enable system databases | -| `masterDatabaseName` | string | "master" | Master system database | -| `modelDatabaseName` | string | "model" | Model system database | -| `tempDatabaseName` | string | "temp" | Temporary database | - -### Logging Configuration - -```toml -[server.logging] -level = "Information" -file_path = "/var/log/sharpcoredb/server.log" -max_file_size_mb = 100 -max_files = 30 -structured_logging = true -console_enabled = true -console_level = "Warning" -``` - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `level` | enum | "Information" | Minimum log level | -| `filePath` | string | null | Log file path | -| `maxFileSizeMb` | int | 100 | Max log file size | -| `maxFiles` | int | 30 | Max log file count | -| `structuredLogging` | bool | true | Enable structured logging | -| `consoleEnabled` | bool | true | Enable console logging | -| `consoleLevel` | enum | "Warning" | Console log level | - -### Performance Tuning - -```toml -[server.performance] -query_cache_size_mb = 256 -connection_pool_max_idle_time_seconds = 300 -max_concurrent_queries = 500 -memory_limit_mb = 4096 -cpu_limit_cores = 4 -enable_query_plan_caching = true -query_plan_cache_size_mb = 128 -``` - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `queryCacheSizeMb` | int | 256 | Query result cache size | -| `connectionPoolMaxIdleTimeSeconds` | int | 300 | Connection pool idle timeout | -| `maxConcurrentQueries` | int | 500 | Max concurrent query execution | -| `memoryLimitMb` | int | 4096 | Memory limit per database | -| `cpuLimitCores` | int | 4 | CPU core limit | -| `enableQueryPlanCaching` | bool | true | Enable query plan caching | -| `queryPlanCacheSizeMb` | int | 128 | Query plan cache size | - -### Projection Runtime (Optional) - -```toml -[server.projections] -enabled = false -enable_hosted_worker = false -use_persistent_checkpoints = false -use_open_telemetry_metrics = false -database_name = "app" -checkpoint_table_name = "scdb_projection_checkpoints" -runtime_database_id = "main" -runtime_tenant_id = "default" -from_global_sequence = 1 -batch_size = 1000 -poll_interval_milliseconds = 250 -run_on_start = true -# max_iterations = 20 # optional; omit for continuous processing -``` - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `enabled` | bool | false | Enables projection runtime service wiring | -| `enableHostedWorker` | bool | false | Starts projection hosted worker | -| `usePersistentCheckpoints` | bool | false | Persists checkpoints in SharpCoreDB tables | -| `useOpenTelemetryMetrics` | bool | false | Replaces default projection metrics with OpenTelemetry adapter | -| `databaseName` | string | null | Database used for event store/checkpoint runtime | -| `checkpointTableName` | string | `scdb_projection_checkpoints` | Table name for persistent checkpoints | -| `runtimeDatabaseId` | string | `main` | Logical projection runtime database scope | -| `runtimeTenantId` | string | `default` | Logical projection runtime tenant scope | -| `fromGlobalSequence` | long | 1 | Start sequence when no checkpoint exists | -| `batchSize` | int | 1000 | Projection run batch size | -| `pollIntervalMilliseconds` | int | 250 | Background worker poll interval | -| `runOnStart` | bool | true | Executes first projection cycle immediately | -| `maxIterations` | int? | null | Optional bounded iterations for controlled runs | +## 3. `Server` section + +| Key | Type | Default | Notes | +|-----|------|---------|-------| +| `ServerName` | string | **required** | Display name | +| `BindAddress` | string | `0.0.0.0` | Bind address | +| `GrpcPort` | int | `5001` | gRPC over TLS port | +| `EnableGrpc` | bool | `true` | Flagship protocol | +| `EnableGrpcHttp3` | bool | `true` | gRPC over HTTP/3 (QUIC) | +| `EnableBinaryProtocol` | bool | `true` | Secondary binary protocol | +| `BinaryProtocolPort` | int | `5433` | Binary-protocol port | +| `EnableHttpsApi` | bool | `true` | REST/JSON over HTTPS | +| `HttpsApiPort` | int | `8443` | REST/JSON port | +| `EnableWebSocket` | bool | `true` | WebSocket endpoint | +| `WebSocketPath` | string | `/ws` | WebSocket path | +| `WebSocketMaxMessageSize` | int | `4194304` (4 MB) | Max WebSocket frame | +| `WebSocketKeepAliveSeconds` | int | `30` | Keep-alive interval | +| `EnableWebAdmin` | bool | `false` | Admin UI | +| `WebAdminPath` | string | `/admin` | Admin UI path (longest-prefix match wins over the global route prefix) | +| `MaxConnections` | int | `10000` | Maximum concurrent connections | +| `ConnectionTimeoutSeconds` | int | `300` | Connection timeout | +| `DefaultDatabase` | string | **required** | Database used when a client does not name one | +| `Databases` | array | **required** | Per-database instances, see §3.1 | +| `SystemDatabases` | object | defaults | see §3.2 | +| `Security` | object | **required** | see §3.3 | +| `Logging` | object | defaults | see §3.4 | +| `Performance` | object | defaults | see §3.5 | +| `ConnectionPool` | object | defaults | see §3.6 | +| `TenantQuotas` | object | defaults | see §3.7 | +| `Projections` | object | disabled | see §3.8 | + +### 3.1 `Databases` (array) + +`Databases` is a **JSON array**, one object per hosted database — not a name-keyed object. + +| Key | Type | Default | Notes | +|-----|------|---------|-------| +| `Name` | string | **required** | Logical name clients use | +| `DatabasePath` | string | **required** | Physical path (`*.scdb` for `SingleFile`) | +| `StorageMode` | string | `SingleFile` | `SingleFile`, `Directory` or `Columnar` | +| `EncryptionEnabled` | bool | `false` | Per-database at-rest encryption | +| `EncryptionKeyFile` | string? | `null` | Key file for this database | +| `EncryptionMasterPassword` | string? | `null` | Resolved master password, as an alternative to a key file | +| `ConnectionPoolSize` | int | `1000` | Pool size for this database | +| `IsSystemDatabase` | bool | `false` | Marks a system database | +| `IsReadOnly` | bool | `false` | Read-only instance | + +### 3.2 `SystemDatabases` + +| Key | Type | Default | +|-----|------|---------| +| `Enabled` | bool | `true` | +| `MasterDatabaseName` | string | `master` | +| `ModelDatabaseName` | string | `model` | +| `MsdbDatabaseName` | string | `msdb` | +| `TempDbDatabaseName` | string | `tempdb` | + +### 3.3 `Security` + +| Key | Type | Default | Notes | +|-----|------|---------|-------| +| `RequireAuthentication` | bool | — | Appears in the shipped sample; **not** a property of `SecurityConfiguration` (the binder ignores it, as it does `AuthMethods`) | +| `TlsEnabled` | bool | `true` | Setting `false` aborts startup — see §5 | +| `MinimumTlsVersion` | string | `Tls12` | `Tls12` (allows TLS 1.2 + 1.3) or `Tls13` (1.3 only); anything else aborts startup | +| `TlsCertificatePath` | string | **required** | `.pfx`, or `.crt`/`.pem` with a private key | +| `TlsPrivateKeyPath` | string? | `null` | **Required** when the certificate is not a PFX | +| `TlsCertificatePassword` | string? | `null` | For password-protected PFX files | +| `JwtSecretKey` | string | **required** | Must be at least 32 characters — see §5 | +| `JwtExpirationHours` | int | `24` | Token lifetime | +| `EnableApiKeys` | bool | `true` | API-key authentication | +| `Users` | array | `[]` | see below | +| `EnableMutualTls` | bool | `false` | Client-certificate authentication | +| `ClientCaCertificatePath` | string? | `null` | CA that must have signed client certificates | +| `CertificateRoleMappings` | array | `[]` | see below | +| `TenantEncryption` | object | disabled | see below | + +**`Users[]`** — `Username` (required), `PasswordHash` (required, **SHA-256 hex**, not plaintext), +`Role` (`admin` | `writer` | `reader`, default `reader`), `TenantId` (default `default`), +`AllowedDatabases` (default `["*"]`), `ScopeVersion` (default `1`). + +**`CertificateRoleMappings[]`** — `Thumbprint` (required, SHA-256 hex, case-insensitive), +`Role` (default `reader`), `Description`. + +**`TenantEncryption`** — `Enabled` (`false`), `RequireDedicatedTenantKey` (`false`), +`DefaultProvider` (`configuration`), `DefaultKeyReference`, `DefaultKeyReferenceByTenant` (map), +`KeyMaterialByReference` (map, for local/dev use). + + +### 3.4 `Logging` + +Note this is `Server:Logging` (SharpCoreDB's own file logging). ASP.NET's log *filter* section is the +separate top-level `Logging:LogLevel` documented by Microsoft. + +| Key | Type | Default | +|-----|------|---------| +| `Level` | string | `Information` | +| `FilePath` | string | `/var/log/sharpcoredb/server.log` | +| `MaxFileSizeMb` | int | `100` | +| `MaxFiles` | int | `30` | +| `StructuredLogging` | bool | `true` | + +### 3.5 `Performance` + +| Key | Type | Default | +|-----|------|---------| +| `QueryCacheSizeMb` | int | `256` | +| `ConnectionPoolMaxIdleTimeSeconds` | int | `300` | +| `MaxConcurrentQueries` | int | `500` | +| `MemoryLimitMb` | long | `4096` | +| `CpuLimitCores` | int | `4` | + +### 3.6 `ConnectionPool` + +| Key | Type | Default | +|-----|------|---------| +| `MinPoolSize` | int | `2` | +| `MaxPoolSize` | int | `1000` | +| `IdleTimeoutSeconds` | int | `300` | +| `AcquireTimeoutSeconds` | int | `30` | +| `EnableHealthChecks` | bool | `true` | +| `HealthCheckIntervalSeconds` | int | `60` | + +### 3.7 `TenantQuotas` + +| Key | Type | Default | +|-----|------|---------| +| `Enabled` | bool | `true` | +| `DefaultMaxActiveSessions` | int | `100` | +| `DefaultMaxRequestsPerSecond` | int | `500` | +| `DefaultMaxStorageMb` | long | `4096` | +| `DefaultMaxBatchSize` | int | `1000` | +| `PlanDefaults` | map | `{}` — plan tier → `{ MaxActiveSessions, MaxRequestsPerSecond, MaxStorageMb, MaxBatchSize }` | + +### 3.8 `Projections` (optional) + +Every projection feature is off by default, so the optional package stays optional. + +| Key | Type | Default | +|-----|------|---------| +| `Enabled` | bool | `false` | +| `EnableHostedWorker` | bool | `false` | +| `UsePersistentCheckpoints` | bool | `false` | +| `UseOpenTelemetryMetrics` | bool | `false` | +| `DatabaseName` | string? | `null` (falls back to `DefaultDatabase`) | +| `CheckpointTableName` | string | `scdb_projection_checkpoints` | +| `RuntimeDatabaseId` | string | `main` | +| `RuntimeTenantId` | string | `default` | +| `FromGlobalSequence` | long | `1` | +| `BatchSize` | int | `1000` | +| `PollIntervalMilliseconds` | int | `250` | +| `RunOnStart` | bool | `true` | +| `MaxIterations` | int? | `null` (continuous) | --- -## Environment Variables +## 4. Environment variables -Configuration can be overridden with environment variables: +The nested form uses `__` between levels. These are the real names (no `SHARPCOREDB_` prefix): ```bash -# Server settings -export SHARPCOREDB_SERVER__SERVERNAME="Production-DB" -export SHARPCOREDB_SERVER__MAXCONNECTIONS=2000 - -# Security settings -export SHARPCOREDB_SERVER__SECURITY__JWTSECRETKEY="your-secret-key" -export SHARPCOREDB_SERVER__SECURITY__TLSCERTIFICATEPATH="/certs/server.crt" - -# Database settings -export SHARPCOREDB_SERVER__DATABASES__MYAPP__CONNECTIONPOOLSIZE=500 - -# Logging -export SHARPCOREDB_SERVER__LOGGING__LEVEL="Debug" +export Server__ServerName="Production-DB" +export Server__MaxConnections=2000 +export Server__Security__JwtSecretKey="" +export Server__Security__TlsCertificatePath="/certs/server.pfx" +export Server__Databases__0__Name="appdb" +export Server__Databases__0__DatabasePath="/data/appdb.scdb" +export Server__Logging__Level="Debug" ``` ---- - -## Configuration Validation - -The server validates configuration on startup: - -### Required Settings -- `server.security.tlsEnabled = true` (TLS cannot be disabled) -- `server.security.tlsCertificatePath` (certificate required) -- `server.security.tlsPrivateKeyPath` (private key required) -- `server.security.jwtSecretKey` (256-bit secret required) -- `server.defaultDatabase` (must reference existing database) - -### Automatic Defaults -- System databases are created automatically if enabled -- Default database is created if it doesn't exist -- Connection pools are sized based on `maxConnections` - -### Runtime Validation -- Certificate files must exist and be readable -- Database paths must be writable (unless read-only) -- Encryption keys must be valid AES-256 keys -- Port numbers must be available +Array elements are addressed by index (`__0__`, `__1__`), which is why `Databases` must be an array in +both JSON and environment-variable form. --- -## Advanced Configuration - -### High Availability Setup - -```toml -[server] -server_name = "SharpCoreDB-Cluster-01" -max_connections = 5000 - -[server.databases.shared] -name = "shared" -database_path = "/shared/data/shared.db" -storage_mode = "Directory" -encryption_enabled = true -connection_pool_size = 1000 - -[server.replication] -enabled = true -role = "primary" # primary, secondary, arbiter -cluster_name = "prod-cluster" -replication_port = 5433 -sync_mode = "synchronous" # synchronous, asynchronous - -[server.replication.nodes] -node1 = "sharpcoredb-02:5433" -node2 = "sharpcoredb-03:5433" -node3 = "sharpcoredb-04:5433" -``` +## 5. Startup validation -### Monitoring Integration +`Program.cs` fails fast rather than starting a half-configured server: -```toml -[server.monitoring] -enabled = true -prometheus_port = 9090 -metrics_prefix = "sharpcoredb" -enable_tracing = true -tracing_endpoint = "http://jaeger:14268/api/traces" +| Check | Behaviour when violated | +|-------|-------------------------| +| `Server` section present | `InvalidOperationException("Server configuration is missing")` | +| `Security:JwtSecretKey` length | `InvalidOperationException("JWT secret key must be at least 32 characters.")` | +| `Security:TlsEnabled` | `InvalidOperationException("TLS must be enabled. Plain HTTP is not supported.")` — there is no plain-HTTP mode | +| `Security:TlsCertificatePath` | `InvalidOperationException("TLS certificate path is required.")` | +| `Security:TlsPrivateKeyPath` | Required unless the certificate ends in `.pfx` | +| `Security:MinimumTlsVersion` | Only `Tls12` and `Tls13` are accepted; anything else throws | -[server.monitoring.alerts] -high_connection_count_threshold = 800 -slow_query_threshold_ms = 1000 -memory_usage_threshold_mb = 3500 -``` - -### Development Configuration - -```toml -[server] -server_name = "SharpCoreDB-Dev" -max_connections = 100 - -[server.security] -tls_enabled = false # Only for development! -jwt_secret_key = "dev-secret-key-12345678901234567890123456789012" - -[server.logging] -level = "Debug" -console_enabled = true -console_level = "Information" - -[server.databases.dev] -name = "dev" -database_path = "./data/dev.db" -storage_mode = "SingleFile" -encryption_enabled = false -``` +Members marked **required** in the schema above (`ServerName`, `DefaultDatabase`, `Databases`, +`Security`, per-database `Name`/`DatabasePath`, `JwtSecretKey`, `TlsCertificatePath`) must be present in +the effective configuration — the shipped `appsettings.json` sets them all. --- -## Configuration File Locations - -The server looks for configuration files in this order: - -1. `appsettings.json` (current directory) -2. `appsettings.{Environment}.json` (e.g., `appsettings.Production.json`) -3. `sharpcoredb.toml` (TOML format) -4. `sharpcoredb.yaml` (YAML format) -5. Environment variables -6. Command line arguments +## 6. Docker -### Docker Configuration +The published image is `ghcr.io/mpcoredeveloper/sharpcoredb-server` (built by +`.github/workflows/docker-publish.yml` on `v*` tags, excluding the parked `v2.0*` line): ```yaml -# docker-compose.yml -version: '3.8' services: sharpcoredb: - image: sharpcoredb/server:1.7.0 + image: ghcr.io/mpcoredeveloper/sharpcoredb-server:2.1.0 ports: - - "5001:5001" # gRPC - - "8443:8443" # HTTPS API + - "5001:5001" # gRPC over TLS + - "8443:8443" # HTTPS REST API volumes: - ./data:/data - ./certs:/certs - ./config:/app/config environment: - - SHARPCOREDB_SERVER__DATABASES__APP__DATABASEPATH=/data/app.db - - SHARPCOREDB_SERVER__SECURITY__TLSCERTIFICATEPATH=/certs/server.crt + - Server__Security__JwtSecretKey=${SHARPCOREDB_JWT_SECRET} + - Server__Security__TlsCertificatePath=/certs/server.pfx + - Server__Databases__0__Name=appdb + - Server__Databases__0__DatabasePath=/data/appdb.scdb configs: - source: server_config target: /app/appsettings.json @@ -458,6 +333,7 @@ configs: --- -**Last Updated:** January 28, 2026 -**Configuration Version:** 1.0 -**Supported Formats:** JSON, TOML, YAML +**Verified against:** `src/SharpCoreDB.Server.Core/Configuration/ServerConfiguration.cs`, +`src/SharpCoreDB.Server/Program.cs`, `src/SharpCoreDB.Server/appsettings.json` — SharpCoreDB 2.1.0. + +``` diff --git a/docs/sharpcoredb-known-issues.md b/docs/sharpcoredb-known-issues.md index ab875dc6..ce6e3cdd 100644 --- a/docs/sharpcoredb-known-issues.md +++ b/docs/sharpcoredb-known-issues.md @@ -5,6 +5,12 @@ provider against **SharpCoreDB NuGet 1.9.3** (source checked out at `D:\repos\MPCoreDeveloper\SharpCoreDB`). The items are intended to be picked up in the SharpCoreDB repository: +> **Audit status (2026-10-10):** every item below was re-checked against the shipped **2.1.0** tree. +> Five of the six are fixed in the engine (2–5) or resolved by a flip/opt-in (1, 6); each item carries +> a `RESOLVED` / `STILL TRUE` note naming the file that proves it. The report is kept as the historical +> record of a 1.9.3-era integration — an un-annotated sentence inside an item is that item's original +> observation, not a claim about 2.1.0. + --- ## 1. 🔴 Table data is NOT encrypted even with `EnableBatchEncryption=true` @@ -57,6 +63,14 @@ when `!noEncryption`, and `FlushBufferedAppends` should actually write the encry ## 2. 🟠 `Table.Insert` throws `ArgumentOutOfRangeException` after reopening a database +> ✅ **RESOLVED (verified 2026-10-10, 2.1.0).** `Database.Core.Load()` now pads *all seven* per-column +> lists to `Columns.Count`, including the two this report found short: +> `DefaultExpressions` and `ColumnCheckExpressions` (the `while (… .Count < table.Columns.Count)` +> block in `src/SharpCoreDB/Database/Core/Database.Core.cs`), alongside `IsAuto`, `IsNotNull`, +> `DefaultValues`, `ColumnCollations` and `ColumnLocaleNames`. Reopening a database and then calling +> `ITable.Insert` no longer indexes a short list, so the provider-side `RepairTableSchemaLists()` +> workaround is no longer needed. + **Location:** `src/SharpCoreDB/Database/Core/Database.Core.cs` → `Load()` (lines 184-428) - After deserialization, `IsAuto`, `IsNotNull`, `DefaultValues`, and `ColumnCollations` are @@ -77,6 +91,13 @@ column count. ## 3. 🟠 Single-file (.scdb) mode has a no-op B-tree index +> ✅ **RESOLVED (verified 2026-10-10, 2.1.0).** `SingleFileTable` now keeps a primary-key hash index +> (`_pkIndex`, marked "Issue A1" in `src/SharpCoreDB/SingleFileTable.cs`) built from the PK column, so +> `FindByPrimaryKey` / `UpdateByPrimaryKey` / `DeleteByPrimaryKey` are O(1) point operations in `.scdb` +> mode — directory mode is no longer required for correct point lookups. The `NullIndex` this report +> saw survives only as the interface's no-op default (`Interfaces/ITable.cs`) for implementations that +> declare no index of their own. + **Location:** `src/SharpCoreDB/Interfaces/ITable.cs` (lines 101-107) and `src/SharpCoreDB/SingleFileTable.cs` @@ -96,6 +117,14 @@ mode. ## 4. 🟡 No read-after-write consistency without an explicit `Flush()` +> ✅ **RESOLVED (verified 2026-10-10, 2.1.0).** `Database.ExecuteQuery` now flushes pending **batch** +> writes before running the query (`src/SharpCoreDB/Database/Execution/Database.Execution.cs`, +> "FIX (Known Issue 4)"), so rows written through `ITable.Insert` / `UpdateByPrimaryKey` are visible to +> `FindByPrimaryKey` / `Select()` without an explicit `db.Flush()`. One asymmetry is deliberate and +> documented in the code: a plain `_metadataDirty` flag is *not* flushed, because SQL `INSERT`s are +> already visible in-memory on the default engine and a blanket metadata flush there costs more than it +> buys. `IDatabase.Flush()` remains the explicit durability boundary. + **Location:** `src/SharpCoreDB/Database/Execution/Database.Execution.cs` - `Database.ExecuteSQL(...)` only flushes before a SELECT when `_metadataDirty || @@ -104,8 +133,10 @@ mode. - After an `ITable.Insert` / `UpdateByPrimaryKey`, changes were invisible to `ITable.FindByPrimaryKey` / `ITable.Select()` until an explicit `database.Flush()` occurred. -**Workaround in ServiceMq.SharpCoreDb:** flush before every `SelectArea` scan and on mutations -with `DurabilityMode.FlushToDisk`. +**Workaround in ServiceMq.SharpCoreDb:** flush before every `SelectArea` scan and on mutations via +`db.Flush()`. (The historical write-up named `DurabilityMode.FlushToDisk`; that member does not exist — +`DurabilityMode` is `FullSync` | `Async`, and forcing data to physical disk is what `FullSync` and an +explicit `IDatabase.Flush()` perform.) **Recommendation:** `ExecuteQuery` should perform the same dirty-flush check as `ExecuteSQL(SELECT)`. @@ -114,6 +145,11 @@ with `DurabilityMode.FlushToDisk`. ## 5. 🟡 SQL parameter validation does not match `@`-prefixed keys +> ✅ **RESOLVED (verified 2026-10-10, 2.1.0).** `SqlQueryValidator` normalises keys before comparing: +> `NormalizeKey` strips a leading `@` or `:` (its comment states "a @-prefixed key and an unprefixed key +> with the same name are equivalent"), so `@name`, `:name` and `name` all satisfy a `@name` placeholder. +> The false "Missing parameters for placeholders" / "Unused parameters provided" warnings are gone. + **Location:** `src/SharpCoreDB/Services/SqlQueryValidator.cs` (lines 129-153) - The validator compares parameter keys **without** the `@` prefix against the `@param` @@ -132,6 +168,13 @@ its behavior is consistent with `SqlParser.ResolveParameter`. ## 6. 🟡 `INTEGER` columns map to Int32 instead of Int64 +> ⚠️ **STILL TRUE BY DEFAULT, with an opt-in (verified 2026-10-10, 2.1.0).** `INTEGER` → +> `DataType.Integer` (Int32) remains the default, for backward compatibility with existing databases and +> consumer code. The engine now offers the SQLite behaviour explicitly: set +> `DatabaseConfig.UseSqliteIntegerAffinity = true` and `INTEGER` maps to `DataType.Long` (Int64) for +> **newly created or added** columns — existing columns keep the type recorded in their table metadata. +> The option is documented on `DatabaseConfig` in `src/SharpCoreDB/DatabaseConfig.cs`. + **Location:** `src/SharpCoreDB/Services/SqlParser.ParseValue` and the DDL type mapping - `INTEGER` → `DataType.Integer` (Int32): values larger than `int.MaxValue` (such as @@ -151,11 +194,11 @@ error message. ## Priority summary -| # | Issue | Severity | Status in ServiceMq.SharpCoreDb | -|---|-------|----------|----------------------------------| -| 1 | Table data not encrypted | 🔴 High | Provider-level AES-256-GCM | -| 2 | `Table.Insert` AOORE after reopen | 🟠 Medium | `RepairTableSchemaLists()` | -| 3 | Single-file no-op index | 🟠 Medium | Directory mode forced | -| 4 | No read-after-write flush | 🟡 Low | Flush calls in provider | -| 5 | SQL validator `@` prefix | 🟡 Low | Keys without `@` prefix | -| 6 | INTEGER → Int32 | 🟡 Low | `BIGINT` columns | \ No newline at end of file +| # | Issue | Severity | Status in ServiceMq.SharpCoreDb | In SharpCoreDB 2.1.0 | +|---|-------|----------|----------------------------------|----------------------| +| 1 | Table data not encrypted | 🔴 High | Provider-level AES-256-GCM | **Fixed** — per-record AES-256-GCM on by default (`EnableAtRestRecordEncryption`), flippable since 2026-09-13 | +| 2 | `Table.Insert` AOORE after reopen | 🟠 Medium | `RepairTableSchemaLists()` | **Fixed** — `Load()` pads every per-column list | +| 3 | Single-file no-op index | 🟠 Medium | Directory mode forced | **Fixed** — `SingleFileTable` keeps a PK hash index | +| 4 | No read-after-write flush | 🟡 Low | Flush calls in provider | **Fixed** — `ExecuteQuery` flushes pending batch writes | +| 5 | SQL validator `@` prefix | 🟡 Low | Keys without `@` prefix | **Fixed** — `NormalizeKey` strips `@` / `:` | +| 6 | INTEGER → Int32 | 🟡 Low | `BIGINT` columns | **Still the default** — opt in with `UseSqliteIntegerAffinity` | \ No newline at end of file diff --git a/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs b/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs index 977b1ecd..999b8116 100644 --- a/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs +++ b/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs @@ -109,6 +109,122 @@ public void DocumentationMarkdownLinks_ShouldResolveToExistingFiles() Assert.Empty(failures); } + [Fact] + public void DocumentationIndex_ShouldOnlyReferenceExistingDocumentationFiles() + { + // Arrange: every backticked `*.md` path listed in docs/INDEX.md must resolve from docs/. + var docsRoot = GetRepositoryFilePath("docs"); + var index = File.ReadAllText(Path.Combine(docsRoot, "INDEX.md")); + var pathPattern = new Regex(@"`([A-Za-z0-9_][A-Za-z0-9_./\-]*\.md)`", RegexOptions.Compiled); + var failures = new List(); + + // Act + foreach (Match match in pathPattern.Matches(index)) + { + var relative = match.Groups[1].Value; + var resolved = Path.GetFullPath(Path.Combine(docsRoot, relative.Replace('/', Path.DirectorySeparatorChar))); + + if (!File.Exists(resolved)) + { + failures.Add(relative); + } + } + + // Assert + Assert.Empty(failures); + } + + [Fact] + public void ServerConfigurationGuide_ShouldNotDocumentTomlOrYamlConfiguration() + { + // Arrange: the server registers JSON configuration sources only. + var content = File.ReadAllText(GetRepositoryFilePath("docs", "server", "CONFIGURATION_SCHEMA.md")); + + // Act / Assert + // No TOML samples, and the old "Supported Formats: TOML" claim must not return. YAML is still + // legitimate in this file for docker-compose, so only the config-format claim is guarded. + Assert.DoesNotContain("```toml", content, StringComparison.OrdinalIgnoreCase); + Assert.DoesNotContain("Supported Formats:** TOML", content, StringComparison.OrdinalIgnoreCase); + Assert.Contains("no TOML", content, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void PlanCacheGuide_ShouldNotPresentInternalApisAsPublic() + { + // Arrange: Database.GetPlanCacheStats()/ClearPlanCache() are internal (InternalsVisibleTo only). + var content = File.ReadAllText(GetRepositoryFilePath("docs", "QUERY_PLAN_CACHE.md")); + + // Act / Assert + Assert.DoesNotContain("db.GetPlanCacheStats()", content, StringComparison.Ordinal); + Assert.DoesNotContain("db.ClearPlanCache()", content, StringComparison.Ordinal); + Assert.Contains("internal", content, StringComparison.Ordinal); + } + + [Fact] + public void KnownIssuesReport_ShouldCarryAVerifiedStatusForEveryItem() + { + // Arrange: the 1.9.3-era external report must annotate each item against the current tree. + var content = File.ReadAllText(GetRepositoryFilePath("docs", "sharpcoredb-known-issues.md")); + var items = Regex.Matches(content, @"^## \d+\.", RegexOptions.Multiline).Count; + var statusNotes = Regex.Matches(content, @"^> (✅|⚠)", RegexOptions.Multiline).Count; + + // Act / Assert + Assert.True(items >= 6, $"Expected at least 6 numbered items, found {items}."); + Assert.True( + statusNotes >= items, + $"{items} items carry only {statusNotes} verified status notes (RESOLVED / STILL TRUE)."); + } + + [Fact] + public void DocumentationPackageReferences_ShouldResolveToRealPackages() + { + // Arrange: collect the ids that actually exist (projects + central package versions). + var repositoryRoot = GetRepositoryFilePath(); + var realIds = new HashSet(StringComparer.OrdinalIgnoreCase); + + foreach (var project in Directory.EnumerateFiles(Path.Combine(repositoryRoot, "src"), "*.csproj", SearchOption.AllDirectories)) + { + realIds.Add(Path.GetFileNameWithoutExtension(project)); + } + + foreach (Match match in Regex.Matches( + File.ReadAllText(Path.Combine(repositoryRoot, "Directory.Packages.props")), + "(); + var docsRoot = Path.Combine(repositoryRoot, "docs"); + + // Act + foreach (var file in Directory.EnumerateFiles(docsRoot, "*.md", SearchOption.AllDirectories)) + { + var name = Path.GetFileName(file); + + if (name is "WORKLOG.md" or "CHANGELOG.md") + { + continue; // historical records carry ids as they were at the time + } + + foreach (Match match in referencePattern.Matches(File.ReadAllText(file))) + { + var id = match.Groups[1].Success ? match.Groups[1].Value : match.Groups[2].Value; + + if (id.StartsWith("SharpCoreDB", StringComparison.Ordinal) && !realIds.Contains(id)) + { + failures.Add($"{Path.GetRelativePath(docsRoot, file)} -> {id}"); + } + } + } + + // Assert + Assert.Empty(failures); + } + private static string GetRepositoryFilePath(params string[] segments) { var current = new DirectoryInfo(AppContext.BaseDirectory); From d0b0f5376f25fb06c1430bd527b6df87620693a1 Mon Sep 17 00:00:00 2001 From: MPCoreDeveloper Date: Sun, 11 Oct 2026 17:57:57 +0200 Subject: [PATCH 6/6] docs(optimizer): the guide's opening pipeline section matched the removed design, not the code The Architecture block still presented five rewrite phases (predicate pushdown, subquery elimination, join reordering) and a "QueryOptimizer.Core" component tree containing SubqueryOptimizer/JoinReorderer/QueryOptimizerExtensions. It now describes the three steps Optimize() actually performs (logical view, costing, physical steps), names the flags that are shape heuristics rather than rewrites, and shows the real component tree including which types are unwired building blocks. Docs suite 9/0. --- docs/internals/OPTIMIZER_GUIDE.md | 60 +++++++++++++++---------------- 1 file changed, 29 insertions(+), 31 deletions(-) diff --git a/docs/internals/OPTIMIZER_GUIDE.md b/docs/internals/OPTIMIZER_GUIDE.md index 7f1264f9..ae8faddc 100644 --- a/docs/internals/OPTIMIZER_GUIDE.md +++ b/docs/internals/OPTIMIZER_GUIDE.md @@ -21,48 +21,46 @@ A lightweight, cost-based query optimizer for SharpCoreDB with logical/physical ## Architecture -### Optimization Pipeline +### Optimization pipeline (what `Optimization.QueryOptimizer.Optimize` does) ``` -Parse AST → Logical Plan → Optimize → Physical Plan → Execute - (5 phases) +Parse AST → Logical view (shape flags, estimated rows) → Costed plan → Physical steps → Execute + one pass, no rewrite phases ``` -**Phase 1: Logical Planning** -- Convert AST to logical operators (Scan, Join, Filter, Aggregate, Sort) -- No execution decisions yet -- Preserves query semantics +`Optimize(SelectNode)` runs three steps, in this order: -**Phase 2: Predicate Pushdown** -- Move filters as close to data source as possible -- Split AND-connected conditions -- Reduce intermediate row counts +**1. Logical view** (`BuildLogicalPlan`) — derives shape flags from the AST and an estimated row count +from `CostEstimator.EstimateScanCost`. It does **not** rewrite the query: +`PredicatesPushed = hasWhere || hasHaving` ("pushdown planned"), `JoinsReordered = hasJoins` +("placeholder flag"), `SubqueriesEliminated = !hasSubquery` ("trivially eliminated"), +`Cacheable = !hasSubquery`. -**Phase 3: Subquery Elimination** -- Convert EXISTS to semi-join -- Convert NOT EXISTS to anti-join -- Convert IN (SELECT) to hash semi-join -- Inline scalar subqueries when beneficial +**2. Costing** (`EstimateCost`) — accumulates a heuristic cost per recognised clause (base scan, then +joins, filters, aggregates, sort) in clause order. -**Phase 4: Join Reordering** -- Estimate cost of different join orders -- Choose order minimizing intermediate results -- Small-to-large strategy +**3. Physical steps** (`BuildPhysicalSteps`) — emits one `PhysicalStep` per clause it recognises: +`TableScan`, `Filter`, `Aggregate`, `Sort`, `Project`. -**Phase 5: Physical Planning** -- Coerce logical operators to physical execution steps -- Choose hash vs nested loop join -- Generate execution step sequence +> **Not part of this pipeline** (they were described here as phases 2-4 in earlier revisions): +> predicate pushdown as a *rewrite*, `EXISTS`/`NOT EXISTS`/`IN (SELECT)` conversion to semi/anti-joins, +> and join reordering. The first is only flagged, the second has no implementation (`SemiJoin` and +> `AntiJoin` do not exist in the source), and the third has algorithms on two components the engine +> does not construct. §"Optimization rules" is the authoritative table. -### Component Interaction +### Component interaction ``` -QueryOptimizer.Core -├── CostEstimator (estimated costs & cardinality) -├── PredicatePushdown (filter optimization) -├── SubqueryOptimizer (EXISTS/IN/scalar elimination) -├── JoinReorderer (join order optimization) -└── QueryOptimizerExtensions (cache integration) +Optimization.QueryOptimizer ← the engine's query path (Services/QueryCompiler.cs) +├── Optimization.CostEstimator (scan/filter/join/aggregate/sort costs) +└── Optimization.PhysicalStepType (TableScan, Filter, Aggregate, Sort, Project) + +Database/Caching/Database.PlanCaching.cs ← compiled-plan cache (normalised SQL + parameter shape) +└── Services.QueryPlanCache (ConcurrentDictionary + LRU) + +Planning/{QueryOptimizer,CardinalityEstimator,PredicatePushdown} ← building blocks, not wired in +Query/CostBasedOptimizer ← building block, not wired in +Execution/{JoinExecutor,SubqueryCache,AggregationOptimizer} ← execution side ``` ## Components