diff --git a/AGENTS.md b/AGENTS.md index 14b7c26c..fc097a62 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,8 +13,7 @@ former v2.1 RC line; see `docs/net11/PROMOTION_TO_MAIN.md`). The net10.0 / C# 14 ## Policy -- Never push. Commit locally to a feature branch such as `perf/autonomous-YYYYMMDD`; never commit to - `master` or `release/*` (Brief §9). + - Never force-push, rewrite history, delete the worklog, change `global.json`, modify release/version metadata or package versions, or pack/publish. - Never disable encryption to win, and never trade durability — WAL semantics, crash-consistency and @@ -101,6 +100,16 @@ former v2.1 RC line; see `docs/net11/PROMOTION_TO_MAIN.md`). The net10.0 / C# 14 +## Push and landing — amended by the owner, 2026-10-11 (deliberately outside the managed block) + +- The agent may **push feature branches** (`perf/autonomous-*`), open and update pull requests, and keep a + feature branch current with `master`. The blanket "Never push / never commit to `master`" line was + removed from the managed block above (which a `bmad-project-context` refresh replaces) and from Brief + §9; this section is where that decision persists. +- **Merging into `master` or `release/*` stays an owner action.** That push runs `pack` and "Push to + NuGet.org" (`.github/workflows/ci.yml`), and `pack/publish` is still a standing rail — so the agent + prepares the branch and the pull request and stops there. + ## Working rules that outrank convenience (owner directive, 2026-09-26 — deliberately outside the managed block) - **A bug you find is yours to fix, in the same session.** A finding never leaves a session as a bare diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 1afb19c2..a47ce90e 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -46,6 +46,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **The ubuntu leg of `CI/CD Pipeline` was red on `master`, and the `Coverage (net11.0)` job withheld its verdict with it, because two documentation links carried the wrong case.** `docs/Vectors/VECTOR_MIGRATION_GUIDE.md` pointed its *Documentation* and *API Reference* links at `../vectors/README.md` while the directory is `docs/Vectors`: a path that resolves on Windows and macOS — the windows and macos legs were green — and 404s on a case-sensitive host, where the link guard `DocumentationMarkdownLinks_ShouldResolveToExistingFiles` (added in this line's own documentation audit, commit `bdb4e094`, which ran on Windows and macOS) failed with `Collection: ["Vectors/VECTOR_MIGRATION_GUIDE.md:793 -> ../vector…", ":794 -> ../vector…"]`, so `tools/measure-coverage.ps1` exited 2 (`FAILED SUITES: SharpCoreDB.Tests - coverage verdict withheld`) instead of evaluating the 18 % floor. Both links now read `../Vectors/README.md` — the form every other vector link in `docs/` already used — and the guard no longer depends on the host filesystem's case sensitivity: both copies of that resolution (the link sweep and `DocumentationIndex_ShouldOnlyReferenceExistingDocumentationFiles`) go through one `PathExistsWithExactCase` helper that matches every path segment ordinally, so a wrong-case link fails on Windows too instead of only in CI. Verified red→green: with only the helper in place the test failed locally with exactly the two entries above, and passes after the link fix (documentation tests 9/0; core suite **2032 / 0 failed**, VectorSearch 248/0, EF Core 116/0, linq2DB 24/0). +- **A `VECTOR(N)` column stored nothing that could be read back: the vector case was missing from eight copies of the per-type row codec, five value tokenizers, the single-file JSON reader and both parameter binders, so a `float[]` was persisted as the literal text `"System.Single[]"`.** The breakage had three independent layers, each confirmed against the source before it was touched. (1) **Row codecs:** `DataType.Vector` was absent from every per-type switch — `Table.Serialization`'s fixed-size map, `WriteTypedValueToSpan`/`ReadTypedValueFromSpan`, `GetEncodedSize`, `EncodeVariablePayload`/`DecodeVariablePayload`, and the same switch again in `FixedWidthRecordLayout`, `InsertBatchOptimized` (writer *and* size estimate), `StreamingRowEncoder`, `BulkInsertValuePipeline`, `BinaryRowDecoder`, `PageBasedDataWriter`, `ColumnarDataReader`, `StructRow` (write, `DeserializeValue` and `DeserializeValueBoxed`), `Table.ParallelScan` and `Table.GetColumnSize` — so a vector fell through to `default`, where the writers store `value.ToString()`. (2) **Tokenizers:** the VALUES literal splitter (`SqlParser.DML.ParseInsertValues`), the UPDATE `SET` splitter, `Database.Batch.ParseValues`, `SingleFileDatabase.Batch.ParseValueList` and `SqlParserExtensions.ExtractValues` all split on every comma outside quotes, so the bracketed literal `[0.5, 0.25, 0.125, 0.0625]` arrived at the value parser as `[0.5` and raised `Invalid value '[0.5' for data type Vector: The input is not a valid Base-64 string`. `ParseInsertValues`' fast path and its documented reference loop now both honour a bracketed literal, and the string-based splitters share one `SqlInPredicate.SplitTopLevelComma`. (3) **Binding:** `ParameterBinder.FormatParameter` (and `CompiledQueryExecutor`) rendered a `float[]` parameter as `'System.Single[]'`, and the single-file store — which writes rows as JSON — read a vector back as a `JsonElement` (`"[0.5,0.25]"`) because `SingleFileTable.FromSerializableRow` converted by JSON kind only. Fixing the cause rather than nine symptoms, the format and the coercion now have one definition: `SharpCoreDB.DataStructures.VectorBinaryCodec` owns the `VEC\0` blob (`docs/Vectors/TECHNICAL_SPEC.md` §7 — magic, version 1, flags, 2 reserved bytes, LE dimension count, float32 payload), the length-prefixed framing the row codecs store, and the accepted representations (`float[]`, `Memory`/`ReadOnlyMemory`, `IEnumerable`, `double[]`, `byte[]` blob or raw float32, JSON array, base64, `0x` hex); `SharpCoreDB.VectorSearch.VectorSerializer` now delegates to it, so the engine and the HNSW/FLAT index cannot disagree about the layout. A value that is not a vector is now a type error naming the accepted forms instead of being stored as text. Two further copies were fixed on the way: the fixed-width codec's string fast path wrote raw UTF-8 into a VECTOR slot where the reader expects the blob (`types[i] is not (DataType.Blob or DataType.Vector)` now, mirroring the Blob exclusion), and `SharpCoreDB.Client`'s `GetFieldType` advertised `double[]` for a VECTOR column while every other surface produces `float[]`. Round trips are pinned by `VectorColumnStorageTests` (9), `VectorColumnRegressionTests` (11 — literal insert, binary `VEC\0` bytes present in the file, JSON-string/`float[]`/`ReadOnlyMemory`/`double[]` parameters, base64, quoted JSON, multi-row VALUES, UPDATE with two assignments, `InsertBatch`, the `StreamingRowEncoder`→`BinaryRowDecoder` pair, NULL, the single-file reopen matrix, and the two loud-failure cases) and `VectorBinaryCodecTests` (8); the storage round trips were red before the fix (8 of the 9 cases failed, first failure `Invalid value '[0.5' for data type Vector`), the 28 are green after it, and the 248-test VectorSearch suite is unchanged by the delegation. + - **The documentation set was audited against the shipped 2.1 engine, and every page that described an API, file or format the engine does not have was corrected.** The sweep diffed each backticked identifier, file path and package id in `docs/**` against the source (symbol sets, the project list, `Test-Path`) and then read every suspect site rather than inferring. What was wrong: `docs/server/CONFIGURATION_SCHEMA.md` advertised *"Supported Formats: TOML, JSON, YAML"* and built ~400 lines of underscore-keyed TOML (`server_name`, `tls_enabled`) for a server that registers **JSON configuration sources only** — no TOML or YAML parser exists anywhere in the repository — listed `sharpcoredb.toml`/`sharpcoredb.yaml` in the load order, documented `Logging`/`Performance`/`Security` property sets that do not exist, invented `[server.replication]`/`[server.monitoring]` sections no type binds (and a `SHARPCOREDB_*` environment prefix where the server uses `Server__…`), and named `sharpcoredb/server:1.7.0` against the real `ghcr.io/mpcoredeveloper/sharpcoredb-server`; it is **rewritten** from `ServerConfiguration`, `Program.cs` and the shipped `appsettings.json`, including the real startup validation (TLS cannot be disabled, 32-character JWT secret, private key required unless the certificate is a `.pfx`). `docs/internals/OPTIMIZER_GUIDE.md` documented components that do not exist — `SubqueryOptimizer` (EXISTS→semi/anti-join rewriting; `SemiJoin`/`AntiJoin` appear nowhere in the source), `JoinReorderer`, `OptimizerCache`, `QueryOptimizerExtensions`, `OptimizerIntegration.GetOrOptimize(...)`, `InvalidateOptimizationCache()`, `FilterExecutor`, `PredicateAnalyzer` — and a 2-5×/5-20×/50-1000× "Execution Impact" table; it now describes the real path (`Optimization.QueryOptimizer` + `CostEstimator`, wired from `Services/QueryCompiler.cs`), the real plan cache (`Database/Caching/Database.PlanCaching.cs` + `Services/QueryPlanCache.cs`), the real subquery subsystem (`SubqueryClassifier`/`Planner`/`Executor`/`Cache`) and the honest placeholders (`JoinsReordered = hasJoins`, `SubqueriesEliminated = !hasSubquery`, `HashJoinThreshold = int.MaxValue`, so joins are nested-loop). `docs/QUERY_PLAN_CACHE.md` presented `db.GetPlanCacheStats()` and `db.ClearPlanCache()` as consumer code; both are `internal` (`InternalsVisibleTo` reaches tests and benchmarks only), and the page now says so. `docs/manual/migration.md`/`providers.md` named a **`SharpCoreDB.FluentMigrator`** package that does not exist — the integration ships inside `SharpCoreDB.Extensions` as `AddSharpCoreDBFluentMigrator()` — and `docs/Vectors/TECHNICAL_SPEC.md` claimed `SharpCoreDB.EntityFrameworkCore` exposes `HasVectorProperty()` + `VectorDistance()` LINQ support and `Microsoft.Extensions.AI` compatibility, where the EF provider contains no vector support at all and no project references `Extensions.AI`. `docs/benchmarks/ISSUE_56_ACCEPTANCE_CRITERIA.md` carried "✅" marks for `REPORT_V1.md`, `REPORT_V2.md`, `REPRODUCING.md`, `INTERPRETING_RESULTS.md` and a committed `tests/benchmarks/results/` directory, none of which exist; it now states per-artefact status. `docs/sharpcoredb-known-issues.md` — a 1.9.3-era external report — was verified item by item against 2.1: items 2-5 are fixed (`Load()` pads all seven per-column lists, `SingleFileTable` keeps a PK hash index, `ExecuteQuery` flushes pending batch writes, `SqlQueryValidator.NormalizeKey` strips `@`/`:`), item 6 (`INTEGER` → Int32) remains the default with the documented `UseSqliteIntegerAffinity` opt-in, and the non-existent `DurabilityMode.FlushToDisk` is corrected. Dead references were also repaired in `docs/INDEX.md` (two `storage/*_v1.7.0.md`, two `migration/FLUENTMIGRATOR_*_v1.7.0.md`), `docs/serialization/README.md` (two `SERIALIZATION_DOCUMENTATION_*.md`), `docs/graphrag/EF_CORE_DOCUMENTATION_INDEX.md` (three never-written files), `docs/analytics/README.md` (a `docs/compatibility/…` page), and the `Option.FromNullOrEmpty` fiction in `docs/FUNCTIONAL_NULL_SAFETY.md` was replaced with the real `Prelude.Optional` (null-only — semantic nulls are handled by the functional read path), alongside stale `v1.7.0` titles and a pointer to a `PERFORMANCE_VALIDATION_AND_ADVICE_CSHARP15_NET11.md` that was never created. Five new `DocumentationConsistencyTests` guards pin each class of defect (INDEX paths resolve; no TOML samples or "Supported Formats: TOML" in the server guide; no internal plan-cache API shown as public; every known-issue item annotated; every install-snippet package id real), each demonstrated red by planting the defect and green after removing it. diff --git a/docs/Vectors/TECHNICAL_SPEC.md b/docs/Vectors/TECHNICAL_SPEC.md index 204c32ef..ba08d997 100644 --- a/docs/Vectors/TECHNICAL_SPEC.md +++ b/docs/Vectors/TECHNICAL_SPEC.md @@ -469,6 +469,12 @@ Header flags (future-proof): 4. **Magic bytes** prevent accidental interpretation of regular BLOBs as vectors 5. **Overhead:** 12 bytes per vector (< 0.2% for 1536-dim) — negligible +### Where the format is implemented + +One type owns it: `SharpCoreDB.DataStructures.VectorBinaryCodec` (`src/SharpCoreDB/DataStructures/VectorBinaryCodec.cs`) — `Serialize` / `WriteTo` / `Deserialize`, the `[int32 length][VEC\0 blob]` framing every row codec stores in a variable-length column, and the coercion rules (`float[]`, `Memory` / `ReadOnlyMemory`, `IEnumerable`, `double[]`, `byte[]` blob or raw float32, JSON array text, base64, `0x` hex). `SharpCoreDB.VectorSearch.VectorSerializer` delegates to it, so the engine, the similarity index and the parameter binder cannot disagree about the layout. + +A vector value reaches a column as a bracketed SQL literal (`INSERT INTO docs (id, embedding) VALUES (1, [0.5, 0.25])` — the VALUES tokenizer treats the commas inside the bracket as part of one literal), as a bound parameter (`float[]`, `ReadOnlyMemory`, a JSON array string, …), or through the object API. A value that is not one of the accepted representations is rejected with a type error naming them, rather than being stored as its `ToString()` text. + ### HNSW Index Persistence ``` diff --git a/docs/Vectors/VECTOR_MIGRATION_GUIDE.md b/docs/Vectors/VECTOR_MIGRATION_GUIDE.md index 2020fd07..fdfedafa 100644 --- a/docs/Vectors/VECTOR_MIGRATION_GUIDE.md +++ b/docs/Vectors/VECTOR_MIGRATION_GUIDE.md @@ -790,8 +790,8 @@ File.Delete("old.db"); ## Support & Resources -- **Documentation:** [Vector Search Guide](../vectors/README.md) -- **API Reference:** [Vector API Docs](../vectors/README.md#api-reference) +- **Documentation:** [Vector Search Guide](../Vectors/README.md) +- **API Reference:** [Vector API Docs](../Vectors/README.md#api-reference) - **Performance:** [Performance Benchmarks](../BENCHMARK_RESULTS.md) - **Issues:** [GitHub Issues](https://github.com/MPCoreDeveloper/SharpCoreDB/issues) diff --git a/docs/performance/AUTONOMOUS_AGENT_BRIEF.md b/docs/performance/AUTONOMOUS_AGENT_BRIEF.md index 932684e5..2dd12e4b 100644 --- a/docs/performance/AUTONOMOUS_AGENT_BRIEF.md +++ b/docs/performance/AUTONOMOUS_AGENT_BRIEF.md @@ -623,8 +623,7 @@ Add stamps only where needed to answer 5.1/5.3. - **Before every commit:** build clean + core suite green + `--gate` pass (or a documented, quiet re-run showing the verdict). -- Commit **locally** to a feature branch (e.g. `perf/autonomous-YYYYMMDD`). **Do not push.** **Do not - commit to `release/*` or `master`.** +- Commit **locally** to a feature branch (e.g. `perf/autonomous-YYYYMMDD`). - **`git` works on this box — commit every session.** Sessions 77 and 78 recorded "`git` invocations hang on this repository" and left their work (and session 79's, before it was re-measured) uncommitted on that basis. **The claim is false**: re-measured 2026-10-10 (session 79) through this harness' own shell, inline diff --git a/docs/performance/WORKLOG.md b/docs/performance/WORKLOG.md index 094df0a3..fdf6d5cb 100644 --- a/docs/performance/WORKLOG.md +++ b/docs/performance/WORKLOG.md @@ -3753,6 +3753,16 @@ Ours is monotone through measured rep 5 — **pass 8 overall** — and only then - Command(s): `--fair-ni-batch` ×2 (5 and 9 measured reps, `SHARPCOREDB_WARMUP_REPS=8`, arms A/B/C rotated every rep) · core suite ×1 · `--gate` ×1 (attempted once, after `dotnet build-server shutdown`) · `quiet-machine.ps1` ×1 - Regime: `REGIME (data dir): D:\scdb-bench-tmp` — Defender exclusion confirmed at **1,16×** by the check · `MaxFreq 100 %` · total CPU **4,5 %** · disk queue **0** · **`VERDICT: NOISY`, one finding: `WSearch` running** — `Stop-Service WSearch` **refused without elevation** (attempted once, not retried, per §5) · SQLite at its built-in reference pragma set - Verdict: **KEPT** — new optional API, no default changed, **no correctness gate missed in any rep of either run**, and the paired medians reproduce across two independent runs + +### 2026-10-11 (session 84 — addendum) — the vector fix's verification sweep and the hand-off to the owner: all 17 suites on this branch green (3533 / 0 failed), the coverage floor measured locally (45,95 % vs 18 %), and a clean fast-forward to `master` left for the owner to perform + +- Session: 1 of 1 for this addendum (item, not perf) — the session-84 entry above was written before the verification sweep, so the evidence it could not yet carry is recorded here rather than silently dropped. +- Command(s): the four CI test hosts and the thirteen CI does not run, all `Release/net11.0`, all `-filterVSTest 'Category!=Debug&Category!=Manual&Category!=Performance'` — core **2032**, VectorSearch **248**, EntityFrameworkCore **116**, Functional.Linq2DB **24**, Server.IntegrationTests **208**, Analytics **393**, CQRS **64**, EventSourcing **72**, Identity **53**, Provider.Sync **135**, Graph.Advanced **65**, Search **58**, Functional **38**, Projections **15**, HybridSearch **6**, Functional.Dapper **3**, Functional.EntityFrameworkCore **3** — **17 suites, 3533 tests, 0 failed / 0 errors / 0 skipped** (`artifacts/s84-*.txt`, tallied in `artifacts/s84-totals.txt` via `artifacts/s84-totals.ps1`) · `pwsh tools/measure-coverage.ps1 -SelfTest` → **10 / 10 passed** · `pwsh tools/measure-coverage.ps1 -Floor 0.18` → **PASS, measured 45,95 %, floor 18,00 %, exit 0** (`artifacts/s84-coverage.txt`; per-assembly: SharpCoreDB 41940 lines / 53,51 %, VectorSearch 83,62 %, EntityFrameworkCore 39,76 %, Client 10,81 %) · `pwsh tools/check-banned-simd-api.ps1 -SelfTest` → **29 / 29**, then the scan → **exit 0, 776 files, 6 comment-only mentions** · `dotnet list SharpCoreDB.CI.slnf package --deprecated --configfile NuGet.Config` and the same list with `--vulnerable` → **none in either** · `git merge-base --is-ancestor master HEAD` → **0** and `git merge-tree --write-tree master HEAD` → a tree, **no CONFLICT**. +- Regime: measurement and recording only — no source, config, threshold, package or version metadata changed by this addendum. Release/JIT, `net11.0` / C# 15 preview (`11.0.100-rc.1.26425.128`), branch `perf/autonomous-20261011`. +- Verdict: **VERIFIED, HANDED OVER.** Every step of `.github/workflows/ci.yml`'s build and coverage jobs was reproduced locally on commit `32227696` — deprecated/vulnerable package gates (the commit touches 0 project or package files), the banned-SIMD scan, the build (0 errors), the four CI suites, the coverage self-test and the 18 % floor. The pack/publish jobs were neither run nor needed: they fire on a push to `master` or a tag, which is an owner action and outside this agent's remit. Landing is a **clean fast-forward** (`master` 92794084 is an ancestor; no conflicts) that would carry **7 commits** — this fix plus the six earlier doc/perf commits already on the branch — so the owner may want to accept them together or take the fix alone. **Nothing was pushed and no commit was made to `master`** (brief §9, `AGENTS.md` policy); the merge and the push are the owner's decision and were explicitly left to them. +- Correction (same session, after this entry was written — the entry above is left as written, per append-only): the landing premise moved under it. Pull request **#447** merged the six earlier doc/perf commits into `master`, so `origin/master` is now `204d9f8a` (a merge commit) while local `master` still sits at `92794084`, 7 behind it. The `--ff-only` hand-off above and its count of 7 are **superseded**: `origin/master` is no longer an ancestor of this branch, so the fix lands as its own pull request — `32227696` (the fix) plus the worklog-addendum commit carrying this record — with *Create a merge commit*, the same path #447 took, which also preserves the SHAs this worklog cites. Re-verified after a fetch: `git rev-list --left-right --count origin/master...HEAD` → **1 / 2** · `git merge-tree --write-tree origin/master HEAD` → a tree, **no CONFLICT** · the branch's own push is a fast-forward (`origin/perf/autonomous-20261011` = `d0b0f537`, branch ahead 2 / behind 0) · working tree tracked-clean. A helper for the owner was added under the untracked `artifacts/` directory: `pwsh -File artifacts/land-vector-fix.ps1 [-Push]` — dry-run by default, pushes only the branch and never `master`, and hard-stops (`STOP:` plus a non-zero exit) on a dirty tree, a moved remote branch or a predicted conflict. The earlier instruction also cost the owner a failed terminal attempt because the `;`-separated line is PowerShell-only (invalid in `cmd.exe`) — the helper exists so quoting and shell choice stop mattering. +- NEXT: unchanged queue head — resume session 81's `NEXT:` (§5.1 INSERT, item (i): the identity-remap fast path for `InsertBatch(table, rows, columns)` whose column order already is the table order — 112 B/row on the array arm). One owner action is pending above it and is *not* a blocker: pushing `perf/autonomous-20261011` (fast-forward, 2 commits over `origin/master` — `32227696` the fix, then the worklog addendum that carries this line) and merging it as a pull request with *Create a merge commit*. Beyond that head, the EF-Core vector path this fix unblocked (`SharpCoreDBTypeMappingSource` vector mapping, `IEmbeddingGenerator` DI wiring, translators) remains feature work outside §5. + - Commit: `perf(batch-dml)`: SQL-free `UpdateBatch`/`DeleteBatch` — the INSERT fast path's siblings, and what they are worth (plan §5, session 45) - NEXT: **extend the same dial to arm B and the no-PK `docs` job** (`RunSharpCoreDBPk`'s `PkUpdateStatements`/`PkDeleteStatements` and the `docs` job's `updateStmts`/`deleteStmts` are the same shape), because 0,39×/0,38× is the cell the mission is actually about and the fair arm was already near parity; then, and only then, decide whether the fair arm's ranges justify a third run at a higher rep count. Four owner items unchanged (dead-config ×7, elevation for `--gate`, the VS C++ workload, retiring the refuted auto-index question). @@ -5120,3 +5130,70 @@ Nothing in the repository explains a stall: no `.git/*.lock`, no fsmonitor daemo - Commit: local commit on the feature branch `perf/autonomous-20261011` — **never on `master`**, never pushed (brief §9). - NEXT: resume the plan's item list from session 81's `NEXT:` line (§5.1 INSERT, its item (i): the identity-remap fast path for `InsertBatch(table, rows, columns)` whose column order already is the table order — 112 B/row on the array arm). This session added no perf item, so the queue's head is unchanged. + +### 2026-10-11 (session 84) — VECTOR(N) columns stored no vector: the vector case was missing from every per-type codec copy, the value tokenizers split bracketed literals on their commas, and both parameter binders rendered `float[]` as text — fixed in-session behind one `VectorBinaryCodec` + +- Session: 1 of 1 for this item. It is **not** a plan §5 item — it is the defect family the EF-Core vector work ran into, taken under the owner's "a bug you find is yours to fix, in the same session" duty (`.clinerules/00-bug-fix-duty.md`). The §5 queue is untouched and its head is unchanged. +- Command(s): the red baseline `tests\SharpCoreDB.Tests\bin\Release\net11.0\SharpCoreDB.Tests.exe -class 'SharpCoreDB.Tests.VectorColumnStorageTests'` → `artifacts/vector-red3.txt` (**8 of 9 failed**, first failure `Invalid value '[0.5' for data type Vector: The input is not a valid Base-64 string`, `SqlParser.Helpers.cs:213` ← `SqlParser.DML.cs:416` ← `ExecuteInsert`) · after the fix the same command → `artifacts/vector-green3.txt` (**9 / 0 failed**), `-class '…VectorColumnRegressionTests' -class '…VectorBinaryCodecTests'` → `artifacts/vector-regression1.txt` (**18 / 0 failed**), and `-filterVSTest 'FullyQualifiedName~VectorColumnStorageTests|FullyQualifiedName~VectorColumnRegressionTests|FullyQualifiedName~VectorBinaryCodecTests'` → `artifacts/vector-all.txt` (**28 / 0 failed**) · `dotnet build tests/SharpCoreDB.Tests/SharpCoreDB.Tests.csproj -c Release -f net11.0` → **0 errors**, 287 warnings (the standing baseline) · the wider core suite `-filterVSTest 'Category!=Debug&Category!=Manual&Category!=Performance'` → `artifacts/s82-vector-core-suite.txt` (**2032 / 0 failed / 0 errors / 0 skipped**, 811,3 s wall — the run overlapped this session's builds; the count is 2004 + the 28 new) · the vector subsystem's own suite `SharpCoreDB.VectorSearch.Tests.exe` → `artifacts/s82-vectorsearch-suite.txt` (**248 / 0 failed**) · `dotnet build src/SharpCoreDB.Client/SharpCoreDB.Client.csproj -c Release` → **0 errors** · source reads: `Table.Serialization.cs`, `FixedWidthCodec.cs`, `FixedWidthRecordLayout.cs`, `StructRow.cs`, `Table.ParallelScan.cs`, `Table.StructScanning.cs`, `Table.CRUD.cs`, `InsertBatchOptimized.cs`, `StreamingRowEncoder.cs`, `BinaryRowDecoder.cs`, `BulkInsertValuePipeline.cs`, `PageBasedDataWriter.cs`, `ColumnarDataReader.cs`, `SingleFileTable.cs`, `SqlParser.{DML,Helpers,Extensions}.cs`, `SqlInPredicate.cs`, `PreparedStatements.cs`, `CompiledQueryExecutor.cs`, `VectorSearch/VectorSerializer.cs`, `Client/SharpCoreDBDataReader.cs`, `Server.Core/DatabaseService.cs`. +- Regime: engine plus the VectorSearch assembly — no encryption, durability, configuration, threshold, package or version metadata touched. Release/JIT, `net11.0` / C# 15 preview (`11.0.100-rc.1.26425.128`), branch `perf/autonomous-20261011`. Nothing was measured, so no number is produced or quoted; the new tests carry no `Performance` trait and stay in the default CI run. +- Verdict: **FIXED, KEPT.** A `VECTOR(N)` column lost its value in three independent layers, each read at its site before being touched: (1) `DataType.Vector` was missing from *every* per-type switch — `Table.Serialization` (fixed-size map, `WriteTypedValueToSpan`, `ReadTypedValueFromSpan`, `GetEncodedSize`, `EncodeVariablePayload`/`DecodeVariablePayload`, `GetColumnSize`), `FixedWidthRecordLayout` (whose copy of the map was deleted; it now calls `Table.GetFixedEncodedSize`), `InsertBatchOptimized` (writer *and* size estimate), `StreamingRowEncoder`, `BulkInsertValuePipeline`, `BinaryRowDecoder`, `PageBasedDataWriter`, `ColumnarDataReader`, `StructRow` (write, `DeserializeValue`, `DeserializeValueBoxed`), `Table.ParallelScan`, `Table.StructScanning` — so the writers fell through to `default` and persisted `value.ToString()`, i.e. the literal text `System.Single[]`; (2) five value tokenizers split the bracketed literal on its commas (`SqlParser.DML.ParseInsertValues` — the fast path *and* its documented reference loop — the UPDATE `SET` splitter, `Database.Batch.ParseValues`, `SingleFileDatabase.Batch.ParseValueList`, `SqlParserExtensions.ExtractValues`), and the string-based ones now share one `SqlInPredicate.SplitTopLevelComma`; (3) both binders rendered a `float[]` parameter as `'System.Single[]'`, and the single-file store's JSON row reader returned a `JsonElement` for a VECTOR column. The format and the coercion now have one definition, `SharpCoreDB.DataStructures.VectorBinaryCodec` (`VEC\0` blob per `docs/Vectors/TECHNICAL_SPEC.md` §7, the `[int32 length][blob]` framing the row codecs store, and the accepted representations: `float[]`, `Memory`/`ReadOnlyMemory`, `IEnumerable`, `double[]`, `byte[]` blob or raw float32, JSON array, base64, `0x` hex); `VectorSearch/VectorSerializer` delegates to it (248/248 unchanged), so the engine and the HNSW/FLAT index can no longer disagree about the layout. Two further copies were fixed on the way and are recorded in `docs/CHANGELOG.md`: `FixedWidthCodec`'s string fast path wrote raw UTF-8 into a VECTOR slot the reader decodes as a blob (`types[i] is not (DataType.Blob or DataType.Vector)`), and `SharpCoreDB.Client`'s `GetFieldType` advertised `double[]` where every other surface produces `float[]`. Nothing was weakened to win: an unconvertible value is now a loud type error naming the accepted forms instead of silently storing text, a vector wider than the parallel-scan cell throws instead of being truncated, and no existing test was edited — the storage round trips went red→green as written (`VectorColumnStorageTests` 9 cases, `VectorColumnRegressionTests` 11, `VectorBinaryCodecTests` 8). +- Commit: local commit on the feature branch `perf/autonomous-20261011` — **never on `master`**, never pushed (brief §9). Evidence artefacts under `artifacts/` are not committed. +- NEXT: resume the plan's item list from session 81's `NEXT:` line (§5.1 INSERT, its item (i): the identity-remap fast path for `InsertBatch(table, rows, columns)` whose column order already is the table order — 112 B/row on the array arm). This session added no perf item, so the queue's head is unchanged. What it unblocked — the EF-Core vector path (`SharpCoreDBTypeMappingSource` vector mapping, `IEmbeddingGenerator` DI wiring, translators) — is feature work outside §5 and stays behind the queue head. + + +--- + +## Session 85 — landing + sync + tree tidy (interactive, owner present, 2026-10-11) + +- Session: 85. No measurement; the session's work is the vector fix's landing path, the sync with `master`, and the tree tidy the owner asked for. +- Policy amended by the owner **before** this session's git work: the blanket "Never push / never commit to `master`" line is gone from both places that carried it — `AGENTS.md` (its managed block) and Brief §9. Because that block is replaced on a `bmad-project-context` refresh, the decision now also lives in a new `## Push and landing` section **outside** the markers, alongside the other owner directives, so a refresh cannot quietly restore the old rule. Recorded as `d5baf541`. **Merging into `master`/`release/*` stays an owner action**: that push runs `pack` + "Push to NuGet.org" (`.github/workflows/ci.yml`) and `pack/publish` is an unamended rail. +- Sync, all three refs, each step verified: local `master` fast-forwarded `92794084 → 204d9f8a` (now identical to `origin/master`, i.e. PR #447's merge commit); `origin/master` merged into `perf/autonomous-20261011` by `ort` with no conflicts (`git merge-tree --write-tree` checked clean *before* the merge, not inferred from an earlier run) → `0ee08d10`, now `0/4` against `origin/master` (nothing on `master` is missing here); branch pushed `a21c2b71..0ee08d10`. +- Verdict: **SYNCED and GREEN on the merged tree.** Post-merge verification scripted as `artifacts/verify-postmerge.ps1`, log `artifacts/postmerge-verify.txt`: restore exit 0; CI slnf build exit 0 (302 warnings, 0 errors); the four CI-filtered suites green — `SharpCoreDB.Tests` **2032/0** (97,2 s), `SharpCoreDB.VectorSearch.Tests` **248/0**, `SharpCoreDB.EntityFrameworkCore.Tests` **116/0**, `SharpCoreDB.Functional.Linq2DB.Tests` **24/0** (2420 tests, 0 failed); deprecated gate ok; vulnerable gate ok; SIMD self-test exit 0 and scan exit 0. Before that, the canaries AGENTS.md names plus every `Vector` test: **97/0**; `DocumentationConsistencyTests` **9/0** after the two doc edits. The master content this merge brought in (the two batch-dispatcher commits) is exactly what that wide run exercised. +- Tree: **clean** — `git status --porcelain` is empty. The 19 comparative result files that session 83/84's publication commit `d5b50ebe` (15:03 local) did not track (`20261011_1131`–`1300` in UTC names, i.e. 13:31–15:00 local; 4 of the day's 23 files were tracked) were **moved, not deleted**, to `artifacts/bench-results-20261011/`: gitignored, next to the other session evidence, and the worklog cites none of them by name, so nothing became unreachable. Deleting them was never an option — they are measurement data. If they were meant to be published, they are one `Move-Item` + `git add` away, and that choice is the owner's. +- Two dependency findings, `BLOCKED` on the "never … modify … package versions" rail (owner decision; both sit outside what CI gates): + 1. **Dependabot's 15 open alerts are all npm, and all in one manifest**: `src/SharpCoreDBServerScriptClients/javascript/sharpcoredb-client/package-lock.json` — `axios` ×12 (7 high, 5 medium), `@grpc/grpc-js` ×2 (1 high, 1 low), `brace-expansion` ×1 (medium) — which is exactly the "8 high / 6 moderate / 1 low" the push banner prints (`gh api repos/MPCoreDeveloper/SharpCoreDB/dependabot/alerts?state=open`). The fix is a lockfile bump, i.e. a version change, so it goes to the owner or through Dependabot's own PRs. + 2. **The NuGet graph is clean in CI scope and not repo-wide.** `dotnet list SharpCoreDB.CI.slnf package --vulnerable --include-transitive` → no vulnerable package in any slnf project (this is why CI — and this session's gate — is green), while `dotnet list SharpCoreDB.sln package --vulnerable --include-transitive` → `SharpCoreDb.Orchardcore` (`Examples/Web/Orchardcore/`, tracked; 13 of the repo's 71 csproj files are outside the slnf) carries **`SixLabors.ImageSharp 3.1.11`, Moderate, `GHSA-gwg2-r3hj-4w44`** transitively. No code change made, for the same rail reason. The CI gate itself is **not** broken — `--configfile`, as `ci.yml` line 81 uses it, evaluates (`exit=0`, output evaluated) — it is scoped to the slnf *and* to top-level packages, which is the whole explanation for "CI green, Dependabot 15". +- Commit: `d5baf541` (the policy change), the merge `0ee08d10`, and this entry (plus a follow-up commit that moved this entry to its correct place at the end of this file — the first attempt landed mid-file because `Measure-Object -Line` does not count empty lines), all on `perf/autonomous-20261011`; branch pushed under the amended rule. `master` was **not** touched by the agent — the fix still lands through a pull request the owner merges, as **Create a merge commit**, so `32227696` survives. +- NEXT: queue head unchanged — session 81's `NEXT:` §5.1 INSERT item (i), the identity-remap fast path for `InsertBatch(table, rows, columns)` whose column order already is the table order (112 B/row on the array arm) — plus the two `BLOCKED` dependency items above (15 npm advisories in the JS client lockfile; `SixLabors.ImageSharp 3.1.11` in `Examples/Web/Orchardcore`), and the open owner action: create and merge the PR `perf/autonomous-20261011 → master` as a merge commit (it now fast-forwards the branch into `master` cleanly). + +### 2026-10-11 (session 85 — addendum) — branch hygiene: six stale local branches and one stale remote branch removed + +- Session: 85 (same session, addendum — cleanup, not measurement; no source, config, threshold, package or version metadata touched). +- Command(s): `git branch --merged origin/master` / `--no-merged` to classify every local branch, `git merge-base --is-ancestor origin/master` per candidate (`exit 0` = the *same commit objects* are reachable from master), `gh pr list --state all --limit 40` for the PR map, then `git branch -d`/`-D` and one `git push origin --delete`. +- Verdict: **TIDY.** Local branches 9 → 3; remote branches (excluding `HEAD`) 6 → 5. Every deletion was proven redundant *before* it happened and is one command from restoration (SHAs below). `git status --porcelain` empty, `HEAD` unchanged at `94c0f24e`, no workflow can break (below). +- Deleted locally, with the tip each had and where its content survives — all five are ancestors of `origin/master`, so the commit objects themselves are still reachable, not merely equal in content: `docs/promotion-outcome-20261010` `ff1742cc`, `fix/sonar-gate-20261010` `b7bd563e`, `promote/net11-main-20261010` `4c4dd391` (plain `-d`); `perf/autonomous-20260921` `887e148c` and `release/v2.1.0.0-RC.3` `82f62ec1` (needed `-D`). +- Why those two needed `-D`, recorded because the refusal looks alarming and is not: `git branch -d` refused them **not** for being unmerged — it printed "merged to HEAD" itself — but because their *upstream* remote branches are behind them (15 and 123 commits respectively) and therefore do not contain them. The safety question is answered by the ancestry check instead, and it passes: `887e148c` and `82f62ec1` are both ancestors of `origin/master` (`exit 0` each). The 15 commits of the 20260921 campaign (sessions 75-80, including the hash-index capacity-hint lever and the array INSERT attribution) reached master through PR #447, which *was* the merge of this branch's own line; and `82f62ec1` is also the exact tip of `origin/perf/backup-20260925`, so the RC.3 line keeps a remote copy besides master. +- `fix/net10-parked-ci` `c13405df` was deleted with `-D` for the opposite reason: it is *not* in master, by design — it is byte-identical to `origin/release/v2.0`'s tip, which local `release/v2.0` tracks. So the parked net10.0 line keeps both a local and a remote branch, and nothing was lost. +- Deleted on the remote: `origin/perf/autonomous-20260921` (`39d0855b`, 2026-09-27) — hard-stale and fully contained in master (that tip is an ancestor of `887e148c`, itself in master; `exit 0` verified). Safe by construction too: `Select-String` over `.github/workflows/*.yml` for `perf/autonomous` → **none**, so no pipeline depends on any such branch; the only visible effect elsewhere is one `git fetch --prune` for another clone. Output confirmed ` - [deleted] perf/autonomous-20260921`, exit 0. +- Kept deliberately, with the reason: `master`; `perf/autonomous-20261011` (it carries the six commits the landing PR needs, and is in sync with its remote); `release/v2.0` (the parked net10.0 line, built by `ci-net10.yml`, tracked and in sync); on the remote `origin/release/v2.0` and `origin/release/v2.1.0.0-RC.3` (release lines — owner territory, and §releasemetadata disallows the agent's hand there), plus `origin/perf/backup-20260925`, which is content-redundant with master but whose *name is its job*: it holds the only remote copy of the RC.3 tip. Deleting a backup is the owner's call, not hygiene. +- Restoration, if any of it turns out to be wanted: `git branch ` for a local one, `git push origin :refs/heads/` for the remote one — every SHA is in the ledger above. +- Commit: this addendum, on `perf/autonomous-20261011` (pushed; the branch currently leads `origin/master` by seven commits). +- NEXT: unchanged queue head — session 81's `NEXT:` §5.1 INSERT item (i), the identity-remap fast path for `InsertBatch(table, rows, columns)` whose column order already is the table order (112 B/row on the array arm) — plus the two `BLOCKED` dependency items from this session's main entry (15 npm advisories in the JS client's lockfile; `SixLabors.ImageSharp 3.1.11` in `Examples/Web/Orchardcore`, outside the CI slnf), and the open owner action: create and merge the PR `perf/autonomous-20261011 → master` as a merge commit (it fast-forwards the branch into `master` cleanly). + + +### 2026-10-11 (session 85 — addendum 2) — the ubuntu leg of `CI/CD Pipeline` was red on `master`: two wrong-case documentation links, fixed with a red→green test + +- Session: 85 (same session, addendum 2 — no measurement; the work is the CI defect the owner surfaced by asking for every runner green). +- Command(s): `gh run list --branch master` / `--branch perf/autonomous-20261011`, `gh run view 38160540441 --json jobs`, `gh run view 38160540441 --log-failed` (log kept at `artifacts/ci-master-failed.log`), then the local red→green via `tests\SharpCoreDB.Tests\bin\Release\net11.0\SharpCoreDB.Tests.exe -filterVSTest ''` on both sides of the fix. +- Verdict: **FIXED** — `master` at `204d9f8a` (PR #447's merge commit) failed `Build and Test (ubuntu-latest)` and, as collateral, `Coverage (net11.0)`; both are green with this addendum's change, which is why the landing PR is being opened now. +- What it actually failed on, read from the failed-step log rather than inferred: one test, `DocumentationConsistencyTests.DocumentationMarkdownLinks_ShouldResolveToExistingFiles`, `Assert.Empty() Failure: Collection was not empty`, `Collection: ["Vectors/VECTOR_MIGRATION_GUIDE.md:793 -> ../vector…", "Vectors/VECTOR_MIGRATION_GUIDE.md:794 -> ../vector…"]`. Nothing else failed anywhere: both windows and macos legs were green, `Compatibility Smoke Tests` was green, and `Pack NuGet Packages` / `Publish to NuGet.org` were *skipped* only because their `needs: build` leg had failed — on the previous green master push (`38130514269`) both ran and succeeded, so a green push to `master` publishes to NuGet.org, and that is the owner's call to make at merge time. +- Root cause, two layers — fixing only the first would have left the defect class alive: (1) the two links in `docs/Vectors/VECTOR_MIGRATION_GUIDE.md` (§ *Support & Resources*) read `../vectors/README.md` while the directory is `docs/Vectors`, a path that resolves on Windows and macOS and 404s on a case-sensitive host; every one of the ~20 other vector links in `docs/` already used `../Vectors/README.md`; (2) the guard that found it was itself host-dependent — `File.Exists`/`Directory.Exists` are case-insensitive on Windows and macOS, so the audit commit `bdb4e094` that *added* the test (it is absent at `92794084`, the last green master) passed on both of those legs and could not have failed there, which is how a red ubuntu leg was merged unnoticed. +- `Coverage (net11.0)` is collateral, not a second defect: `tools/measure-coverage.ps1` exited 2 on the same single failure (`FAILED SUITES: SharpCoreDB.Tests - coverage verdict withheld`, its documented "a suite failed" code), so the 18 % floor was never evaluated in that run. +- Fix, both layers: the two links now read `../Vectors/README.md`; both copies of the resolution — the link sweep and `DocumentationIndex_ShouldOnlyReferenceExistingDocumentationFiles`, which resolved backticked `*.md` paths the same way — now go through one new `PathExistsWithExactCase` helper that walks from the repository root and matches each segment with `StringComparison.Ordinal`, so a wrong-case link fails on a case-insensitive host too. No third copy exists: `linkPattern` / markdown-link resolution appears once across `tests/`, `scripts/` and `tools/`. +- Red→green, in that order, on this box: with only the helper in place (links still wrong) the test failed on Windows with *exactly* the two entries the ubuntu log shows — `Collection: ["Vectors\VECTOR_MIGRATION_GUIDE.md:793 -> ../vector…, ":794 -> ../vector…"]` — and passed after the link fix (`DocumentationConsistencyTests` 9/0). +- Wider suite after the fix, all four suites CI runs, CI's own filter (`Category!=Debug&Category!=Manual&Category!=Performance`): core **2032 / 0 failed / 0 skipped**, VectorSearch **248 / 0**, EF Core **116 / 0**, linq2DB **24 / 0** — 2420 tests, 0 failed. Build clean (0 errors). +- Recorded in `docs/CHANGELOG.md` under `[Unreleased] → Fixed`, because a documentation link is user-visible, and in this worklog. +- Commit: the doc fix and the test hardening as two commits that are each green on their own (fix first, then the guard that would have caught it), pushed to `perf/autonomous-20261011`, and the landing PR is opened from it so the runners verify the fix rather than this box claiming it. +- NEXT: unchanged queue head — session 81's `NEXT:` §5.1 INSERT item (i), the identity-remap fast path for `InsertBatch(table, rows, columns)` when the column order already matches the table order (112 B/row on the array arm). + + +### 2026-10-11 (session 85 — addendum 3) — verified by the runners, not by this box: PR #448 is green and the coverage floor is evaluated again + +- Session: 85 (same session, addendum 3 — the verification half of addendum 2; no source change beyond it). +- Command(s): `git push origin perf/autonomous-20261011` (`d0c75081..62908c7a`), `gh pr create --base master --head perf/autonomous-20261011` → **PR #448**, then `gh run list --branch perf/autonomous-20261011` / `gh run view --json jobs` until every run completed. +- Verdict: **GREEN on the runners.** On `62908c7a`, `CI/CD Pipeline` run `38163780903` completed with every job `success` — `Build and Test` on windows-latest, ubuntu-latest and macos-latest, and `Coverage (net11.0)` (job `114541977186`) — and `Compatibility Smoke Tests` succeeded. Both failures addendum 2 diagnosed are gone: ubuntu passes the link test, and the coverage job no longer exits 2, it evaluates the floor again: `TOTAL 60529 lines, 27819 covered, 45.96%` → `PASS measured 45.96%, floor 18.00%` (job log kept at `artifacts/ci-coverage-green.log`). +- Why a PR and not a push to `master`: CI runs on pushes to `master`/`develop` only, plus pull requests (`.github/workflows/ci.yml` lines 5-12), so this branch's own pushes were "verified" by `Automatic Dependency Submission` alone. A fix for a red runner has to be run by the runners before it can be called fixed — which is also why the red→green proof in addendum 2 was set up to fail on Windows rather than only on ubuntu. +- Confirmed while reading the skip reason, because it changes what merging means: on a green push to `master` the `pack` and `publish` jobs run (`if: github.event_name == 'push' && github.ref == 'refs/heads/master'`). They were skipped on `204d9f8a` only because their `needs: build` leg had failed, and on the previous green master push (run `38130514269`) both ran and succeeded. Merging PR #448 as a merge commit therefore publishes packages to NuGet.org — an owner action, and the reason the agent stops at the open PR. +- Commit: this addendum, on `perf/autonomous-20261011`; PR #448 is left open and green for the owner. +- NEXT: unchanged queue head — session 81's `NEXT:` §5.1 INSERT item (i), the identity-remap fast path for `InsertBatch(table, rows, columns)` when the column order already matches the table order (112 B/row on the array arm). + diff --git a/src/SharpCoreDB.Client/SharpCoreDBDataReader.cs b/src/SharpCoreDB.Client/SharpCoreDBDataReader.cs index e59fbcbb..ffa90ea2 100644 --- a/src/SharpCoreDB.Client/SharpCoreDBDataReader.cs +++ b/src/SharpCoreDB.Client/SharpCoreDBDataReader.cs @@ -126,7 +126,11 @@ public override Type GetFieldType(int ordinal) DataType.Guid => typeof(string), DataType.Ulid => typeof(string), DataType.Rowref => typeof(string), - DataType.Vector => typeof(double[]), + + // float[], not double[]: a VECTOR column is a float32 array everywhere else in the engine + // (VectorBinaryCodec, VectorTypeProvider, the vector binary format), so the client must not + // advertise a CLR type the reader can never produce. + DataType.Vector => typeof(float[]), DataType.Decimal => typeof(decimal), _ => typeof(object) }; diff --git a/src/SharpCoreDB.VectorSearch/VectorSerializer.cs b/src/SharpCoreDB.VectorSearch/VectorSerializer.cs index 73220a69..60efaabc 100644 --- a/src/SharpCoreDB.VectorSearch/VectorSerializer.cs +++ b/src/SharpCoreDB.VectorSearch/VectorSerializer.cs @@ -5,83 +5,38 @@ namespace SharpCoreDB.VectorSearch; using System.Runtime.CompilerServices; -using System.Runtime.InteropServices; -using System.Text.Json; +using SharpCoreDB.DataStructures; /// -/// High-performance vector serialization using zero-copy memory reinterpretation. -/// Converts between float[] and byte[] without element-by-element copying. +/// High-performance vector serialization for the similarity-search index. +/// +/// The binary format itself lives in one place — in the core assembly — and +/// every method here delegates to it. Before that, this class was one of eight independent copies of the +/// vector layout, and a vector written by the engine's row codec was unreadable to the index (and the other +/// way round) as soon as either copy changed. +/// /// public static class VectorSerializer { - /// Magic bytes identifying a SharpCoreDB vector binary blob. - private static ReadOnlySpan Magic => "VEC\0"u8; - - /// Current binary format version. - private const byte FormatVersion = 1; - /// Header size in bytes: 4 (magic) + 1 (version) + 1 (flags) + 2 (reserved) + 4 (dims) = 12. - internal const int HeaderSize = 12; + internal const int HeaderSize = VectorBinaryCodec.HeaderSize; /// /// Serializes a float vector to the SharpCoreDB vector binary format. /// Format: [Magic:4][Version:1][Flags:1][Reserved:2][Dims:4][float32 data]. /// [MethodImpl(MethodImplOptions.AggressiveOptimization)] - public static byte[] Serialize(ReadOnlySpan vector) - { - int dataBytes = vector.Length * sizeof(float); - var result = new byte[HeaderSize + dataBytes]; - var span = result.AsSpan(); - - // Write header - Magic.CopyTo(span); - span[4] = FormatVersion; - span[5] = 0; // flags (reserved for quantization) - span[6] = 0; // reserved - span[7] = 0; // reserved - BitConverter.TryWriteBytes(span[8..], vector.Length); - - // Write float data via zero-copy reinterpret - MemoryMarshal.AsBytes(vector).CopyTo(span[HeaderSize..]); - - return result; - } + public static byte[] Serialize(ReadOnlySpan vector) => VectorBinaryCodec.Serialize(vector); /// - /// Deserializes a SharpCoreDB vector binary blob back to float[]. + /// Deserializes a SharpCoreDB vector binary blob back to float[]. Data without the magic header is read as + /// raw float32 bytes (external embeddings), the tolerance this method always had. /// [MethodImpl(MethodImplOptions.AggressiveOptimization)] public static float[] Deserialize(byte[] data) { ArgumentNullException.ThrowIfNull(data); - - if (data.Length < HeaderSize) - { - // Fallback: assume raw float data without header (backward compat / external data) - return DeserializeRaw(data); - } - - var span = data.AsSpan(); - - // Check magic bytes - if (!span[..4].SequenceEqual(Magic)) - { - // Not our format — treat as raw float bytes - return DeserializeRaw(data); - } - - int dims = BitConverter.ToInt32(span[8..12]); - int expectedBytes = dims * sizeof(float); - int availableBytes = data.Length - HeaderSize; - - if (availableBytes < expectedBytes) - { - throw new InvalidOperationException( - $"Vector data corrupted: expected {expectedBytes} bytes for {dims} dimensions, got {availableBytes}"); - } - - return MemoryMarshal.Cast(span.Slice(HeaderSize, expectedBytes)).ToArray(); + return VectorBinaryCodec.Deserialize(data); } /// @@ -90,7 +45,8 @@ public static float[] Deserialize(byte[] data) [MethodImpl(MethodImplOptions.AggressiveInlining)] public static float[] DeserializeRaw(byte[] data) { - return MemoryMarshal.Cast(data.AsSpan()).ToArray(); + ArgumentNullException.ThrowIfNull(data); + return VectorBinaryCodec.DeserializeRaw(data); } /// @@ -99,15 +55,7 @@ public static float[] DeserializeRaw(byte[] data) /// [MethodImpl(MethodImplOptions.AggressiveInlining)] public static ReadOnlySpan DeserializeSpan(ReadOnlySpan data) - { - if (data.Length >= HeaderSize && data[..4].SequenceEqual(Magic)) - { - int dims = BitConverter.ToInt32(data[8..12]); - return MemoryMarshal.Cast(data.Slice(HeaderSize, dims * sizeof(float))); - } - - return MemoryMarshal.Cast(data); - } + => VectorBinaryCodec.DeserializeSpan(data); /// /// Parses a JSON float array string (e.g., "[0.1, 0.2, 0.3]") to float[]. @@ -115,29 +63,18 @@ public static ReadOnlySpan DeserializeSpan(ReadOnlySpan data) public static float[] FromJson(string json) { ArgumentException.ThrowIfNullOrWhiteSpace(json); - return JsonSerializer.Deserialize(json) - ?? throw new ArgumentException("Invalid vector JSON: expected a float array"); + return VectorBinaryCodec.TryParse(json, out var vector) + ? vector + : throw new ArgumentException("Invalid vector JSON: expected a float array"); } /// /// Converts a float vector to a JSON array string. /// - public static string ToJson(ReadOnlySpan vector) - { - return JsonSerializer.Serialize(vector.ToArray()); - } + public static string ToJson(ReadOnlySpan vector) => VectorBinaryCodec.ToJson(vector); /// /// Gets the number of dimensions stored in a serialized vector blob. /// - public static int GetDimensions(ReadOnlySpan data) - { - if (data.Length >= HeaderSize && data[..4].SequenceEqual(Magic)) - { - return BitConverter.ToInt32(data[8..12]); - } - - // Raw format: infer from byte count - return data.Length / sizeof(float); - } + public static int GetDimensions(ReadOnlySpan data) => VectorBinaryCodec.GetDimensions(data); } diff --git a/src/SharpCoreDB/DataStructures/FixedWidthCodec.cs b/src/SharpCoreDB/DataStructures/FixedWidthCodec.cs index 1d3ace7f..d6740edb 100644 --- a/src/SharpCoreDB/DataStructures/FixedWidthCodec.cs +++ b/src/SharpCoreDB/DataStructures/FixedWidthCodec.cs @@ -49,7 +49,10 @@ public static byte[] SerializeRow( { WriteNullVariableSlot(slot); } - else if (value is string text && types[i] != DataType.Blob + // A string fast path is only valid where the slot is *read back as text*. Blob excluded it + // already; so must Vector — a JSON vector literal handed in as a string has to go through + // EncodeVariablePayload, which writes the VEC\0 blob the reader decodes, not raw UTF-8. + else if (value is string text && types[i] is not (DataType.Blob or DataType.Vector) && TryWriteInlineStringSlot(slot, layout, text)) { continue; // encoded into the slot directly: no throw-away payload array @@ -119,7 +122,7 @@ public static byte[] SerializeRow( { WriteNullVariableSlot(slot); } - else if (value is string text && types[i] != DataType.Blob + else if (value is string text && types[i] is not (DataType.Blob or DataType.Vector) && TryWriteInlineStringSlot(slot, layout, text)) { continue; // encoded into the slot directly: no throw-away payload array diff --git a/src/SharpCoreDB/DataStructures/FixedWidthRecordLayout.cs b/src/SharpCoreDB/DataStructures/FixedWidthRecordLayout.cs index a76852d0..71a82508 100644 --- a/src/SharpCoreDB/DataStructures/FixedWidthRecordLayout.cs +++ b/src/SharpCoreDB/DataStructures/FixedWidthRecordLayout.cs @@ -61,7 +61,7 @@ public static FixedWidthRecordLayout Compute(IReadOnlyList columnTypes int offset = 0; for (int i = 0; i < count; i++) { - int fixedSize = GetFixedEncodedSize(columnTypes[i]); + int fixedSize = Table.GetFixedEncodedSize(columnTypes[i]); if (fixedSize < 0) { // Variable-length column: [null-flag(1)][overflow offset(4)] plus, when enabled, the inline @@ -90,19 +90,4 @@ public static FixedWidthRecordLayout Compute(IReadOnlyList columnTypes InlineValueBytes = inline }; } - - // Slot size including the 1-byte null flag — must match Table.Serialization.GetFixedEncodedSize. - private static int GetFixedEncodedSize(DataType type) => type switch - { - DataType.Integer => 5, - DataType.Long => 9, - DataType.RowRef => 9, - DataType.Real => 9, - DataType.Boolean => 2, - DataType.DateTime => 9, - DataType.Decimal => 17, - DataType.Ulid => 31, - DataType.Guid => 17, - _ => -1 // String / Blob are variable-length - }; } diff --git a/src/SharpCoreDB/DataStructures/StructRow.cs b/src/SharpCoreDB/DataStructures/StructRow.cs index b93a2234..2195c50f 100644 --- a/src/SharpCoreDB/DataStructures/StructRow.cs +++ b/src/SharpCoreDB/DataStructures/StructRow.cs @@ -207,6 +207,16 @@ private static byte[] SerializeValue(object? value, DataType type) bytes.CopyTo(buffer, 1); return buffer; } + case DataType.Vector: + { + // [null flag][int32 length][VEC\0 blob] — the row codec's framing, from the one vector codec. + var payload = VectorBinaryCodec.Serialize(VectorBinaryCodec.ToVector(value)); + var buffer = new byte[5 + payload.Length]; + buffer[0] = 1; + BinaryPrimitives.WriteInt32LittleEndian(buffer.AsSpan(1, 4), payload.Length); + payload.CopyTo(buffer.AsSpan(5)); + return buffer; + } default: { byte[] payload = type == DataType.Blob && value is byte[] blob @@ -505,6 +515,12 @@ private static T DeserializeValue(ReadOnlySpan data, DataType type) throw new InvalidCastException($"Cannot cast {type} to {typeof(T)}"); return (T)(object)valueData.ToArray(); } + else if (type == DataType.Vector) + { + if (typeof(T) != typeof(float[])) + throw new InvalidCastException($"Cannot cast {type} to {typeof(T)}"); + return (T)(object)VectorBinaryCodec.ReadLengthPrefixed(valueData, out _); + } else throw new NotSupportedException($"Type {type} not supported for deserialization to {typeof(T)}"); } @@ -534,6 +550,7 @@ private static object DeserializeValueBoxed(ReadOnlySpan data, DataType ty DataType.String => DeserializeString(valueData), DataType.Ulid => DeserializeUlid(valueData), DataType.Blob => valueData.ToArray(), + DataType.Vector => VectorBinaryCodec.ReadLengthPrefixed(valueData, out _), _ => DeserializeString(valueData) // Default to string }; } diff --git a/src/SharpCoreDB/DataStructures/Table.ParallelScan.cs b/src/SharpCoreDB/DataStructures/Table.ParallelScan.cs index 94f295ec..e2ab773e 100644 --- a/src/SharpCoreDB/DataStructures/Table.ParallelScan.cs +++ b/src/SharpCoreDB/DataStructures/Table.ParallelScan.cs @@ -444,6 +444,26 @@ public StructRowEnumerable SelectStructParallel(string? where = null, string? or var guid = value is Guid g ? g : Guid.Parse(value.ToString() ?? string.Empty); guid.TryWriteBytes(cellSpan.Slice(1, 16)); break; + case DataType.Vector: + { + // [null flag][int32 length][VEC\0 blob], the same framing the row codec uses. The + // cell was sized by GetColumnSize(DataType.Vector); a vector wider than that cell is + // refused loudly — silently truncating it (what the shared default branch below does + // with an over-long payload) would leave an unreadable blob in the buffer. + var vector = VectorBinaryCodec.ToVector(value); + int vectorLength = VectorBinaryCodec.EncodedLength(vector); + if (cellSpan.Length < 5 + vectorLength) + { + throw new InvalidOperationException( + $"VECTOR value of {vectorLength} bytes does not fit the parallel-scan cell " + + $"({cellSpan.Length} bytes); read the column with the dictionary path " + + "(ExecuteQuery) instead of SelectStructParallel."); + } + + BinaryPrimitives.WriteInt32LittleEndian(cellSpan.Slice(1, 4), vectorLength); + _ = VectorBinaryCodec.WriteTo(cellSpan.Slice(5), vector); + break; + } default: var payload = dataType == DataType.Blob && value is byte[] blob ? blob diff --git a/src/SharpCoreDB/DataStructures/Table.Serialization.cs b/src/SharpCoreDB/DataStructures/Table.Serialization.cs index 610ef456..7dbcbc4e 100644 --- a/src/SharpCoreDB/DataStructures/Table.Serialization.cs +++ b/src/SharpCoreDB/DataStructures/Table.Serialization.cs @@ -144,8 +144,13 @@ private int[] GetColumnOffsetsCached() return offsets; } - /// Fixed encoded size of a column (1 null flag + payload), or -1 for variable-length types. - private static int GetFixedEncodedSize(DataType type) => type switch + /// + /// Fixed encoded size of a column (1 null flag + payload), or -1 for variable-length types. + /// writes a vector as [int32 length][blob] — like Blob — so VECTOR + /// is variable-length and gets the same -1. Single definition: calls + /// this one instead of keeping its own copy of the map. + /// + internal static int GetFixedEncodedSize(DataType type) => type switch { DataType.Integer => 5, DataType.Long => 9, @@ -156,7 +161,7 @@ private int[] GetColumnOffsetsCached() DataType.Decimal => 17, DataType.Ulid => 31, DataType.Guid => 17, - _ => -1, // String, Blob, other: variable + _ => -1, // String, Blob, Vector, other: variable }; /// Encoded size (in bytes) of a value for the given column type. @@ -181,6 +186,9 @@ private static int GetEncodedSize(object? value, DataType type) return 5 + System.Text.Encoding.UTF8.GetByteCount((string)value); case DataType.Blob: return 5 + ((byte[])value).Length; + case DataType.Vector: + // [null flag 1][int32 length 4][VEC\0 blob] — must match WriteTypedValueToSpan exactly. + return 5 + VectorBinaryCodec.EncodedLength(VectorBinaryCodec.ToVector(value)); default: return 5 + System.Text.Encoding.UTF8.GetByteCount(value.ToString() ?? string.Empty); } @@ -497,6 +505,10 @@ internal static byte[] EncodeVariablePayload(DataType type, object value) return type switch { DataType.Blob => (byte[])value, + + // The overflow-arena payload *is* the vector binary format, so the fixed-width path and the + // inline path store the same bytes the row codec writes (Table.Serialization). + DataType.Vector => VectorBinaryCodec.Serialize(VectorBinaryCodec.ToVector(value)), _ => System.Text.Encoding.UTF8.GetBytes(value?.ToString() ?? string.Empty), }; } @@ -506,6 +518,7 @@ internal static object DecodeVariablePayload(DataType type, byte[] payload) return type switch { DataType.Blob => payload, + DataType.Vector => VectorBinaryCodec.Deserialize(payload), _ => System.Text.Encoding.UTF8.GetString(payload), }; } @@ -1369,6 +1382,12 @@ private int EstimateRowSize(Dictionary row) bytesWritten += blobBytes.Length; break; + case DataType.Vector: + bytesWritten += VectorBinaryCodec.WriteLengthPrefixed( + buffer.Slice(bytesWritten), VectorBinaryCodec.ToVector(value)); + break; + + case DataType.String: // ✅ PHASE 1: Zero-allocation string encoding using GetBytes(chars, Span) bytesWritten += WriteStringZeroAlloc(buffer.Slice(bytesWritten), (string)value); @@ -1514,6 +1533,14 @@ private int EstimateRowSize(Dictionary row) bytesRead += 4 + blobLen; return buffer.Slice(5, blobLen).ToArray(); + case DataType.Vector: + // Same framing as Blob — [int32 length][payload] — where the payload is the VEC\0 blob, so + // the reader and the writer cannot disagree about the vector layout. + var vectorValue = VectorBinaryCodec.ReadLengthPrefixed(buffer.Slice(1), out int vectorBytes); + bytesRead += vectorBytes; + return vectorValue; + + case DataType.String: if (buffer.Length < 5) // 1 byte null flag + 4 bytes length throw new InvalidOperationException( @@ -1622,6 +1649,12 @@ private static int GetColumnSize(DataType type) DataType.Guid => 17, // 1 null flag + 16 bytes DataType.String => 4 + 256, // 1 null flag + 4 length + estimated 256 bytes DataType.Blob => 4 + 1024, // 1 null flag + 4 length + estimated 1024 bytes + + // The fast-path StructRow buffer needs one fixed cell width per column, but vectors are + // variable-length: this is the width the documented 1536-dimension embedding needs + // (docs/Vectors/TECHNICAL_SPEC.md) — 1 null flag + 4 length + 12 header + 1536 float32. + // SelectStructParallel refuses a wider vector instead of truncating it (see Table.ParallelScan). + DataType.Vector => 5 + VectorBinaryCodec.EncodedLength(1536), _ => 4 + 256 // default estimate }; } @@ -1774,6 +1807,10 @@ private static bool IsValidType(object value, DataType type) DataType.Guid => value is Guid, DataType.RowRef => value is long, DataType.Blob => value is byte[], + // A vector accepts the representations VectorBinaryCodec coerces (float[], Memory, + // IEnumerable, double[], byte[], JSON-array or base64 text); anything else is a type + // mismatch instead of being stored as value.ToString(). + DataType.Vector => VectorBinaryCodec.TryToVector(value, out _), _ => true, }; } @@ -1926,6 +1963,16 @@ private static bool TryCoerceValue(object value, DataType targetType, out object return true; } break; + + case DataType.Vector: + // Accepts float[] and friends, a byte[] blob, and the JSON-array / base64 text forms, so a + // client can bind a vector as float[], ReadOnlyMemory or a string. + if (VectorBinaryCodec.TryToVector(value, out var vectorValue)) + { + coercedValue = vectorValue; + return true; + } + break; case DataType.Ulid: if (value is string strUlid) diff --git a/src/SharpCoreDB/DataStructures/Table.StructScanning.cs b/src/SharpCoreDB/DataStructures/Table.StructScanning.cs index 61fedabf..5a4aa7d4 100644 --- a/src/SharpCoreDB/DataStructures/Table.StructScanning.cs +++ b/src/SharpCoreDB/DataStructures/Table.StructScanning.cs @@ -1039,6 +1039,7 @@ private static (int size, bool isVariable) GetColumnSizeAndVariability(DataType DataType.String => (-1, true), // 1 null flag + 4 length + variable DataType.Blob => (-1, true), // 1 null flag + 4 length + variable DataType.Ulid => (-1, true), // 1 null flag + 4 length + variable (stored as string) + DataType.Vector => (-1, true), // 1 null flag + 4 length + VEC\0 blob (variable) _ => (-1, true) // Unknown = variable }; } diff --git a/src/SharpCoreDB/DataStructures/VectorBinaryCodec.cs b/src/SharpCoreDB/DataStructures/VectorBinaryCodec.cs new file mode 100644 index 00000000..4a704226 --- /dev/null +++ b/src/SharpCoreDB/DataStructures/VectorBinaryCodec.cs @@ -0,0 +1,400 @@ +// +// Copyright (c) 2026 MPCoreDeveloper and GitHub Copilot. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// +namespace SharpCoreDB.DataStructures; + +using System.Buffers.Binary; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; +using System.Text.Json; + +/// +/// The one definition of the SharpCoreDB vector binary format (docs/Vectors/TECHNICAL_SPEC.md §7: +/// magic VEC\0, version 1, a flags byte, 2 reserved bytes, a 4-byte little-endian dimension count, +/// then the float32 payload), and the one place a CLR value is coerced to a array for a +/// column. +/// +/// Every row codec — Table.WriteTypedValueToSpan/ReadTypedValueFromSpan, the variable-payload +/// encoder, the batched writers (InsertBatchOptimized, StreamingRowEncoder, +/// BulkInsertValuePipeline, PageBasedDataWriter), the decoders (BinaryRowDecoder, +/// ColumnarDataReader), SharpCoreDB.VectorSearch.VectorSerializer and the prepared-statement +/// binder — routes through this type. The vector case used to be missing from eight separate per-type +/// switches, so a float[] inserted into a VECTOR(N) column fell through the serializer's +/// default branch and was persisted as the literal text "System.Single[]" (while a +/// bracketed literal failed earlier still, in the VALUES tokenizer). One writer, one reader, one coercion +/// rule set; the format cannot drift between the call sites again. +/// +/// +public static class VectorBinaryCodec +{ + /// Magic bytes identifying a SharpCoreDB vector binary blob. + private static ReadOnlySpan Magic => "VEC\0"u8; + + /// Binary format version written into the blob header. + public const byte FormatVersion = 1; + + /// Header size in bytes: magic(4) + version(1) + flags(1) + reserved(2) + dims(4). + public const int HeaderSize = 12; + + /// Largest payload the row codecs accept, matching every other 100 MB guard in them. + public const int MaxEncodedBytes = 1024 * 1024 * 100; + + /// Bytes a serialized vector of dimensions occupies. + /// Number of float32 elements. + /// Header plus payload size in bytes. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int EncodedLength(int dimensions) => HeaderSize + (dimensions * sizeof(float)); + + /// Bytes a serialized occupies. + /// The vector. + /// Header plus payload size in bytes. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int EncodedLength(ReadOnlySpan vector) => HeaderSize + (vector.Length * sizeof(float)); + + /// + /// Serializes a float vector to the vector binary format. + /// Format: [Magic:4][Version:1][Flags:1][Reserved:2][Dims:4][float32 data]. + /// + /// The vector to serialize. + /// The encoded blob. + [MethodImpl(MethodImplOptions.AggressiveOptimization)] + public static byte[] Serialize(ReadOnlySpan vector) + { + var result = new byte[EncodedLength(vector)]; + _ = WriteTo(result, vector); + return result; + } + + /// Writes the vector binary format into . + /// Destination buffer, at least bytes. + /// The vector to write. + /// Number of bytes written. + [MethodImpl(MethodImplOptions.AggressiveOptimization)] + public static int WriteTo(Span destination, ReadOnlySpan vector) + { + int total = EncodedLength(vector); + if (destination.Length < total) + { + throw new InvalidOperationException( + $"Buffer too small to write Vector: need {total} bytes, have {destination.Length}"); + } + + Magic.CopyTo(destination); + destination[4] = FormatVersion; + destination[5] = 0; // flags (reserved for quantization) + destination[6] = 0; // reserved + destination[7] = 0; // reserved + BinaryPrimitives.WriteInt32LittleEndian(destination[8..], vector.Length); + MemoryMarshal.AsBytes(vector).CopyTo(destination[HeaderSize..]); + + return total; + } + + /// Whether starts with the vector magic and is long enough to hold a header. + /// Candidate blob. + /// true when the data is in the enveloped vector format. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static bool HasHeader(ReadOnlySpan data) + => data.Length >= HeaderSize && data[..4].SequenceEqual(Magic); + + /// + /// Deserializes a vector blob. Data without the magic bytes is read as raw float32 payload, so externally + /// produced embeddings stay readable (the same tolerance VectorSerializer always had). + /// + /// The encoded blob, enveloped or raw. + /// The vector. + [MethodImpl(MethodImplOptions.AggressiveOptimization)] + public static float[] Deserialize(ReadOnlySpan data) + { + if (!HasHeader(data)) + { + // Raw float32 payload: a trailing partial element cannot be interpreted, and silently dropping it + // would shorten a vector — which corrupts similarity search — so it is rejected instead. + if (data.Length % sizeof(float) != 0) + { + throw new InvalidOperationException( + $"Vector data corrupted: {data.Length} bytes is not a whole number of float32 elements"); + } + + return DeserializeRaw(data); + } + + int dimensions = GetDimensions(data); + int expectedBytes = dimensions * sizeof(float); + int availableBytes = data.Length - HeaderSize; + if (dimensions < 0 || expectedBytes > MaxEncodedBytes || availableBytes < expectedBytes) + { + throw new InvalidOperationException( + $"Vector data corrupted: expected {expectedBytes} bytes for {dimensions} dimensions, got {availableBytes}"); + } + + return MemoryMarshal.Cast(data.Slice(HeaderSize, expectedBytes)).ToArray(); + } + + /// + /// Reads the payload as raw float32 data, ignoring any header. A trailing partial float32 element is + /// dropped, which is the tolerance the raw path has always had for externally produced byte payloads. + /// + /// Raw float32 little-endian bytes. + /// The vector. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static float[] DeserializeRaw(ReadOnlySpan data) + => MemoryMarshal.Cast(data).ToArray(); + + /// + /// Zero-copy view over the float payload of a serialized blob. Only valid while the source is alive. + /// + /// The encoded blob, enveloped or raw. + /// A span over the payload. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static ReadOnlySpan DeserializeSpan(ReadOnlySpan data) + { + if (HasHeader(data)) + { + int dimensions = GetDimensions(data); + int payloadBytes = Math.Min(dimensions * sizeof(float), data.Length - HeaderSize); + return MemoryMarshal.Cast(data.Slice(HeaderSize, payloadBytes)); + } + + return MemoryMarshal.Cast(data); + } + + /// Gets the number of dimensions stored in a serialized blob. + /// The encoded blob, enveloped or raw. + /// The dimension count; for raw payloads it is inferred from the byte count. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static int GetDimensions(ReadOnlySpan data) + => HasHeader(data) ? BinaryPrimitives.ReadInt32LittleEndian(data[8..]) : data.Length / sizeof(float); + + /// + /// Writes the length-prefixed form the row codecs store in a variable-length column: + /// [int32 length][vector blob]. Paired with , + /// so no call site can get the framing wrong on its own. + /// + /// Destination buffer. + /// The vector to write. + /// Number of bytes written (4 + blob length). + [MethodImpl(MethodImplOptions.AggressiveOptimization)] + public static int WriteLengthPrefixed(Span destination, ReadOnlySpan vector) + { + int payload = EncodedLength(vector); + if (destination.Length < 4 + payload) + { + throw new InvalidOperationException( + $"Buffer too small for Vector write: need {4 + payload} bytes, have {destination.Length}"); + } + + BinaryPrimitives.WriteInt32LittleEndian(destination, payload); + return 4 + WriteTo(destination[4..], vector); + } + + /// Reads the length-prefixed form written by . + /// Source buffer positioned at the length prefix. + /// Output: bytes consumed, including the 4-byte prefix. + /// The vector. + [MethodImpl(MethodImplOptions.AggressiveOptimization)] + public static float[] ReadLengthPrefixed(ReadOnlySpan source, out int bytesRead) + { + if (source.Length < 4) + { + throw new InvalidOperationException( + $"Buffer too small for Vector length: need 4 bytes, have {source.Length}"); + } + + int length = BinaryPrimitives.ReadInt32LittleEndian(source); + if (length < HeaderSize || length > MaxEncodedBytes) + { + throw new InvalidOperationException( + $"Invalid Vector length: {length} (expected {HeaderSize}-{MaxEncodedBytes})"); + } + + if (source.Length < 4 + length) + { + throw new InvalidOperationException( + $"Buffer too small for Vector data: need {4 + length} bytes, have {source.Length}"); + } + + bytesRead = 4 + length; + return Deserialize(source.Slice(4, length)); + } + + /// + /// Parses a vector from its textual form: a JSON float array ([0.5, 0.25, 0.125]) or a + /// base64-encoded blob (enveloped or raw float32 bytes). Single or double quotes around the text are + /// stripped, so a quoted literal from a SQL statement parses the same as a bare one. + /// + /// The text to parse. + /// Output: the parsed vector. + /// true when the text was a recognised vector representation. + public static bool TryParse(string? text, [NotNullWhen(true)] out float[]? vector) + { + vector = null; + if (string.IsNullOrWhiteSpace(text)) + { + return false; + } + + text = text.Trim(); + while (text.Length >= 2 && + ((text.StartsWith('\'') && text.EndsWith('\'')) || (text.StartsWith('"') && text.EndsWith('"')))) + { + text = text[1..^1].Trim(); + } + + if (text.Length == 0) + { + return false; + } + + if (text.StartsWith('[') && text.EndsWith(']')) + { + try + { + var parsed = AotJsonSerializer.Deserialize(text); + if (parsed is null) + { + return false; + } + + vector = parsed; + return true; + } + catch (JsonException) + { + return false; + } + } + + // 0x-prefixed hex blob: the form ParameterBinder renders a byte[] parameter in, and the form a + // byte[] blob literal is written as in SQL. + if (text.StartsWith("0x", StringComparison.OrdinalIgnoreCase)) + { + try + { + var hex = text[2..]; + if (hex.Length == 0 || hex.Length % 2 != 0) + { + return false; + } + + var hexBytes = Convert.FromHexString(hex); + if (hexBytes.Length == 0 || hexBytes.Length % sizeof(float) != 0) + { + return false; + } + + vector = Deserialize(hexBytes); + return true; + } + catch (FormatException) + { + return false; + } + } + + // Base64 fallback. The byte count must be a whole number of float32 elements, so a typo cannot + // silently shorten the vector. + try + { + var bytes = Convert.FromBase64String(text); + if (bytes.Length == 0 || bytes.Length % sizeof(float) != 0) + { + return false; + } + + vector = Deserialize(bytes); + return true; + } + catch (FormatException) + { + return false; + } + } + + /// + /// Coerces a CLR value to the array a column stores. + /// Accepted: float[], Memory<float> / ReadOnlyMemory<float>, + /// any IEnumerable<float> (e.g. List<float>), double[] / + /// IEnumerable<double>, byte[] (vector blob or raw float32), and the textual forms + /// accepts. + /// + /// The value to coerce. + /// Output: the coerced vector. + /// true when the value could be coerced. + public static bool TryToVector(object? value, [NotNullWhen(true)] out float[]? vector) + { + switch (value) + { + case null or DBNull: + vector = null; + return false; + case float[] floats: + vector = floats; + return true; + case Memory memory: + vector = memory.ToArray(); + return true; + case ReadOnlyMemory readOnlyMemory: + vector = readOnlyMemory.ToArray(); + return true; + case IReadOnlyList list: + vector = [.. list]; + return true; + case IEnumerable floatsEnumerable: + vector = [.. floatsEnumerable]; + return true; + case double[] doubles: + vector = Array.ConvertAll(doubles, static d => (float)d); + return true; + case IEnumerable doublesEnumerable: + { + var buffer = new List(); + foreach (var element in doublesEnumerable) + { + buffer.Add((float)element); + } + + vector = [.. buffer]; + return true; + } + + case byte[] bytes: + vector = Deserialize(bytes); + return true; + case string text: + return TryParse(text, out vector); + default: + vector = null; + return false; + } + } + + /// + /// Coerces or throws with the list of accepted representations. Used by the row + /// codecs, where silently storing value.ToString() is what lost vector data before. + /// + /// The value to coerce. + /// Column name, for the error message. + /// The coerced vector. + public static float[] ToVector(object? value, string? columnName = null) + { + if (TryToVector(value, out var vector)) + { + return vector; + } + + var where = columnName is null ? string.Empty : $" for column '{columnName}'"; + throw new InvalidOperationException( + $"Cannot store {value?.GetType().Name ?? "null"} in a VECTOR column{where}. Accepted: float[], " + + "Memory/ReadOnlyMemory, IEnumerable, double[], byte[] (VEC\\0 blob or raw " + + "float32 bytes), a JSON array string like \"[0.5, 0.25]\" or a base64 string."); + } + + /// Formats a vector as the JSON array text the SQL literal and parameter paths accept. + /// The vector to format. + /// A JSON float array. + public static string ToJson(ReadOnlySpan vector) => AotJsonSerializer.Serialize(vector.ToArray()); +} diff --git a/src/SharpCoreDB/Database/Execution/Database.Batch.cs b/src/SharpCoreDB/Database/Execution/Database.Batch.cs index 76c5e8f5..ecf832c2 100644 --- a/src/SharpCoreDB/Database/Execution/Database.Batch.cs +++ b/src/SharpCoreDB/Database/Execution/Database.Batch.cs @@ -113,6 +113,7 @@ public Dictionary ParseValues(ReadOnlySpan valuesClause) int valueIndex = 0; bool inQuotes = false; int parenDepth = 0; + int bracketDepth = 0; // bracketed literals ([0.5, 0.25] — a VECTOR value) never split for (int i = 0; i < valuesClause.Length && valueIndex < Columns.Count; i++) { @@ -126,7 +127,9 @@ public Dictionary ParseValues(ReadOnlySpan valuesClause) { if (c == '(') parenDepth++; else if (c == ')') parenDepth--; - else if (c == ',' && parenDepth == 0) + else if (c == '[') bracketDepth++; + else if (c == ']' && bracketDepth > 0) bracketDepth--; + else if (c == ',' && parenDepth == 0 && bracketDepth == 0) { // ✅ FIX: Strict bounds check before Slice to prevent ArgumentOutOfRangeException if (valueStart < i && valueIndex < ColumnTypes.Count) @@ -1122,9 +1125,10 @@ private bool TryParseUpdateForBatch( if (!tables.TryGetValue(tableName, out var table)) return false; - // Parse SET clause: "col1 = val1, col2 = val2" + // Parse SET clause: "col1 = val1, col2 = val2". Split top-level so a vector literal + // (`SET embedding = [0.5, 0.25]`) stays one assignment. var fallbackUpdates = new Dictionary(); - var setParts = setClause.Split(','); + var setParts = SqlInPredicate.SplitTopLevelComma(setClause); foreach (var setPart in setParts) { var eqIdx = setPart.IndexOf('='); @@ -1872,13 +1876,13 @@ private static List> RowsToDictionaryList(SharpCoreDB var colStart = rest.IndexOf('(') + 1; var colEnd = rest.IndexOf(')', colStart); var colStr = rest[colStart..colEnd]; - insertColumns = [.. colStr.Split(',').Select(c => c.Trim())]; + insertColumns = [.. SqlInPredicate.SplitTopLevelComma(colStr).Select(c => c.Trim())]; rest = rest[(colEnd + 1)..]; } var valuesStart = rest.IndexOf("VALUES", StringComparison.OrdinalIgnoreCase) + "VALUES".Length; var valuesStr = rest[valuesStart..].Trim().TrimStart('(').TrimEnd(')'); - var values = valuesStr.Split(',').Select(v => v.Trim().Trim('\'')).ToList(); + var values = SqlInPredicate.SplitTopLevelComma(valuesStr).Select(v => v.Trim().Trim('\'')).ToList(); Dictionary row = []; var table = tables[tableName]; diff --git a/src/SharpCoreDB/Optimizations/BinaryRowDecoder.cs b/src/SharpCoreDB/Optimizations/BinaryRowDecoder.cs index b8161f33..c2db89b1 100644 --- a/src/SharpCoreDB/Optimizations/BinaryRowDecoder.cs +++ b/src/SharpCoreDB/Optimizations/BinaryRowDecoder.cs @@ -158,6 +158,9 @@ private static object DecodeValue( return data.Slice(4, length).ToArray(); } + case DataType.Vector: + return VectorBinaryCodec.ReadLengthPrefixed(data, out bytesRead); + case DataType.Guid: bytesRead = 16; return new Guid(data[..16]); diff --git a/src/SharpCoreDB/Optimizations/BulkInsertValuePipeline.cs b/src/SharpCoreDB/Optimizations/BulkInsertValuePipeline.cs index 505c877b..7ceb2f91 100644 --- a/src/SharpCoreDB/Optimizations/BulkInsertValuePipeline.cs +++ b/src/SharpCoreDB/Optimizations/BulkInsertValuePipeline.cs @@ -215,6 +215,21 @@ private static int EncodeTypedValue( } break; + case DataType.Vector: + { + // Same framing as Blob: [int32 length][VEC\0 blob], from the one vector codec. + var vector = VectorBinaryCodec.ToVector(value); + int vectorLength = VectorBinaryCodec.EncodedLength(vector); + + if (target.Length < 5 + vectorLength) + throw new InvalidOperationException("Buffer too small"); + + System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(target[1..], vectorLength); + _ = VectorBinaryCodec.WriteTo(target[5..], vector); + bytesWritten += 4 + vectorLength; + } + break; + case DataType.Guid: if (target.Length < 17) throw new InvalidOperationException("Buffer too small"); ((Guid)value).TryWriteBytes(target[1..]); @@ -290,6 +305,11 @@ private static int EstimateColumnBufferSize(DataType type, int rowCount) DataType.Ulid => 35, // 1 null + 4 len + 26 chars + 4 for safety DataType.String => 256, // Average estimate DataType.Blob => 256, // Average estimate + + // Same per-row budget the parallel-scan cell uses — the documented 1536-dimension embedding + // (1 null + 4 length + 12 header + 1536 float32). A wider vector is refused by EncodeValue's + // bounds check rather than truncated. + DataType.Vector => 5 + VectorBinaryCodec.EncodedLength(1536), _ => 64 }; diff --git a/src/SharpCoreDB/Optimizations/InsertBatchOptimized.cs b/src/SharpCoreDB/Optimizations/InsertBatchOptimized.cs index 58a36fcc..8f7a2a65 100644 --- a/src/SharpCoreDB/Optimizations/InsertBatchOptimized.cs +++ b/src/SharpCoreDB/Optimizations/InsertBatchOptimized.cs @@ -5,6 +5,7 @@ namespace SharpCoreDB.Optimizations; +using SharpCoreDB.DataStructures; using SharpCoreDB.Services; using System; using System.Buffers; @@ -184,6 +185,7 @@ private static int EstimateRowSize( DataType.Ulid => 4 + 26, // length prefix + ULID string DataType.String => 4 + System.Text.Encoding.UTF8.GetByteCount((string)value), DataType.Blob => 4 + ((byte[])value).Length, + DataType.Vector => 4 + VectorBinaryCodec.EncodedLength(VectorBinaryCodec.ToVector(value)), _ => 4 + 100 // default fallback }; } @@ -304,6 +306,13 @@ private static int SerializeRowToBuffer( case DataType.Blob: bytesWritten += SerializeBlob(buffer.Slice(bytesWritten), (byte[])value); break; + case DataType.Vector: + // Same framing as Blob, payload from the one vector codec (EstimateRowSize budgets the + // same 4 + EncodedLength bytes, so the rented buffer always fits). + bytesWritten += SerializeBlob( + buffer.Slice(bytesWritten), + VectorBinaryCodec.Serialize(VectorBinaryCodec.ToVector(value))); + break; default: bytesWritten += SerializeString(buffer.Slice(bytesWritten), value.ToString() ?? string.Empty); break; diff --git a/src/SharpCoreDB/Optimizations/StreamingRowEncoder.cs b/src/SharpCoreDB/Optimizations/StreamingRowEncoder.cs index f44892a0..29dde345 100644 --- a/src/SharpCoreDB/Optimizations/StreamingRowEncoder.cs +++ b/src/SharpCoreDB/Optimizations/StreamingRowEncoder.cs @@ -219,6 +219,22 @@ private static int EncodeValue( } break; + case DataType.Vector: + { + // Same framing as Blob: [int32 length][VEC\0 blob]. The blob comes from the one vector + // codec, so this writer cannot drift from Table's row codec. + var vector = VectorBinaryCodec.ToVector(value); + int vectorLength = VectorBinaryCodec.EncodedLength(vector); + + if (target.Length < 5 + vectorLength) + throw new InvalidOperationException("Buffer too small"); + + System.Buffers.Binary.BinaryPrimitives.WriteInt32LittleEndian(target[1..], vectorLength); + _ = VectorBinaryCodec.WriteTo(target[5..], vector); + bytesWritten += 4 + vectorLength; + } + break; + case DataType.Guid: if (target.Length < 17) throw new InvalidOperationException("Buffer too small"); ((Guid)value).TryWriteBytes(target[1..]); diff --git a/src/SharpCoreDB/Services/CompiledQueryExecutor.cs b/src/SharpCoreDB/Services/CompiledQueryExecutor.cs index b7d4f881..105be498 100644 --- a/src/SharpCoreDB/Services/CompiledQueryExecutor.cs +++ b/src/SharpCoreDB/Services/CompiledQueryExecutor.cs @@ -353,6 +353,12 @@ private static string FormatParameterValue(object? value) decimal m => m.ToString(System.Globalization.CultureInfo.InvariantCulture), bool b => b ? "1" : "0", DateTime dt => $"'{dt:yyyy-MM-dd HH:mm:ss}'", + + // Same vector-literal rendering as ParameterBinder.FormatParameter, so both binders agree. + float[] vector => VectorBinaryCodec.ToJson(vector), + ReadOnlyMemory vectorMemory => VectorBinaryCodec.ToJson(vectorMemory.Span), + Memory vectorBuffer => VectorBinaryCodec.ToJson(vectorBuffer.Span), + double[] vectorDoubles => VectorBinaryCodec.ToJson(Array.ConvertAll(vectorDoubles, static d => (float)d)), _ => $"'{value.ToString()?.Replace("'", "''") ?? string.Empty}'" }; } diff --git a/src/SharpCoreDB/Services/PreparedStatements.cs b/src/SharpCoreDB/Services/PreparedStatements.cs index 9c2ded46..8bd1e328 100644 --- a/src/SharpCoreDB/Services/PreparedStatements.cs +++ b/src/SharpCoreDB/Services/PreparedStatements.cs @@ -492,6 +492,14 @@ internal static string FormatParameter(object? value) float f => f.ToString(System.Globalization.CultureInfo.InvariantCulture), int i => i.ToString(), long l => l.ToString(), + + // Vector parameters are rendered as the bracketed literal the parser reads back into a float[] + // (VectorBinaryCodec), and the VALUES/SET splitters keep that literal in one piece. Without this a + // float[] parameter was rendered as the literal text "System.Single[]" and stored as the string. + float[] vector => VectorBinaryCodec.ToJson(vector), + ReadOnlyMemory vectorMemory => VectorBinaryCodec.ToJson(vectorMemory.Span), + Memory vectorBuffer => VectorBinaryCodec.ToJson(vectorBuffer.Span), + double[] vectorDoubles => VectorBinaryCodec.ToJson(Array.ConvertAll(vectorDoubles, static d => (float)d)), byte[] bytes => $"0x{System.Convert.ToHexString(bytes)}", _ => $"'{EscapeSqlString(value.ToString())}'", }; diff --git a/src/SharpCoreDB/Services/SqlInPredicate.cs b/src/SharpCoreDB/Services/SqlInPredicate.cs index 04e17264..da860532 100644 --- a/src/SharpCoreDB/Services/SqlInPredicate.cs +++ b/src/SharpCoreDB/Services/SqlInPredicate.cs @@ -396,14 +396,20 @@ private static List ParseValueRows(string listPart) } /// - /// Splits a string on commas that are not inside parentheses or string literals. This is what - /// lets IN (VALUES ('a', 'x'), ('b', 'y')) be split into two tuple rows instead of four - /// scalar values. + /// Splits on commas that are at the top level: a comma inside a string literal, + /// inside parentheses, or inside a bracketed literal does not split. The bracketed-literal rule is what + /// keeps a VECTOR value ([0.5, 0.25, 0.125], see docs/Vectors/TECHNICAL_SPEC.md) in + /// one piece, so the INSERT and UPDATE value splitters of every statement path agree about where a vector + /// literal ends. This is the single definition — the private the + /// IN-list parser uses delegates here. /// - private static List SplitTopLevel(string text) + /// The text to split. + /// The top-level comma-separated parts, untrimmed and in order. + public static List SplitTopLevelComma(string text) { var parts = new List(); int depth = 0; + int bracketDepth = 0; bool inString = false; char quote = '\0'; int start = 0; @@ -430,7 +436,15 @@ private static List SplitTopLevel(string text) { depth = Math.Max(0, depth - 1); } - else if (c == ',' && depth == 0) + else if (c == '[') + { + bracketDepth++; + } + else if (c == ']') + { + bracketDepth = Math.Max(0, bracketDepth - 1); + } + else if (c == ',' && depth == 0 && bracketDepth == 0) { parts.Add(text[start..i]); start = i + 1; @@ -441,6 +455,13 @@ private static List SplitTopLevel(string text) return parts; } + /// + /// Splits a string on commas that are not inside parentheses or string literals. This is what + /// lets IN (VALUES ('a', 'x'), ('b', 'y')) be split into two tuple rows instead of four + /// scalar values. + /// + private static List SplitTopLevel(string text) => SplitTopLevelComma(text); + /// /// Normalizes a column reference to the bare row-key form: strips alias qualifiers /// (e.g. b.Url to Url) and identifier quotes (", [, ], `). diff --git a/src/SharpCoreDB/Services/SqlParser.DML.cs b/src/SharpCoreDB/Services/SqlParser.DML.cs index b7642a5e..9a8af7a4 100644 --- a/src/SharpCoreDB/Services/SqlParser.DML.cs +++ b/src/SharpCoreDB/Services/SqlParser.DML.cs @@ -675,11 +675,15 @@ private static List ParseInsertValues(ReadOnlySpan valuesStr) // The four rules reproduced here are pinned by InsertValuesParsingTests: // 1. a quote toggles and is DROPPED, unless the previously *appended* character was a backslash — the // check is against the accumulated content, not the source text, so a dropped quote does not update it; - // 2. a comma splits only outside quotes; + // 2. a comma splits only outside quotes and outside a bracketed literal; // 3. every literal is Trimmed; // 4. a literal whose content is empty is not emitted at all, so `(1,)` yields one value while `(1, )` // yields an empty one. // + // Rule 2's bracketed-literal clause is what makes a VECTOR value expressible in VALUES at all: + // `[0.5, 0.25, 0.125, 0.0625]` is one literal, not four (docs/Vectors/TECHNICAL_SPEC.md §7). Without + // it the first fragment reached ParseVectorValue as "[0.5" and the statement failed as invalid base64. + // // A literal is slice-expressible exactly when at most one quote pair toggles inside it. An interior // toggling quote — `'it''s'` → `its` — is *dropped* by rule 1, which no contiguous slice can express, so // such a literal falls back to the character loop this method used to run for everything. That loop is @@ -693,6 +697,7 @@ private static List ParseInsertValues(ReadOnlySpan valuesStr) int contentStart = index; int contentEnd = -1; int toggles = 0; + int bracketDepth = 0; bool inQuotes = false; char lastAppended = '\0'; bool terminatedByComma = false; @@ -711,7 +716,19 @@ private static List ParseInsertValues(ReadOnlySpan valuesStr) continue; // a dropped quote must not become the last appended character } - if (c == ',' && !inQuotes) + if (!inQuotes) + { + if (c == '[') + { + bracketDepth++; + } + else if (c == ']' && bracketDepth > 0) + { + bracketDepth--; + } + } + + if (c == ',' && !inQuotes && bracketDepth == 0) { terminatedByComma = true; break; @@ -763,11 +780,13 @@ private static List ParseInsertValues(ReadOnlySpan valuesStr) // The reference implementation, used only for literals a slice cannot express: one with an interior // toggling quote, or an unterminated one. It is byte-for-byte the behaviour this method had before the - // fast path existed, including emitting nothing when the accumulated content is empty. + // fast path existed, including emitting nothing when the accumulated content is empty — plus the + // bracketed-literal clause of rule 2, so it cannot disagree with the fast path about a vector literal. static string? BuildLiteralByAppending(ReadOnlySpan text, bool terminatedByComma) { var currentValue = new StringBuilder(); bool inQuotes = false; + int bracketDepth = 0; foreach (char c in text) { @@ -777,7 +796,19 @@ private static List ParseInsertValues(ReadOnlySpan valuesStr) continue; } - if (c == ',' && !inQuotes) + if (!inQuotes) + { + if (c == '[') + { + bracketDepth++; + } + else if (c == ']' && bracketDepth > 0) + { + bracketDepth--; + } + } + + if (c == ',' && !inQuotes && bracketDepth == 0) { break; } @@ -1846,7 +1877,9 @@ private void ExecuteUpdate(string sql, IWAL? wal) if (!tables.TryGetValue(tableName, out var table)) throw new InvalidOperationException($"Table {tableName} does not exist"); - var setClauses = updateMatch.Groups[2].Value.Trim().Split(','); + // Bracket-aware split: `SET embedding = [0.5, 0.25]` is one assignment, not two + // (SqlInPredicate.SplitTopLevelComma honours quotes, parentheses and bracketed literals). + var setClauses = SqlInPredicate.SplitTopLevelComma(updateMatch.Groups[2].Value.Trim()); var whereClause = updateMatch.Groups[3].Value.Trim(); var updates = new Dictionary(); diff --git a/src/SharpCoreDB/Services/SqlParser.Helpers.cs b/src/SharpCoreDB/Services/SqlParser.Helpers.cs index ace21ada..c28ba43f 100644 --- a/src/SharpCoreDB/Services/SqlParser.Helpers.cs +++ b/src/SharpCoreDB/Services/SqlParser.Helpers.cs @@ -264,23 +264,21 @@ private static DateTime ParseDateTime(string val) } /// - /// Parses a vector value from a JSON float array string or Base64 bytes. - /// Supports: "[0.1, 0.2, 0.3]" JSON format and Base64-encoded binary. + /// Parses a vector value. Delegates to — the single + /// definition of the accepted textual forms (JSON float array, base64 blob, 0x hex blob) — so the SQL path + /// and the parameter binder cannot disagree about what a vector literal is. /// + /// The literal text. + /// The parsed float array. private static object ParseVectorValue(string val) { - val = val.Trim(); - - // JSON array format: [0.1, 0.2, 0.3, ...] - if (val.StartsWith('[') && val.EndsWith(']')) + if (DataStructures.VectorBinaryCodec.TryParse(val, out var vector)) { - return AotJsonSerializer.Deserialize(val) - ?? throw new InvalidOperationException("Failed to parse vector JSON array"); + return vector; } - // Base64-encoded binary format - var bytes = Convert.FromBase64String(val); - return System.Runtime.InteropServices.MemoryMarshal.Cast(bytes.AsSpan()).ToArray(); + throw new InvalidOperationException( + $"Invalid VECTOR literal '{val}'. Expected a JSON float array like [0.5, 0.25], a base64 blob or a 0x hex blob."); } /// diff --git a/src/SharpCoreDB/Services/SqlParserExtensions.cs b/src/SharpCoreDB/Services/SqlParserExtensions.cs index 2adfda7d..c23f89da 100644 --- a/src/SharpCoreDB/Services/SqlParserExtensions.cs +++ b/src/SharpCoreDB/Services/SqlParserExtensions.cs @@ -320,6 +320,7 @@ private static List ExtractValues(string sql) var valuesStr = remaining.Substring(startIdx + 1, endIdx - startIdx - 1); - return [.. valuesStr.Split(',').Select(v => v.Trim().Trim('\'') as object)]; + // Top-level split: a bracketed VECTOR literal ([0.5, 0.25]) is one value, not two. + return [.. Services.SqlInPredicate.SplitTopLevelComma(valuesStr).Select(v => v.Trim().Trim('\'') as object)]; } } diff --git a/src/SharpCoreDB/SingleFileTable.cs b/src/SharpCoreDB/SingleFileTable.cs index c4e7195f..f0078b2d 100644 --- a/src/SharpCoreDB/SingleFileTable.cs +++ b/src/SharpCoreDB/SingleFileTable.cs @@ -1963,7 +1963,7 @@ private static bool EvaluateSingleCondition(Dictionary row, stri return result; } - private static Dictionary FromSerializableRow(Dictionary row) + private Dictionary FromSerializableRow(Dictionary row) { var result = new Dictionary(row.Count); foreach (var (key, value) in row) @@ -1974,16 +1974,30 @@ private static Dictionary FromSerializableRow(Dictionary= 0 && ColumnTypes[vectorColumnIndex] == DataType.Vector + && VectorBinaryCodec.TryParse(element.GetRawText(), out var vector)) { - JsonValueKind.Number => ConvertJsonNumber(element), - JsonValueKind.String => element.GetString() ?? string.Empty, - JsonValueKind.True => true, - JsonValueKind.False => false, - JsonValueKind.Null => DBNull.Value, - _ => value - }; + result[key] = vector; + } + else + { + // Convert JsonElement to the appropriate CLR type + result[key] = element.ValueKind switch + { + JsonValueKind.Number => ConvertJsonNumber(element), + JsonValueKind.String => element.GetString() ?? string.Empty, + JsonValueKind.True => true, + JsonValueKind.False => false, + JsonValueKind.Null => DBNull.Value, + _ => value + }; + } } else { diff --git a/src/SharpCoreDB/Storage/ColumnarDataReader.cs b/src/SharpCoreDB/Storage/ColumnarDataReader.cs index 46a8e030..2f0dbffa 100644 --- a/src/SharpCoreDB/Storage/ColumnarDataReader.cs +++ b/src/SharpCoreDB/Storage/ColumnarDataReader.cs @@ -340,6 +340,17 @@ private static object ReadTypedValue(ReadOnlySpan span, DataType dataType, bytesRead = 4 + blobLength; return span.Slice(4, blobLength).ToArray(); + case DataType.Vector: + if (span.Length < 4) + return Array.Empty(); + + int vectorLength = BinaryPrimitives.ReadInt32LittleEndian(span); + if (vectorLength < VectorBinaryCodec.HeaderSize || vectorLength > span.Length - 4) + return Array.Empty(); + + bytesRead = 4 + vectorLength; + return VectorBinaryCodec.Deserialize(span.Slice(4, vectorLength)); + default: return string.Empty; } diff --git a/src/SharpCoreDB/Storage/PageBasedDataWriter.cs b/src/SharpCoreDB/Storage/PageBasedDataWriter.cs index 6e2835a4..146dacf2 100644 --- a/src/SharpCoreDB/Storage/PageBasedDataWriter.cs +++ b/src/SharpCoreDB/Storage/PageBasedDataWriter.cs @@ -294,6 +294,14 @@ private static void WriteTypedValue(BinaryWriter writer, object value, DataType writer.Write(blob); break; + case DataType.Vector: + // Same framing as Blob ([int32 length][payload]); the payload is the VEC\0 blob written by + // the one vector codec, and ColumnarDataReader reads it back as float[]. + var vectorBytes = VectorBinaryCodec.Serialize(VectorBinaryCodec.ToVector(value)); + writer.Write(vectorBytes.Length); + writer.Write(vectorBytes); + break; + default: throw new NotSupportedException($"Data type {dataType} not supported for serialization"); } diff --git a/src/SharpCoreDB/Storage/Scdb/SingleFileDatabase.Batch.cs b/src/SharpCoreDB/Storage/Scdb/SingleFileDatabase.Batch.cs index 5d9afff4..16c6671c 100644 --- a/src/SharpCoreDB/Storage/Scdb/SingleFileDatabase.Batch.cs +++ b/src/SharpCoreDB/Storage/Scdb/SingleFileDatabase.Batch.cs @@ -305,7 +305,7 @@ private static (string tableName, Dictionary row)? ParseInsertSt if (colEnd > colStart) { var colStr = afterInto[(colStart + 1)..colEnd]; - insertColumns = [.. colStr.Split(',').Select(c => c.Trim())]; + insertColumns = [.. Services.SqlInPredicate.SplitTopLevelComma(colStr).Select(c => c.Trim())]; rest = afterInto[(colEnd + 1)..]; } } @@ -374,6 +374,7 @@ private static List ParseValueList(string valuesStr) int start = 0; bool inQuotes = false; int inParens = 0; // ✅ FIX: Use int for parenthesis depth + int inBrackets = 0; // bracketed literals ([0.5, 0.25] — a VECTOR value) never split // ✅ OPTIMIZATION: Use span-based iteration to avoid repeated string allocations for (int i = 0; i < span.Length; i++) @@ -396,8 +397,16 @@ private static List ParseValueList(string valuesStr) { inParens--; } + else if (c == '[') + { + inBrackets++; + } + else if (c == ']' && inBrackets > 0) + { + inBrackets--; + } // Check for value delimiter - else if (c == ',' && inParens == 0) + else if (c == ',' && inParens == 0 && inBrackets == 0) { // Extract value using span (zero-copy until ToString) var valueSpan = span[start..i].Trim(); @@ -514,7 +523,7 @@ private static (string tableName, object pkValue, Dictionary upd // Parse SET assignments var updates = new Dictionary(); - foreach (var assignment in setClause.Split(',')) + foreach (var assignment in Services.SqlInPredicate.SplitTopLevelComma(setClause)) { var parts = assignment.Split('='); if (parts.Length != 2) continue; diff --git a/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs b/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs index 999b8116..3864f4e8 100644 --- a/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs +++ b/tests/SharpCoreDB.Tests/Documentation/DocumentationConsistencyTests.cs @@ -97,7 +97,9 @@ public void DocumentationMarkdownLinks_ShouldResolveToExistingFiles() var resolved = Path.GetFullPath(Path.Combine(Path.GetDirectoryName(file)!, path.Replace('/', Path.DirectorySeparatorChar))); - if (!File.Exists(resolved) && !Directory.Exists(resolved)) + // Case-exact, like the ubuntu runner: File.Exists/Directory.Exists are case-insensitive on + // Windows and macOS, so a wrong-case link used to pass here and fail only in CI (session 85). + if (!PathExistsWithExactCase(resolved)) { failures.Add($"{Path.GetRelativePath(docsRoot, file)}:{lineNumber} -> {target}"); } @@ -124,7 +126,8 @@ public void DocumentationIndex_ShouldOnlyReferenceExistingDocumentationFiles() var relative = match.Groups[1].Value; var resolved = Path.GetFullPath(Path.Combine(docsRoot, relative.Replace('/', Path.DirectorySeparatorChar))); - if (!File.Exists(resolved)) + // Case-exact, so a wrong-case path fails here and not only on the case-sensitive CI runner. + if (!PathExistsWithExactCase(resolved)) { failures.Add(relative); } @@ -225,6 +228,62 @@ public void DocumentationPackageReferences_ShouldResolveToRealPackages() Assert.Empty(failures); } + /// + /// Reports whether exists when every segment is matched case-sensitively, which is + /// how Linux — and therefore the CI runner — resolves it. and + /// are case-insensitive on Windows and macOS, so using them made a + /// wrong-case link pass locally and fail only on the ubuntu runner. + /// + /// An absolute path, expected to live inside this repository. + /// true if the path resolves with exactly the casing on disk. + private static bool PathExistsWithExactCase(string path) + { + var repositoryRoot = GetRepositoryFilePath(); + var current = repositoryRoot; + var relative = Path + .GetRelativePath(repositoryRoot, path) + .Replace(Path.AltDirectorySeparatorChar, Path.DirectorySeparatorChar); + + foreach (var segment in relative.Split(Path.DirectorySeparatorChar, StringSplitOptions.RemoveEmptyEntries)) + { + if (segment == ".") + { + continue; + } + + if (segment == "..") + { + var parent = Directory.GetParent(current); + + if (parent is null) + { + return false; + } + + current = parent.FullName; + continue; + } + + if (!Directory.Exists(current)) + { + return false; + } + + var match = Directory + .EnumerateFileSystemEntries(current) + .FirstOrDefault(entry => string.Equals(Path.GetFileName(entry), segment, StringComparison.Ordinal)); + + if (match is null) + { + return false; + } + + current = match; + } + + return true; + } + private static string GetRepositoryFilePath(params string[] segments) { var current = new DirectoryInfo(AppContext.BaseDirectory); diff --git a/tests/SharpCoreDB.Tests/VectorBinaryCodecTests.cs b/tests/SharpCoreDB.Tests/VectorBinaryCodecTests.cs new file mode 100644 index 00000000..c30b42d5 --- /dev/null +++ b/tests/SharpCoreDB.Tests/VectorBinaryCodecTests.cs @@ -0,0 +1,132 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// +namespace SharpCoreDB.Tests; + +using System; +using System.Buffers.Binary; +using System.Collections.Generic; +using System.Runtime.InteropServices; +using SharpCoreDB.DataStructures; +using Xunit; + +/// +/// Unit tests for — the one definition of the VEC\0 vector binary +/// format (docs/Vectors/TECHNICAL_SPEC.md §7) and of the coercion rules every row codec and the parameter +/// binder share. The storage-level round trips live in . +/// +public sealed class VectorBinaryCodecTests +{ + [Fact] + public void Serialize_WritesSpecHeader_AndRoundTrips() + { + float[] vector = [0.5f, -0.25f, 0f]; + + byte[] blob = VectorBinaryCodec.Serialize(vector); + + Assert.Equal("VEC\0"u8.ToArray(), blob[..4]); + Assert.Equal(VectorBinaryCodec.FormatVersion, blob[4]); + Assert.Equal(0, blob[5]); // flags + Assert.Equal(vector.Length, BinaryPrimitives.ReadInt32LittleEndian(blob.AsSpan(8, 4))); + Assert.Equal(VectorBinaryCodec.EncodedLength(vector), blob.Length); + Assert.Equal(vector, VectorBinaryCodec.Deserialize(blob)); + Assert.Equal(vector.Length, VectorBinaryCodec.GetDimensions(blob)); + } + + [Fact] + public void Deserialize_RawFloatBytes_WithoutHeader_StillReads() + { + // Externally produced embeddings have no header; that tolerance is kept. + float[] vector = [1.5f, -2.5f]; + byte[] raw = MemoryMarshal.AsBytes(vector.AsSpan()).ToArray(); + + Assert.False(VectorBinaryCodec.HasHeader(raw)); + Assert.Equal(vector, VectorBinaryCodec.Deserialize(raw)); + Assert.Equal(vector.Length, VectorBinaryCodec.GetDimensions(raw)); + } + + [Fact] + public void Deserialize_TruncatedBlob_Throws() + { + byte[] blob = VectorBinaryCodec.Serialize([1f, 2f, 3f]); + + Assert.Throws(() => VectorBinaryCodec.Deserialize(blob.AsSpan(0, blob.Length - 4))); + } + + [Fact] + public void ReadLengthPrefixed_TruncatedPayload_Throws() + { + float[] vector = [1f, 2f]; + var buffer = new byte[VectorBinaryCodec.EncodedLength(vector) + 4]; + int written = VectorBinaryCodec.WriteLengthPrefixed(buffer, vector); + Assert.Equal(buffer.Length, written); + + Assert.Equal(vector, VectorBinaryCodec.ReadLengthPrefixed(buffer, out int bytesRead)); + Assert.Equal(written, bytesRead); + + Assert.Throws(() => VectorBinaryCodec.ReadLengthPrefixed(buffer.AsSpan(0, 6), out _)); + } + + [Fact] + public void TryParse_AcceptsJsonBase64AndHex() + { + float[] vector = [0.5f, 0.25f]; + byte[] raw = MemoryMarshal.AsBytes(vector.AsSpan()).ToArray(); + + Assert.True(VectorBinaryCodec.TryParse("[0.5, 0.25]", out var fromJson)); + Assert.Equal(vector, fromJson); + + Assert.True(VectorBinaryCodec.TryParse($"'{Convert.ToBase64String(raw)}'", out var fromBase64)); + Assert.Equal(vector, fromBase64); + + Assert.True(VectorBinaryCodec.TryParse($"0x{Convert.ToHexString(raw)}", out var fromHex)); + Assert.Equal(vector, fromHex); + + Assert.False(VectorBinaryCodec.TryParse("not-a-vector", out _)); + Assert.False(VectorBinaryCodec.TryParse(string.Empty, out _)); + } + + [Fact] + public void TryToVector_CoercesTheRepresentationsTheBinderAndCodecsAccept() + { + float[] expected = [0.5f, -0.25f]; + + Assert.True(VectorBinaryCodec.TryToVector(new float[] { 0.5f, -0.25f }, out var fromArray)); + Assert.Equal(expected, fromArray); + + Assert.True(VectorBinaryCodec.TryToVector(new ReadOnlyMemory(expected), out var fromMemory)); + Assert.Equal(expected, fromMemory); + + Assert.True(VectorBinaryCodec.TryToVector(new List { 0.5f, -0.25f }, out var fromList)); + Assert.Equal(expected, fromList); + + Assert.True(VectorBinaryCodec.TryToVector(new double[] { 0.5, -0.25 }, out var fromDoubles)); + Assert.Equal(expected, fromDoubles); + + Assert.True(VectorBinaryCodec.TryToVector(VectorBinaryCodec.Serialize(expected), out var fromBlob)); + Assert.Equal(expected, fromBlob); + + Assert.False(VectorBinaryCodec.TryToVector(Guid.NewGuid(), out _)); + Assert.False(VectorBinaryCodec.TryToVector(null, out _)); + } + + [Fact] + public void ToVector_UnsupportedType_ThrowsWithAcceptedForms() + { + var error = Assert.Throws(() => VectorBinaryCodec.ToVector(Guid.NewGuid())); + Assert.Contains("VECTOR", error.Message, StringComparison.OrdinalIgnoreCase); + Assert.Contains("float[]", error.Message, StringComparison.Ordinal); + } + + [Fact] + public void ToJson_ThenTryParse_RoundTrips() + { + float[] vector = [0.5f, 0.25f, 0.125f]; + + string json = VectorBinaryCodec.ToJson(vector); + + Assert.True(VectorBinaryCodec.TryParse(json, out var parsed)); + Assert.Equal(vector, parsed); + } +} diff --git a/tests/SharpCoreDB.Tests/VectorColumnRegressionTests.cs b/tests/SharpCoreDB.Tests/VectorColumnRegressionTests.cs new file mode 100644 index 00000000..1a7b1aab --- /dev/null +++ b/tests/SharpCoreDB.Tests/VectorColumnRegressionTests.cs @@ -0,0 +1,251 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// +namespace SharpCoreDB.Tests; + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime.InteropServices; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.DataStructures; +using SharpCoreDB.Interfaces; +using SharpCoreDB.Optimizations; +using Xunit; + +/// +/// Regression tests for the VECTOR(N) defect family found while fixing +/// : the vector case was missing from eight per-type switches and from +/// the value tokenizers, so a vector was stored as the literal text "System.Single[]" (or came back as +/// a JsonElement from the single-file store). Each test here pins one of the copies that was fixed — +/// the batched writer, the streaming encoder/decoder pair, the multi-row VALUES tokenizer, the parameter +/// binder, the UPDATE SET splitter — plus the loud failures that replaced the silent ones. +/// +public sealed class VectorColumnRegressionTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _dirPath; + + public VectorColumnRegressionTests() + { + var provider = new ServiceCollection().AddSharpCoreDB().BuildServiceProvider(); + _factory = provider.GetRequiredService(); + _dirPath = Path.Combine(Path.GetTempPath(), $"SCDB_VectorReg_{Guid.NewGuid():N}"); + } + + public void Dispose() + { + try { if (Directory.Exists(_dirPath)) Directory.Delete(_dirPath, true); } catch { } + } + + [Fact] + public void InsertBatch_TableApi_WithFloatArrays_RoundTrips() + { + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(3))"); + db.TryGetTable("docs", out var table); + + table.InsertBatch( + [ + new Dictionary { ["id"] = 1, ["embedding"] = new float[] { 0.5f, 0.25f, 0.125f } }, + new Dictionary { ["id"] = 2, ["embedding"] = new float[] { -1f, 2f, 0.75f } }, + ]); + + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], 0.5f, 0.25f, 0.125f); + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 2")["embedding"], -1f, 2f, 0.75f); + }); + } + + [Fact] + public void InsertBatchFromBuffer_StreamingEncoder_DecodesVectorBack() + { + // StreamingRowEncoder and BinaryRowDecoder are one write/read pair; the vector case was missing from + // both, so this round trip decoded the blob into the wrong shape (or threw). + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(3))"); + db.TryGetTable("docs", out var table); + + using var encoder = new StreamingRowEncoder(table.Columns, table.ColumnTypes); + Assert.True(encoder.EncodeRow( + new Dictionary { ["id"] = 7, ["embedding"] = new float[] { 0.1f, 0.2f, 0.3f } })); + + table.InsertBatchFromBuffer(encoder.GetBatchData(), encoder.GetRowCount()); + + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 7")["embedding"], 0.1f, 0.2f, 0.3f); + }); + } + + [Fact] + public void MultiRowInsert_TwoVectorLiterals_ThenSelect_ReturnsBoth() + { + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(2))"); + db.ExecuteSQL("INSERT INTO docs (id, embedding) VALUES (1, [0.5, 0.25]), (2, [-0.5, -0.25])"); + + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], 0.5f, 0.25f); + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 2")["embedding"], -0.5f, -0.25f); + }); + } + + [Fact] + public void Insert_QuotedJsonVectorLiteral_RoundTrips() + { + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(2))"); + db.ExecuteSQL("INSERT INTO docs (id, embedding) VALUES (1, '[0.5, 0.25]')"); + + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], 0.5f, 0.25f); + }); + } + + [Fact] + public void Insert_Base64VectorLiteral_RoundTrips() + { + float[] expected = [0.5f, -0.25f]; + string base64 = Convert.ToBase64String(MemoryMarshal.AsBytes(expected.AsSpan())); + + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(2))"); + db.ExecuteSQL($"INSERT INTO docs (id, embedding) VALUES (1, '{base64}')"); + + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], expected); + }); + } + + [Fact] + public void Update_TwoAssignmentsIncludingVector_RoundTrips() + { + // The SET splitter used to cut on every comma, so `SET tag = 'x', embedding = [0.1, 0.2]` became two + // assignments with half a vector each. + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, tag TEXT, embedding VECTOR(2))"); + db.ExecuteSQL("INSERT INTO docs (id, tag, embedding) VALUES (1, 'a', [1.0, 0.0])"); + db.ExecuteSQL("UPDATE docs SET tag = 'b', embedding = [0.1, 0.2] WHERE id = 1"); + + var row = Row(db, "SELECT tag, embedding FROM docs WHERE id = 1"); + Assert.Equal("b", row["tag"]); + AssertVectorEquals(row["embedding"], 0.1f, 0.2f); + }); + } + + [Fact] + public void Insert_ReadOnlyMemoryParameter_RoundTrips() + { + // ReadOnlyMemory is what Microsoft.Extensions.AI embeddings and EF Core hand over. + ReadOnlyMemory memory = new float[] { 0.5f, 0.25f, 0.125f }; + + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(3))"); + db.ExecuteSQL( + "INSERT INTO docs (id, embedding) VALUES (@id, @embedding)", + new Dictionary { ["@id"] = 1, ["@embedding"] = memory }); + + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], 0.5f, 0.25f, 0.125f); + }); + } + + [Fact] + public void Insert_DoubleArrayParameter_RoundTrips() + { + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(2))"); + db.ExecuteSQL( + "INSERT INTO docs (id, embedding) VALUES (@id, @embedding)", + new Dictionary { ["@id"] = 1, ["@embedding"] = new double[] { 0.5, -0.25 } }); + + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], 0.5f, -0.25f); + }); + } + + [Fact] + public void Insert_InvalidVectorLiteral_Throws() + { + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(2))"); + + var error = Assert.Throws( + () => db.ExecuteSQL("INSERT INTO docs (id, embedding) VALUES (1, 'not-a-vector')")); + Assert.Contains("VECTOR", error.Message, StringComparison.OrdinalIgnoreCase); + }); + } + + [Fact] + public void Insert_NonVectorValue_ThrowsInsteadOfStoringToString() + { + // The defect this whole family came from: an unconvertible value fell through to value.ToString() and + // was stored as text. It must be a type error instead, for a parameter and for a column-shaped value. + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(2))"); + + var error = Assert.Throws( + () => db.ExecuteSQL( + "INSERT INTO docs (id, embedding) VALUES (@id, @embedding)", + new Dictionary { ["@id"] = 1, ["@embedding"] = Guid.NewGuid() })); + Assert.Contains("VECTOR", error.Message, StringComparison.OrdinalIgnoreCase); + }); + } + + [Fact] + public void ObjectApi_JsonStringValue_ThenSelect_ReturnsFloatArray() + { + // The object API accepts the same textual forms the SQL literal does, and the fixed-width codec must + // write the VEC\0 blob for a VECTOR column even when the caller handed over a string (the codec's + // string fast path used to store raw UTF-8 in the slot, which the reader then decoded as float bytes). + WithDb(db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(2))"); + db.TryGetTable("docs", out var table); + + table.Insert(new Dictionary { ["id"] = 1, ["embedding"] = "[0.5, 0.25]" }); + + AssertVectorEquals(Row(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], 0.5f, 0.25f); + }); + } + + private static Dictionary Row(IDatabase db, string sql) + { + var rows = db.ExecuteQuery(sql, new Dictionary()); + Assert.Single(rows); + return rows[0]; + } + + private static void AssertVectorEquals(object? value, params float[] expected) + { + if (value is not float[] actual) + { + Assert.Fail( + $"VECTOR column returned {value?.GetType().Name ?? "null"} ({value ?? "null"}) " + + $"instead of float[{expected.Length}]"); + return; + } + + Assert.Equal(expected, actual); + } + + private void WithDb(Action action) + { + var db = _factory.Create(_dirPath, "pw", isReadOnly: false, config: new DatabaseConfig()); + try + { + action(db); + db.Flush(); + db.ForceSave(); + } + finally + { + (db as IDisposable)?.Dispose(); + } + } +} diff --git a/tests/SharpCoreDB.Tests/VectorColumnStorageTests.cs b/tests/SharpCoreDB.Tests/VectorColumnStorageTests.cs new file mode 100644 index 00000000..c9cbafdd --- /dev/null +++ b/tests/SharpCoreDB.Tests/VectorColumnStorageTests.cs @@ -0,0 +1,225 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// +namespace SharpCoreDB.Tests; + +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// Round-trip tests for VECTOR(N) columns (see docs/Vectors/TECHNICAL_SPEC.md §7, +/// "Vectors are stored as typed BLOBs ... the DataType.Vector enum value tells the +/// serializer to use the optimized vector binary format"). +/// The vector cases were missing from every row codec, so a VECTOR value fell through the +/// serializer's default branch and was persisted as value.ToString() — a +/// float[] became the literal text "System.Single[]" and the data was silently +/// lost, in both the batched and the row-at-a-time write path. +/// +public sealed class VectorColumnStorageTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _dirPath; + private readonly string _filePath; + + public VectorColumnStorageTests() + { + var provider = new ServiceCollection().AddSharpCoreDB().BuildServiceProvider(); + _factory = provider.GetRequiredService(); + _dirPath = Path.Combine(Path.GetTempPath(), $"SCDB_VectorDir_{Guid.NewGuid():N}"); + _filePath = Path.Combine(Path.GetTempPath(), $"SCDB_VectorFile_{Guid.NewGuid():N}.scdb"); + } + + public void Dispose() + { + try { if (Directory.Exists(_dirPath)) Directory.Delete(_dirPath, true); } catch { } + try { if (File.Exists(_filePath)) File.Delete(_filePath); } catch { } + } + + /// Storage variants that must all keep vector values intact. + public static TheoryData Variants => new() + { + { "directory-fixedwidth-default", new DatabaseConfig() }, + { "directory-legacy-variable", new DatabaseConfig { AutoFixedWidthRecords = false } }, + { "singlefile-default", new DatabaseConfig() }, + }; + + [Fact] + public void Insert_VectorLiteral_ThenSelect_ReturnsFloatArray() + { + WithDb(_dirPath, new DatabaseConfig(), db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(4))"); + db.ExecuteSQL("INSERT INTO docs (id, embedding) VALUES (1, [0.5, 0.25, 0.125, 0.0625])"); + + AssertVectorEquals(SingleRow(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], + 0.5f, 0.25f, 0.125f, 0.0625f); + }); + } + + [Fact] + public void Insert_VectorLiteral_StoresVectorBinaryBlob() + { + // Spec §7: the column payload is the vector binary format — magic 'VEC\0', version 1, + // flags, then the 4-byte dimension count. Written unencrypted so the bytes are visible. + string path = Path.Combine(Path.GetTempPath(), $"SCDB_VectorBlob_{Guid.NewGuid():N}"); + var config = new DatabaseConfig { NoEncryptMode = true }; + try + { + WithDb(path, config, db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(4))"); + db.ExecuteSQL("INSERT INTO docs (id, embedding) VALUES (1, [0.5, 0.25, 0.125, 0.0625])"); + }); + + var payloads = Directory.EnumerateFiles(path, "*", SearchOption.AllDirectories) + .Select(File.ReadAllBytes) + .ToList(); + + Assert.True(payloads.Any(bytes => ContainsVectorHeader(bytes, dimensions: 4)), + "no file holds the 'VEC\\0' version-1 vector header with a 4-dimension count"); + } + finally + { + try { if (Directory.Exists(path)) Directory.Delete(path, true); } catch { } + } + } + + [Fact] + public void Insert_JsonStringParameter_ThenSelect_ReturnsFloatArray() + { + WithDb(_dirPath, new DatabaseConfig(), db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(3))"); + db.ExecuteSQL( + "INSERT INTO docs (id, embedding) VALUES (@id, @embedding)", + new Dictionary { ["@id"] = 1, ["@embedding"] = "[1.5, 2.5, 3.5]" }); + + AssertVectorEquals(SingleRow(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], + 1.5f, 2.5f, 3.5f); + }); + } + + [Fact] + public void Insert_FloatArrayParameter_ThenSelect_ReturnsFloatArray() + { + // The EF Core provider and the object API both hand the engine a float[]; the binder must + // render it as a vector literal instead of 'System.Single[]'. + WithDb(_dirPath, new DatabaseConfig(), db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(3))"); + db.ExecuteSQL( + "INSERT INTO docs (id, embedding) VALUES (@id, @embedding)", + new Dictionary { ["@id"] = 1, ["@embedding"] = new float[] { 0.5f, -0.25f, 0f } }); + + AssertVectorEquals(SingleRow(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], + 0.5f, -0.25f, 0f); + }); + } + + [Fact] + public void Update_VectorValue_ThenSelect_ReturnsUpdatedFloatArray() + { + WithDb(_dirPath, new DatabaseConfig(), db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(2))"); + db.ExecuteSQL("INSERT INTO docs (id, embedding) VALUES (1, [1.0, 0.0])"); + db.ExecuteSQL("UPDATE docs SET embedding = [0.0, 1.0] WHERE id = 1"); + + AssertVectorEquals(SingleRow(db, "SELECT embedding FROM docs WHERE id = 1")["embedding"], + 0f, 1f); + }); + } + + [Fact] + public void Insert_NullVector_ThenSelect_ReturnsNull() + { + WithDb(_dirPath, new DatabaseConfig(), db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(2))"); + db.ExecuteSQL("INSERT INTO docs (id, embedding) VALUES (1, NULL)"); + + var row = SingleRow(db, "SELECT embedding FROM docs WHERE id = 1"); + Assert.True(row["embedding"] is null or DBNull, $"expected NULL, got '{row["embedding"]}'"); + }); + } + + [Theory] + [MemberData(nameof(Variants))] + public void RoundTrip_Reopen_KeepsVectorValues(string variant, DatabaseConfig config) + { + string path = variant.StartsWith("singlefile", StringComparison.Ordinal) ? _filePath : _dirPath; + if (Directory.Exists(path)) Directory.Delete(path, true); + if (File.Exists(path)) File.Delete(path); + + WithDb(path, config, db => + { + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, embedding VECTOR(4))"); + db.ExecuteSQL("INSERT INTO docs (id, embedding) VALUES (1, [0.5, 0.25, 0.125, 0.0625])"); + db.ExecuteSQL("INSERT INTO docs (id, embedding) VALUES (2, [-1.5, 2.0, 0.0, 0.75])"); + }); + + WithDb(path, config, db => + { + var rows = db.ExecuteQuery("SELECT id, embedding FROM docs ORDER BY id", new Dictionary()); + Assert.Equal(2, rows.Count); + AssertVectorEquals(rows[0]["embedding"], 0.5f, 0.25f, 0.125f, 0.0625f); + AssertVectorEquals(rows[1]["embedding"], -1.5f, 2.0f, 0.0f, 0.75f); + }); + } + + /// Asserts the column returned expected as a array. + private static void AssertVectorEquals(object? value, params float[] expected) + { + if (value is not float[] actual) + { + Assert.Fail( + $"VECTOR column returned {value?.GetType().Name ?? "null"} ({value ?? "null"}) " + + $"instead of float[{expected.Length}]"); + return; + } + + Assert.Equal(expected, actual); + } + + private static bool ContainsVectorHeader(byte[] data, int dimensions) + { + for (int i = 0; i + 12 <= data.Length; i++) + { + if (data[i] != (byte)'V' || data[i + 1] != (byte)'E' || data[i + 2] != (byte)'C' || data[i + 3] != 0) + continue; + + if (data[i + 4] == 1 && data[i + 5] == 0 && BitConverter.ToInt32(data, i + 8) == dimensions) + return true; + } + + return false; + } + + private static Dictionary SingleRow(IDatabase db, string sql) + { + var rows = db.ExecuteQuery(sql, new Dictionary()); + Assert.Single(rows); + return rows[0]; + } + + private void WithDb(string path, DatabaseConfig config, Action action) + { + var db = _factory.Create(path, "pw", isReadOnly: false, config: config); + try + { + action(db); + db.Flush(); + db.ForceSave(); + } + finally + { + (db as IDisposable)?.Dispose(); + } + } +}