[vector store contract 4/6] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization by edwinyyyu · Pull Request #1618 · MemMachine/MemMachine

edwinyyyu · 2026-09-14T19:38:13Z

Purpose of the change

QdrantConf gains hnsw_config, optimizers_config and quantization_config, plain mappings mirroring qdrant-client's HnswConfigDiff, OptimizersConfigDiff and QuantizationConfig, so qdrant-client stays optional for configuration parsing; the store's params validate them against qdrant's own models. They apply to the store's one collection (its registry is SQL tables since #1631). The sample configurations and the configuration docs' parameter table carry the three keys.

m must be 0 or unset: the collection is multi-tenant and disables the global graph in favor of per-partition payload indexing, so a deployment tunes payload_m, which defaults to 16 as before.

Two commits: the options, and the docs that describe them, the parameter table rows and the sample configurations' comments. Adaptation for main: the options land on the one-collection store of [vector store scale-out 5/5], whose tests build their store through the module's _params helper, and the collection they apply to is the store's one native collection rather than a registry-minted one.

Stack

23 PRs on main: 5 independent ones, and three consecutive stages numbered on their own: the vector store scale-out, the SQLite store fixes and the vector store contract changes. A stacked PR's diff on GitHub is cumulative until the PRs under it merge.

Independent PRs, each directly on main; review and merge in any order:

#	PR	change
—	#1541	fix(event-backend): make expand_context return timeline-neighbor episodes
—	#1661	Overhaul segment store: shared tables with incarnation-scoped tenant keys (port of #1548)
—	#1630	Bound every request to a remote vector store by a configured timeout
—	#1682	Hand the Qdrant vector store a metrics factory, so its tracker emits (port of #1532)
—	#1624	Make no memory request create a project

The vector store stages, consecutive: each stacked on the one below. [vector store scale-out 1/5] is directly on main and 2/5 on it alone; they do not depend on the independent PRs. From 3/5 up, the chain's history also carries the independent PRs' commits beneath it, since the later stages were written on top of them ([vector store contract 2/6] on #1624's session policy, for one); so a merge of an independent PR rebases the chain without conflict, and until they merge those PRs' changes show in a stacked PR's diff.

#	PR	change
		Stage 1, the vector store scale-out: what horizontal scalability without sharding requires.
[vector store scale-out 1/5]	#1671	Remove custom sharding from the Qdrant store (port of #1654)
[vector store scale-out 2/5]	#1631	Mint an incarnation per collection life in a SQL-arbitrated registry, so any process may serve any Qdrant or Milvus collection
[vector store scale-out 3/5]	#1670	Remove per-project filterable properties (port of #1606)
[vector store scale-out 4/5]	#1702	Keep user properties out of the vector store
[vector store scale-out 5/5]	#1627	Make a vector store one collection, with string-keyed partitions
		Stage 2, the SQLite store fixes, on [vector store scale-out 5/5]: re-derived on the one-collection store.
[sqlite store fixes 1/7]	#1460	Publish vector index files atomically (but not durably) (as merged into `speedkick`, #1588)
[sqlite store fixes 2/7]	#1469	Never reuse a row id in SQLiteVectorStore (as merged into `speedkick`, #1589)
[sqlite store fixes 3/7]	#1672	Own the search engine's concurrency in the store, not in each engine (port of #1612)
[sqlite store fixes 4/7]	#1673	Serialize a partition's writes so the engine sees them in order (port of #1607)
[sqlite store fixes 5/7]	#1674	Refuse a pending row replay cannot honor, instead of dropping it (port of #1608)
[sqlite store fixes 6/7]	#1675	Take SQLite's write lock at BEGIN, not at the first write (port of #1609)
[sqlite store fixes 7/7]	#1676	Give every write a fresh row id, so a key names one version (port of #1610)
		Stage 3, the vector store contract changes, on [sqlite store fixes 7/7].
[vector store contract 1/6]	#1663	Answer with cosine scores and uuids, not vectors and stale properties, and require a vector on the record (port of #1598 and #1603)
[vector store contract 2/6]	#1622	Create a session's storage with the session, never on a request
[vector store contract 3/6]	#1625	Remove open-or-create and close from both stores
[vector store contract 4/6]	#1618 (this PR)	Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization
[vector store contract 5/6]	#1628	Make a vector store filter only on the properties it declares
[vector store contract 6/6]	#1616	Close the filter union, and make negation the complement on every backend

This PR's own change is its 2 commits, 39387d635, e298be279; the rest of its diff is the PRs under it. Stacked on #1625; #1628 is stacked on it.

Verification

At every commit from [sqlite store fixes 1/7] (d7a6042b2) to the tip 545ec401e, on 2026-09-21: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); the full server suite without integration tests passes (pytest packages/server/server_tests -m "not integration"); 1967 tests at this PR's head e298be279. Those commits were verified before the re-order that moved #1460 into the stage, which left their trees unchanged (git diff against the verified commits is empty). The 22 commits below d7a6042b2 (the scale-out stage and the independent PRs' copies in its history), whose trees the re-order changed only by the absence of #1460's files, are being re-verified as this is written; this paragraph is replaced with the result.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…e the stores are configured MilvusVectorStoreCollection.get() was the one client request without `timeout=`; the spy test now exercises it, so a request without the timeout fails the test. The configuration parameter table gains `request_timeout_seconds`, the databases page's Milvus example carries it, and the configuration page gains a Qdrant example beside the Milvus one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

QdrantVectorStore was the fourth component built without a metrics factory. OperationTracker accepts metrics_factory=None and then discards every timing without an error, so the store looked instrumented and emitted nothing - the same defect as the Neo4j store, the episode store and the session store, which is why no Qdrant latency was observable. QdrantConf gains MetricsFactoryIdMixin so it can resolve one, and database_manager passes it through. test_qdrant_creates_vector_store pinned the exact params and had to change. It now asserts metrics_factory is not None rather than pinning it: passing the keyword is not the property worth guarding, since None is accepted and silently discards everything. Removing the wiring fails it. Ported to main without MemMachine#1532's Dockerfile change (the EXTRAS build arg), which is unrelated to the wiring; the `metrics_factory_id` key is added to the database configuration table in the docs. (cherry picked from commit b6c90ab) Claude-Session: https://claude.ai/code/session_01Nr9kacmpFVTTfkZRw6esxP Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

The simple chatbot example, the TypeScript REST demo and the Dify plugin's add-memory tool wrote to a project without creating it, relying on the write to create it. Each now creates its project before its first memory request and accepts 409 as the project already existing. No behavior changes for them; they stop depending on a write creating a project. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn (cherry picked from commit 4cae58a)

Adding memories to, or searching, a project that did not exist created it, with the server's default configuration, without the caller's knowledge. Now only the create-project request creates a project: a write or a search opens the session and answers 404 for an unknown project, as the search endpoint already promised; the manager's open-or-create goes. Two callers depended on the implicit creation. `org_id` and `project_id` default to `universal`, so the API promises the project `universal/universal`; the server creates it, once, at startup, and leaves one that already exists as it is. The MCP add tool names its own project and has no create-project counterpart, so it creates the project it writes to, once, and says so. The API doc strings and the OpenAPI document say which requests create a project. A breaking API change on `speedkick`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn (cherry picked from commit d21a5d0)

…anything, on every entry point A write persisted its episodes before opening episodic memory, and a write or search that targeted only semantic memory never opened it, so a request naming a project nobody created could still insert episode rows under that key and answer 200. Every write now checks the registry first, and a search that does not open episodic memory checks it too; the check is one registry read, and the episodic open, which refuses an unknown project itself, is unchanged. `memmachine-server --stdio` built its resources without setting or starting the module-level MemMachine the tools read, and so never created the default project either; it now starts and stops through the same calls as the HTTP servers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

… points Qdrant and Milvus have no transactions, unique constraints or conditional writes, so the catalog each store keeps inside its backend (the per-namespace `__registry` collections) cannot arbitrate two server processes creating, deleting or reclaiming the same logical collection: create is a read then a write under a per-process lock, and two processes both succeed. That is what the VectorStore contract's "at most one process per collection" sentence has been standing in for. SqlCollectionRegistry keeps the catalog in the deployment's relational database instead: one table pair per backend kind, shared by every store on that kind of backend and scoped by the configured backend's id, with a row per live collection keyed by (backend, namespace, name) that carries the incarnation minted for that life of the name and the configuration the collection was created with, and a purge queue of dead incarnations. Creation is an insert the primary key arbitrates (a minted incarnation colliding with a live or queued one is re-minted, up to _MAX_MINT_ATTEMPTS, then VectorStoreAttemptsExhaustedError); deletion is one transaction that moves the row to the queue; a purge claim is a row lock under FOR UPDATE SKIP LOCKED on PostgreSQL, so concurrent purgers split a backlog. A queue entry is the incarnation's tombstone: the backend holds the points, and a write the registry read as live can land there after the purge that followed the deletion, so one purge cannot be the last. A round that finds points keeps the entry due; a round that finds none stamps it clean; the entry is removed only by a round that finds nothing again once the tombstone retention (on the database clock) has passed since the stamp, and until then the incarnation is never re-minted. The claim carries the namespace, name and configuration so a purger can find the native collection the points are in. VectorStoreCollectionHandleStaleError is the error a handle bound to a dead incarnation raises; the stores adopt it in the commits that put them on the registry. Tests run the registry on SQLite and PostgreSQL: creation, concurrent creators, backend scoping, deletion, the purge rounds, the retention, a raising round, and the re-mint paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…y; delete logically, reclaim by purge QdrantVectorStore keeps its catalog in SqlCollectionRegistry instead of the per-namespace `__registry` collections, and the per-process creation locks go with them. A logical collection is identified to callers by its (namespace, name) and inside the store by the incarnation the registry mints when it is created: every point carries the incarnation in the payload field that carried the name (the `is_tenant` index stays), and a handle is bound to one incarnation. A collection deleted and re-created under the same name starts empty, its predecessor's points are never seen by it and never reclaimed out from under it, and the old handle raises VectorStoreCollectionHandleStaleError on every operation, `get` included. Creation creates the native collection first (idempotent, as MemMachine#1681 left it) and registers last, so a crash between the two leaves an empty native collection the next creation of the same configuration adopts, never a row whose points have nowhere to go; the registry's primary key arbitrates, so a racing creator on any process gets AlreadyExists and open-or-create adopts the winner's row. Deletion is one registry transaction: the collection is unreachable when it commits and its points wait on the queue. The new `purge_deleted_collections` does one round on the oldest tombstone due: it looks for one point under the incarnation in the native collection the tombstone names and, if there is one, deletes by filter in a single server-side operation, `wait=True`; the registry keeps or removes the tombstone by what the round found. A handle checks liveness before an operation, to refuse a handle known to be dead, and after it, so an operation completed under an incarnation that died meanwhile raises instead of reporting success. No lock spans the remote call: a write that lands under a dead incarnation is the purge's to reclaim, which is what the tombstone's retention is for. The VectorStore contract states this: the ABC's "at most one process per collection" sentence becomes each store's own statement (QdrantVectorStore serves any process sharing the backend and the registry database; the SQLite stores keep their bound in their class docstrings), a handle's staleness is part of the collection contract, and `purge_deleted_collections` is part of the store contract, returning False on the stores whose deletion reclaims physically (both SQLite stores, and Milvus until its own commit). QdrantConf gains `registry_database`, the name of a relational database under `resources.databases`, required, and `tombstone_retention_seconds` (default 86400); `registry_replication_factor` goes with the registry collections it sized. DatabaseManager hands the store the engine and the backend's id, which scopes its rows in the tables every Qdrant store on one relational database shares. The wizard and the sample configurations' Qdrant blocks point the registry at their SQLite database, and the docs' parameter table and databases page describe the two keys and which stores serve any process. Tests: the collection lifecycle contract (stale handles, empty re-creation, open-or-create adopting the live incarnation, idempotent deletion, purge reclaiming what deletion deferred and sparing live collections, a write landing under a dead incarnation) runs against the local-mode client and, as integration tests, the REST and gRPC clients; the cross-worker tests use two clients over one registry and pin that a strict create is created once. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…y; delete logically, reclaim by purge The same change as the Qdrant commit, for MilvusVectorStore: the catalog moves from the per-namespace `memmachine_<ns>__registry` collections to SqlCollectionRegistry, the per-process creation locks go, and every entity carries the incarnation of its collection's life in the native partition-key field that carried the name (a uuid hex fits the field's 32 characters) and in its primary id, so a handle bound to a dead incarnation cannot reach the entities of the next life and raises VectorStoreCollectionHandleStaleError on every operation, before and after the remote call. Creation is native collection first, registry row last; deletion is one registry transaction; `purge_deleted_collections` looks for one entity under the claimed incarnation and deletes by filter when it finds one, reporting False when the native collection itself is gone. This also closes the second half of the create race the registry-in-Milvus had: `insert` into a Milvus collection does not enforce primary-key uniqueness, so two creators left two live entries under one name; the SQL registry's primary key leaves one. MilvusConf gains `registry_database` (required) and `tombstone_retention_seconds` (default 86400), DatabaseManager hands the store the engine and the backend's id, and the wizard, the sample configurations' Milvus blocks and the Milvus examples in the docs point the registry at the relational database beside them. The lifecycle contract runs against Milvus Lite; the test that pinned the registry lookup's output fields goes with the lookup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn (cherry picked from commit 09329d2)

…n the collection a racing creator won The registry-backed stores never schedule their own reclamation; the resource manager starts one sweeper per vector store the first time it hands the store out, driving `purge_deleted_collections` after a short pause while rounds find something and after the idle interval otherwise, logging and retrying a round that raises. close() cancels the sweepers before the database manager closes the clients, and a get after close is refused with ResourceManagerClosedError. Concurrent sweepers on other processes need no coordination: a purge claim is a row lock the registry hands to one of them. The event backend's service locator creates a session's collection when it finds none, strictly. With creation arbitrated across processes, that create can lose to another worker's; the locator then opens the collection the winner registered instead of failing the request. The semantic manager's open-or-create needs nothing: the store adopts the winner's row itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn (cherry picked from commit 23ccac0)

…ck) (MemMachine#1606) * Regenerate the OpenAPI document under the locked FastAPI `docs/openapi.json` predates the FastAPI release in `uv.lock` (0.141.1), whose `ValidationError` component carries `input` and `ctx`; regenerating the document with `docs/tools/generate_openapi.py` adds the two fields and changes nothing else. Separate from the API changes above it so their diffs of this file show only what they change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn * Remove per-project filterable properties A project could declare `properties_schema`, a set of caller property keys with types, on its long-term memory configuration; the event backend merged it into the vector store collection's indexed schema and rejected filters on any other `m.<key>`. That let a tenant create database resources (indexes, columns) by naming them in a request, which is what forced per-collection native resources named by a hash of their schema on the backends that limit them. The option is removed from the server configuration, the project API and the memory-configuration API, the Python SDK, the sample configurations, the configuration docs and the OpenAPI document. A filter may name any `m.<key>`; the stores evaluate it on the properties they hold. What a store indexes is decided by the deployment, not per project. A breaking API change on `speedkick`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit a8322a7)

The event backend wrote every property of an event into its vector record and mapped the caller's whole filter onto the vector store, so a user key that a deployment never declared was both stored and filtered there: on Qdrant and Milvus as unindexed payload a filtered query scans for. The segment store already holds every property and already receives the whole filter for the context windows, so the vector side only duplicated work the segment store does anyway. The vector record now carries the keys the collection declares: EventMemory's reserved segment uuid and timestamp, and the `_`-prefixed system properties an adapter stamps on the event, which after MemMachine#1670 are exactly the collection's schema. The vector store is queried with the conjuncts of the filter that name only such fields; a conjunct is dropped whole when any field under it is a user property, so dropping only ever widens the vector search, and the segment store narrows it back on the windows. User keys never reach the vector store, so a tenant's properties cannot shape what it stores or scans. `filter_fields` joins the filter parser: every field name a tree addresses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…res' lookup to get_partition A vector store's logical collection becomes a partition, the segment store's word for the same thing, and both stores' lookup is get_partition, answering None like a Python get. Identifiers only, produced by the script below; the (namespace, name) identity, the per-partition config and every docstring are as they were, and the next change gives them their meaning. The native clients' create_collection and delete_collection keep their names, as do the registry's table names until the next change redefines its keys. On this base the vocabulary also reaches what sits under the rename now: the SQL registry module and its classes, the stale-handle error, the purge method (`purge_deleted_partitions`, the segment store's name for it) and open-or-create, which stays until MemMachine#1625 removes it. ```sh set -e cd "$(git rev-parse --show-toplevel)" git mv packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_collection.py \ packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_partition.py git mv packages/server/server_tests/memmachine_server/common/vector_store/collection_lifecycle_contract.py \ packages/server/server_tests/memmachine_server/common/vector_store/partition_lifecycle_contract.py git mv packages/server/src/memmachine_server/common/vector_store/sql_collection_registry.py \ packages/server/src/memmachine_server/common/vector_store/sql_partition_registry.py git mv packages/server/server_tests/memmachine_server/common/vector_store/test_sql_collection_registry.py \ packages/server/server_tests/memmachine_server/common/vector_store/test_sql_partition_registry.py git ls-files -z 'packages/server/*.py' 'docs/*.mdx' | xargs -0 perl -0pi -e ' s/VectorStoreCollection(?!Config)/VectorStorePartition/g; s/in_memory_vector_store_collection/in_memory_vector_store_partition/g; s/collection_lifecycle_contract/partition_lifecycle_contract/g; s/CollectionLifecycleContract/PartitionLifecycleContract/g; s/sql_collection_registry/sql_partition_registry/g; s/SqlCollectionRegistry/SqlPartitionRegistry/g; s/RegisteredCollection/RegisteredPartition/g; s/vector_store_collection(?!_schema|_namespace)/vector_store_partition/g; s/open_or_create_collection/open_or_create_partition/g; s/open_collection/get_partition/g; s/purge_deleted_collections/purge_deleted_partitions/g; s/def create_collection\(/def create_partition(/g; s/def delete_collection\(/def delete_partition(/g; s/\.create_collection\((\s*namespace=)/.create_partition($1/g; s/\.delete_collection\((\s*namespace=)/.delete_partition($1/g; s/\.create_collection(?=\s*=\s*AsyncMock|\.assert_|\.side_effect|\.await_count)/.create_partition/g; s/\.delete_collection(?=\s*=\s*AsyncMock|\.assert_|\.side_effect|\.await_count)/.delete_partition/g; s/"create_collection"/"create_partition"/g; s/"delete_collection"/"delete_partition"/g; s/"open_or_create_collection"/"open_or_create_partition"/g; s/only delete_collection is invoked/only delete_partition is invoked/g; s/test_delete_collection_/test_delete_partition_/g; s/open_partition/get_partition/g; ' uv run ruff check --fix --quiet packages/server uv run ruff format --quiet packages/server ``` Then, by hand: the Milvus timeout test's list of native client requests keeps `"create_collection"`, which the script had turned into the client's unrelated `create_partition`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…uilt by the composition root A vector store was a factory of logical collections, each identified by a (namespace, name) pair and created with its own dimensions, metric and schema; Qdrant and Milvus shared one native collection among logical collections of equal configuration, under a name derived from a hash of that configuration, and the registry mapped (namespace, name) to it. A store is now one collection: `VectorStore(collection, vector_dimensions, similarity_metric, indexed_properties)` names its one native collection (or its tables and index files) at construction, every partition of it shares the collection's dimensions, metric and schema, and `provision()` creates the collection's durable resources idempotently, before `startup`. `create_partition(key)`, `open_or_create_partition(key)`, `get_partition(key)` and `delete_partition(key)` take a string key; a partition is a payload value (Qdrant), a partition-key value (Milvus) or a pair of tables (the SQLite stores) inside the collection, and the registry beside it records what each partition was created under, so a store built with other dimensions, another metric or another schema raises VectorStorePartitionSchemaMismatchError instead of reading columns and vectors that are not there. Collection names may be 64 bytes; the hash-derived native names go, and with them `VectorStoreCollectionConfig` and the per-partition config. The SQL registry is keyed by (backend, collection, partition key) and stores the partition's schema. `DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=, similarity_metric=, indexed_properties=)` builds and caches one store per (backend, collection), provisioning it until a schema command exists; asking for a collection again with other dimensions, another metric or other keys is a configuration error. The event backend's collection is `long_term_memory__<embedder>` and the semantic memory's `semantic_memory__<embedder>`, one cell of the purpose-by-embedder matrix each; the two SQLite stores of one backend share its engine. The event backend still opens a session's partition lazily, and the registry arbitrates the race between workers. The SQLite stores change shape only: their registry tables become `vector_store_sqlite_pt` and `vector_store_sqlite_vec_pt`, keyed by collection and partition key, the pending-operation log is keyed the same way, and per-partition table names embed the collection. No migration is provided; existing SQLite vector data is orphaned. Their data path, the pending-operation protocol and the index files are as they were. The data path is as it was: a partition stores every property of a record and filters on any key, with the declared keys indexed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

`_params(..., **overrides)` in the four store test modules built a `dict | dict` union and spread it into the parameter model, which ty reads as untyped keyword arguments; a `dict[str, Any]` updated with the overrides is what it accepts. The Milvus timeout test also passes the vector it built rather than the record's optional one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

`_vector_store_purge_tasks` is keyed by (backend, collection) since a store is one collection; the test that checks the purge task does not pin the manager still filed its task under the backend name alone, which ty rejects. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…on registry The `registry_database` descriptions still called it the collection registry and said the backend cannot arbitrate collection creation: the wording of MemMachine#1631, when a registry row stood for a collection. A row is a partition of the store's one collection now, and the docs say so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…emMachine#1588) * Atomically swap vector search engine index files on save SQLiteVectorStore persists each collection's index by calling the search engine's save(), which wrote directly to the final path. A crash mid-write left a truncated/corrupt file. Because index_saved=True makes the on-disk index a durable contract (missing/corrupt is a hard IndexLoadError, not a silent empty rebuild), an interrupted save could render a collection unrecoverable. Write the index to a sibling temp file and swap it into place with os.replace (atomic on POSIX and Windows on the same filesystem), so a reader sees either the old or new index, never a partial write; a failed save leaves the previous index intact. Leftover temp files are cleared on load so a crash does not leak them across restarts. Implemented in the engines (shared index_persistence helper) rather than in SQLiteVectorStore/SQLiteVectorStoreCollection, since the index save location and number of files written differ across engine implementations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Make the index swap durable, not only atomic The swap protects a reader from a torn index, but the vector store also trims its pending-operation log once `save` returns -- and that log is the only other copy of those vectors, since the records table stores no vector column. So the swap reaching disk is load-bearing rather than a bonus: - fsync the parent directory after the replace, since POSIX `rename(2)` leaves the new directory entry in the page cache. Best-effort and ignored on failure, matching SQLite's `unixSync`; a no-op on Windows, which has no equivalent operation. - stop swallowing a failed fsync of the temp file. SQLite draws the same line -- a file fsync failure raises SQLITE_IOERR_FSYNC while a directory fsync failure is ignored -- and `EIO` means the writeback already failed and the dirty pages were dropped, which is exactly when the save must not be reported as committed. The existing cleanup then leaves the previous index in place with the log untrimmed, so the next save retries. - use F_FULLFSYNC on macOS, where plain `fsync` leaves the data in the drive's volatile write cache, falling back when a filesystem refuses it. State the resulting obligation on `VectorSearchEngine.save` itself, since that is what the store now relies on: replace atomically, then make the replacement as durable as the platform allows. An engine whose backend already implements the whole protocol can delegate to it and skip these helpers; the rest use `atomic_index_write`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Let the engine own index durability, not the vector store The pending log holds the only durable copy of a vector between checkpoints -- the records table has no vector column -- so trimming it is safe only at an instant when the index provably holds those vectors. The temp-write + rename protocol this PR shipped could not provide that instant. A rename changes a directory entry, and Windows exposes no way to flush one: os.fsync is _commit, which is FlushFileBuffers, which is for file data, and you cannot open a directory to fsync it. The decisive evidence is SQLite's own -- it threads a directory-sync flag through every commit-relevant directory operation, honors it in unixDelete, and declares it /* Not used on win32 */ in winDelete. So os.replace could return, _save_collection_index could commit its trim durably behind it, and a power cut could still roll the rename back: records forward, index back, no copy of the difference left. MOVEFILE_WRITE_THROUGH is not a fix; its documented guarantee covers copy-and-delete (cross-volume) moves, not same-volume renames. Take SQLite's answer, which was not to harden the directory operation but to stop using one as a commit point (PERSIST commits by zeroing a header, TRUNCATE by truncating, WAL by appending frames). A base path now expands into two index slots plus a generation record each, created once and thereafter only overwritten. A checkpoint writes the index over the inactive slot and flushes it, then writes that slot's generation record and flushes that. The record is the commit, and it is a write into a file that already exists. It holds the generation and its bitwise complement, so a torn write reads as absent rather than as some other generation -- all or nothing without needing single-sector atomicity from the hardware. load takes the highest believable generation, and deliberately does not fall back to the older slot when the published index will not parse: the log was trimmed against the newer one, so the older is stale by exactly the ops that can no longer be replayed. Both backends already write straight to the path they are given, which is what this protocol wants -- verified that repeated saves preserve the inode and leave no stray files -- so no engine gains a temp file, a buffer, or a rename. Durability is entirely the engine's, including which artifact is live. The store keeps no slot pointer, manifest, or generation, so no schema change and no migration: what remains is one rule, never trim past what save says is durable, and _save_collection_index already had that order. index_path becomes index_base_path since it no longer names a file, and discarding a collection asks the engine layer which files that covers. BREAKING CHANGE: an index written by the previous protocol is not published under the new one, so a collection with index_saved=True raises IndexLoadError until its index directory is cleared and the records re-ingested. Anomaly tests walk every crash point in the publish sequence by constructing the on-disk state each would leave, plus one that pins the ordering itself (a failed index write must publish nothing) since state-based tests cannot observe it. Verified against three deliberate breaks -- dropping the complement check, writing the record first, and reusing one slot instead of alternating -- each caught. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Publish the index atomically, and stop promising durability The two-slot generation-record protocol bought a guarantee we have decided not to make: that a save survives a power failure. Every engine would have to implement and maintain that protocol, and the failure it buys out is bounded -- search recall for the records applied since the last checkpoint, repaired by re-ingesting them. The direction that actually costs, a published index that will not parse, is closed by the atomic swap on its own. So this returns to the temp-file-plus-rename publication and spends the difference on stating the contract instead of strengthening it: `save` publishes atomically, never durably; the store trims the pending log behind a publication a power failure can revert; a record whose vector is lost that way still resolves by uuid, is absent from search until it is upserted again, and nothing here detects the gap for the caller. Reverts the durability and engine-owned-publication commits, keeps the atomic swap, and adds a store-level test that reconstructs a reverted publication deterministically -- restore the previous index bytes after the trim -- to pin the direction it fails in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Report a lost embedding as lost, not as a missing feature `update_feature` reads the stored embedding back when a caller updates a feature without supplying one, and that is the only place in the server that depends on the index still holding a vector. With publication now atomic rather than durable, a power failure can leave a feature whose row is intact and whose vector is not -- a state this path reported as "Vector record not found", which points the caller at the wrong thing and hides the repair. Split the two cases. A record that is genuinely absent keeps the old message; a record whose embedding the index no longer holds says so and names the fix, which is to pass a fresh embedding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Let a failed fsync fail the save, and cut the essay around it `_flush_to_disk` wrapped its fsync in `contextlib.suppress(OSError)` and called itself best-effort. A failed fsync is exactly the evidence that the bytes are not safe to publish -- on Linux an EIO from fsync means writeback failed, reported once and then cleared -- so swallowing it and renaming anyway published a file we had positive evidence was bad. The safeguard cost something and, in the one case it existed for, guaranteed nothing. Nothing tested it either. Let it propagate. `atomic_index_write` already unlinks the temp and re-raises, so a failed flush now leaves the previously published index standing, which is the correct outcome. A test pins that. The fsync is not best-effort, and the docstring should not have said so: it rules out a class rather than narrowing a window. Because the flush completes before the rename is issued, and a durable write does not un-happen, the new name can never appear over incomplete bytes. What the missing directory fsync costs is the other direction -- the rename may not survive, so the publish reverts -- and that is the benign one this store already accepts. The module docstring was 76 lines against 49 of everything else, most of it argument rather than documentation: a walk through SQLite's `unixDelete` / `winDelete` sync-flag handling, and a rejected two-slot commit protocol. That is the PR's case for the design, not something to re-read every time someone opens a 20-line module, and the PR body carries it. What a reader here needs is the guarantee, the non-guarantee, and the cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Say why the swap is a rename, and what the reopened fd cannot see Two things the module was silent on. Why a rename at all. The stronger answer is to put the commit point inside the file, where an fsync reaches it portably -- SQLite never renames, and commits by truncating or zeroing its rollback journal, or in WAL mode by appending frames whose checksums make a torn tail self-identifying. Both need the writer to own the file format. A search engine owns its own and exposes `save(path)`, so above that call a rename is the only atomicity primitive left, and an engine whose format already commits that way needs none of this. Worth saying, because "why not do the better thing" is the first question the module invites. What the reopened descriptor cannot see. Flushing is fine on a fresh fd -- dirty pages belong to the file, not to the descriptor that dirtied them -- but error reporting is not: Linux hands a writeback error to descriptors open when it was recorded, so one recorded between the engine's close and this open is never reported and the save proceeds on bytes already known bad. Same shape as the 2018 PostgreSQL fsync report. It cannot be closed from here: the engine writes through its own descriptor and closes it before returning, and closing the window needs an engine that writes through a handle the caller supplies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Fsync the index on a descriptor that predates the write The fsync was on a descriptor opened after the engine had written and closed its own, which flushes correctly -- dirty pages belong to the file, not to the descriptor that dirtied them -- but reports nothing useful. Linux samples the writeback error sequence when a file is opened, so a descriptor opened after an error was recorded never learns of it: the fsync returns success and the save publishes bytes already known bad. Same shape as the 2018 PostgreSQL fsync report. Open the temp before yielding it and hold it across the caller's write, so the descriptor predates the bytes and any error from writing them is reported here, where it fails the save. That assumes the caller writes in place. Both engines do -- verified: the inode is unchanged across `save_index` and `save`, and the held descriptor sees the written size -- but it is their behaviour, not their contract. An engine that built a file of its own and renamed it over the temp would leave this descriptor on an orphaned inode, and the fsync would report on a file nobody is about to publish. So it is checked before the fsync, and a mismatch fails the save rather than passing it quietly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * State the in-place rule where the caller reads it The descriptor held across the body only flushes what the body wrote if the body writes the yielded path in place, and that requirement was recorded in `_flush_to_disk` -- a private function nobody writing an engine opens. It belongs on `atomic_index_write`, which is the API they use, alongside what happens when it is broken: an `OSError` and no publication, so the mistake surfaces at the first save rather than at a power cut. `_flush_to_disk` keeps the mechanism -- why the descriptor has to predate the write -- and now points at the rule instead of restating it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Ask the drive to flush on macOS, where fsync does not `os.fsync` is not the same guarantee on all three platforms this ships to. On Linux it flushes to the device, and on Windows `FlushFileBuffers` does the same. Darwin's `fsync` explicitly does not: it returns once the data reaches the drive, which may hold it in a volatile write cache. So on macOS the ordering this module is built on -- data durable before the rename is issued -- did not hold at the device, which is exactly the case it claims to rule out. `F_FULLFSYNC` asks the drive to flush that cache. Filesystems that cannot refuse it, and there `fsync` is the most that can be asked, so a refusal falls back; any other error is a write failure and propagates, as before. The flush-failure test patched `os.fsync`, which Darwin no longer reaches. It patches the module's own `_fsync` instead, which every platform does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Stop guessing which errnos mean "this drive cannot do that" The fallback from `F_FULLFSYNC` to `fsync` was gated on an errno allowlist -- ENOTSUP, EOPNOTSUPP, EINVAL -- so that a genuine write failure would propagate rather than be quietly downgraded. Probing the actual returns on Darwin shows the list is both incomplete and partly invented: ENOTSUP=45 EOPNOTSUPP=102 distinct here, so both are needed /dev/null F_FULLFSYNC -> ENODEV(19), while fsync succeeds pipe/socket F_FULLFSYNC -> EBADF(9) EINVAL never came from F_FULLFSYNC at all; it came from fsync So ENODEV -- a real refusal, on a path anyone can reproduce -- would have raised instead of falling back, and EINVAL was in the list by analogy rather than evidence. What a network mount answers is not knowable from here, which makes the whole list a guess that fails closed on whatever it missed. This codebase does not classify driver errors by guessing, and this was that. Fall through on any failure instead. It is not a suppression: `fsync` runs on the same descriptor and raises in its turn, so a flush that cannot happen still fails the save. What the fallback gives up is the drive-cache flush -- the guarantee this had before `F_FULLFSYNC` was asked for at all. That is also what SQLite does with this same call, for the same reason. The test drives it through `/dev/null`, which refuses with ENODEV and accepts `fsync`; it fails against the allowlist and passes without it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Fail the save when the drive cannot be told to flush `F_FULLFSYNC` failing fell through to `fsync`, on the reasoning that a refusal is a statement about the filesystem rather than a write failure. That reasoning does not survive asking what `fsync` alone actually buys on Darwin. Against a process or kernel crash it is enough: the data has left the OS for the drive before the rename is issued, so the rename dies in the page cache and the old index stands. Against power loss it is not. The data sits in the drive's volatile cache, the rename's metadata joins it moments later, and nothing orders them -- and the rename is a few bytes against an index of megabytes, so a drive flushing as it pleases can easily put the new name on media while the bytes behind it are still queued. That is the torn publication this module exists to prevent, in precisely the scenario its docstring is about. So the fallback answered a request for ordering with a flush that does not provide it, and said nothing. A filesystem that cannot order data ahead of a rename is not one to publish an index onto; raise, and let the operator point `index_directory` at storage that can. This is also the simpler code. Refusal and failure now take the same path, so no errno is inspected -- there is no line to draw and no list to get wrong, which is what the previous two revisions kept getting wrong in opposite directions. The `/dev/null` test went with the fallback it pinned. The earlier defence of falling back rested on network and FUSE mounts being a realistic home for an index directory. They are not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 21105d6)

Never reuse a row id in SQLiteVectorStore The records table used a plain INTEGER PRIMARY KEY, which is SQLite's rowid, assigned as max(rowid) + 1: deleting the highest row freed its id for the very next insert. query() scores keys in the search engine and resolves them to rows in a second step, holding nothing in between, so a reused id let a record that was never scored come back wearing the score of the record that was. Nothing about that result looks wrong: the record exists and the score is in range. Declare the table with sqlite_autoincrement=True so ids are never reused. A stale engine key then matches no row and is dropped. Both tests fail without the flag: one pins the id policy directly, the other parks a query between scoring and row lookup, retires the scored record, inserts another, and asserts the query returns nothing. First half of MemMachine#1468. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…(speedkick) (MemMachine#1612) Own the search engine's concurrency in the store, not in each engine Each engine wrapped its five methods in its own read-write lock and the abstract class promised concurrent use. The store is the only caller, and it already knows which calls must exclude which: a search shares the engine with other searches, and everything else runs alone. Holding that lock in the store puts every exclusion in the one file that reads the engine, and lets a rule the engines could not express hold: a rewrite's remove and add are one step to a reader, where before a search could run between them and see neither version. Engines drop their lock and keep only the index calls; the abstract class now states that the owner serializes. The store keeps one read-write lock per collection beside the engine, kept for the store's lifetime like the engine's other per-collection state, and takes it at every engine call: searches on the read side; mutations, loads, and the index save on the write side. A rewrite's remove and add sit under one hold. The save's trim runs after the lock is released, so readers wait for the file write and never for SQL. The row-id regression test from MemMachine#1589 parked inside a wrapper engine's search, outside the real engine's lock; under the store's lock that parks the read side, and the writes it then awaits cannot proceed. It now parks where it meant to, between the engine search and the row lookup. One new test pins the one-step rewrite; it fails against the engines' own locks. The turbovec engine in flight carries the same lock and needs the same subtraction. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…edkick) (MemMachine#1607) Serialize a collection's writes so the engine sees them in order A write commits to SQLite and only then applies to the search engine, so two writers to one uuid could reach the engine in the opposite order to the one they committed in: an upsert overtaking a delete re-adds a vector for a record that is gone, and two upserts inverting leave the engine serving the older vector. Never reusing a row id does not cover this, because an upsert of an existing uuid keeps its row id. A save in that window costs a write outright: it publishes the index and trims every applied log row, and a write that applied after the index was written is then in neither. A per-collection asyncio.Lock now spans a write from SQL commit through engine apply, mark-applied, and any save it triggers; shutdown's save takes it too. The lock belongs to the store, not to a collection handle: a handle is constructed per open_collection call, so several can address one collection, and only a shared lock serializes them. Readers are untouched. Three tests fail without the lock, each interleaving made deterministic by gating the engine: an upsert overtaking a delete of its uuid, a save trimming a write it did not publish, and that overtake across two handles, which a per-handle lock passes. The rest pin behavior the lock must preserve: an upsert surviving a delete of another record, disjoint concurrent upserts and deletes, writes racing a checkpoint, and batches that name one uuid twice. Fixes MemMachine#1468. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…edkick) (MemMachine#1608) Refuse a pending row replay cannot honor, instead of dropping it Replay matched `operation_type == "upsert" and vector is not None` and let everything else fall through. An upsert row with no vector, a vector that is not a whole number of float32s, or an unknown operation_type was skipped, and a decodable vector of the wrong width reached the engine, which refused it with its own error at startup. The skipped rows were the worse case: and because the skipped row reached neither the engine remove set nor the mark-applied update it survived the restart to be skipped again on the next one. Between a write returning and the next index save the log holds the only copy of the vector, so the outcome was a record that exists in SQLite and can never be found by search. That is damage to a durable record, not a state to heal. Replay now raises PendingOperationCorruptError for all four, naming the collection, the row and the fault, and leaves the log intact for whoever repairs it. Both error types' docstrings now say what a caller should do: read the cause of an IndexLoadError before choosing a remedy, and never clear the log to get past a PendingOperationCorruptError. Four tests corrupt a log row each way and assert the restart refuses. The rest of TestPendingLogStates pins what replay guarantees for intact rows: a rewritten uuid replays its last write, an upsert then delete stays deleted, a failed save leaves the write replayable, and the save threshold counts log rows rather than writes. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…MemMachine#1609) Take SQLite's write lock at BEGIN, not at the first write `delete` resolves row ids and then writes, in one transaction. Under SQLite's default deferred BEGIN the write lock is taken at the first write, so another writer can take it between that read and the write, and the transaction that read first is the one that loses: its own write cannot upgrade, and SQLite reports that immediately rather than waiting out busy_timeout, because waiting could only deadlock. Reproduced in both journal modes: journal_mode BEGIN other writer our write delete deferred shut out fails delete IMMEDIATE shut out ok wal deferred commits fails wal IMMEDIATE shut out ok The store now emits BEGIN explicitly and lets a transaction ask for BEGIN IMMEDIATE, which every write path does. The mode is chosen per transaction, not per engine: a hook that asked for IMMEDIATE unconditionally would make every read take the write lock, and two readers would then serialize against each other. Two tests fail without it. One holds a competing lock across a read-then-write transaction and asserts that transaction completes; asserting instead that the other writer is excluded passes either way, because a deferred BEGIN shuts it out too, later and by a different lock. The other races four creates of one name: under a deferred BEGIN the losers read no stored config, go on to CREATE TABLE, and fail there with "table already exists" instead of VectorStoreCollectionAlreadyExistsError. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…k) (MemMachine#1610) Give every write a fresh row id, so a key names one version An upsert of an existing uuid kept its row id, so one engine key spanned every version of a record. A query reads the engine, then each candidate's properties, then its uuid, at three instants with nothing held between them, and a rewrite of the very record being returned could land between those reads: the score of one version paired with the filter verdict of another, or, when the overfetch loop rescored a key whose verdict was already cached, the other way round. No lock covers this, because reads deliberately take none. Every write now deletes the previous row and inserts a new one in the same transaction, stages a delete for the old key beside the upsert for the new, and the engine removes the old key and adds the new. Rows are immutable, so a key names one version: the score computed under it, the filter verdict for it, and the uuid it resolves to belong to that version, and a key whose version has been rewritten resolves to no row and is dropped. AUTOINCREMENT remains what keeps a retired key from being reissued, and the write lock what keeps two rewrites of one record in order. A batch that names a uuid twice is collapsed to its last record before the insert, which the on-conflict update used to do implicitly. The save threshold counts log rows and a rewrite now adds two, so a rewrite-heavy workload checkpoints about twice as often; that test's expectation changes accordingly. Measured on this machine, medians of three runs, records per second, 64-dimensional vectors, file-backed store with an index directory: batch save threshold insert before / after rewrite before / after 500 1000 20477 / 19290 16450 / 12902 500 none 21228 / 22486 19013 / 15899 1 none 341 / 345 316 / 274 Inserts move within run-to-run noise, in both directions. Rewrites cost 13-22% more, the upper end where the doubled log rows double the saves. Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

… (speedkick) (MemMachine#1598) * Answer with cosine scores, not with stored vectors Two changes to one contract, taken together so its signatures are not churned twice. Cosine is now the only similarity. Every embedder MemMachine ships already produced vectors meant to be compared that way -- OpenAI hard-coded it, Bedrock defaulted to it, SentenceTransformer only reported what the model declared -- while every layer that touched a score paid for the other three metrics in direction flags, threshold directions, and per-backend tables mapping the enum onto native metric names. `SimilarityMetric` is gone; scores are cosine similarities in [-1, 1] and the names say so: `QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`, `query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which no longer needs a direction to be meaningful. The Bedrock embedder's `similarity_metric` config key goes with it, and the install and configuration docs drop it. And `VectorStoreCollection.get` is removed, with nothing offered in its place. It had one production caller: the semantic storage read a feature's stored embedding back so it could write the same embedding again with fresh properties, because `upsert` demands a whole record. `return_vector=True` was passed at that one call site and nowhere else, and `VectorSearchEngine.get_vectors` existed to serve it. `set_properties` serves that caller directly -- correcting a record's properties no longer requires holding its vector -- and on SQLite and sqlite-vec it is an UPDATE that never touches the index. No scoring-by-id entry point takes `get`'s place. One would be needed if a caller assembled a candidate set outside the store and asked for those ids to be scored, which is what a selective-filter plan running above the store would do. Property filtering stays inside the store instead, so the candidate set stays there too, and a store-side regime can reach engine keys directly without a public method addressed by record UUID. Two consequences beyond the vector store. The vector graph stores carried a metric per stored embedding, as a companion property beside every vector; that is gone and `Node.embeddings` holds plain vectors. NebulaGraph indexes only L2 and IP and its `cosine()` cannot be APPROXIMATE, so with cosine alone no index it can build serves a query -- its ANN branch and vector index creation could no longer run and are removed; search there is always exact KNN. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Let each store own its own mapping, and stop copying it The vector store is not the authority for anything but vectors, so a consumer that read a record's properties back out of it was reading a copy. Both consumers did that, and for the same reason: to get from a search hit back to a domain id. What the copy cost them differed. Semantic memory's was of mutable columns, correct only as far as the last write to it and stale from then on. Event memory's was an immutable uuid pairing, so it stayed correct -- it simply duplicated a mapping the segment store already served from an index, and spent a reserved property name doing it. `QueryMatch` therefore answers `record_uuid` and a score. Properties stay stored and filterable, because a filter is evaluated against the copy rather than trusted as the record, and are no longer returned. That is the end of `return_properties`, of `Record` on the read path, and of `set_properties`, which existed only to keep a copy fresh that nobody reads now. Event memory used a `_segment_uuid` property to reach a derivative's segment. The segment store already holds that mapping on the derivative's own row, non-null, under a primary key that leads on exactly the columns the lookup filters -- so the copy bought nothing an indexed read does not, and `get_segment_uuids_by_derivative_uuids` mirrors the forward lookup that was already there. The property is gone, and with it a reserved field name. Semantic memory had no mapping to reach for. Its record uuid was `uuid5(namespace, feature_id)`, which is one-way, so the feature id had to ride along in the properties. It now owns a `vector_uuid` column, unique and minted per feature, and resolves hits through it -- the vector id is the vector's own, and the caller keeps the correspondence. Every property that collection carried was a copy of a column on the feature row, never filtered on, so the payload goes entirely. The delete paths read that column before deleting the rows, since the row is what says which vector record a feature owns. Data on speedkick does not survive this: existing semantic features have no `vector_uuid`, and existing collections carry properties nothing reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Restore the Nebula tests a bad edit took with it Cutting `test_similarity_metric_mappings` searched for the next `\ndef ` or `\nclass ` to find where the function ended. Every test after it is `async def`, so both searches missed, the end fell through to end-of-file, and the edit deleted 429 lines: ten test functions, of which four had anything to do with similarity metrics. Gone with it were the empty-input cases for adding nodes and edges, the none-property cases for both, the wrong-collection delete, the nonexistent-uid read, the multi-property directional search -- and `test_search_similar_nodes_cosine_metric`, the test for the one metric that survives this change. Nothing caught it because these tests skip everywhere, CI included: the vector support they exercise needs NebulaGraph Enterprise >= 5.0, which the fixture reaches at NEBULA_HOST and skips without. Rebuilt from the file on speedkick with the metric stripping applied, then removing only what has no subject left: the mapping helpers' test, the dot and manhattan searches, and the ANN search -- Nebula indexes only L2 and IP, so with cosine alone no index it can build serves a query. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Keep NebulaGraph's ANN by spelling cosine as an inner product Nebula's `cosine()` is KNN-only -- it cannot take APPROXIMATE -- and its vector indexes offer only L2 and IP. From that I concluded cosine could never be approximate here and deleted the ANN search branch and vector index creation as unreachable. That was wrong: cosine similarity between unit vectors *is* their inner product, so an IP index over normalized vectors gives cosine ranking with ANN. Vectors are normalized in `_vector_to_gql_literal`, which is the single place any vector becomes a literal -- node writes, edge writes, ANN queries and exact queries all pass through it, so the stored side and the query side cannot disagree about it. `inner_product()` DESC against an IP index replaces the metric lookups. IP rather than L2, though both rank identically over unit vectors (‖a-b‖² = 2(1-cos), monotone in cos, and verified equal by argsort). The reason is not numerical: measured over near-duplicate float32 unit vectors, recovering cosine as 1 - d²/2 is 1.1x worse than reading it off IP, which is nothing. It is that the contract answers a cosine similarity, and with IP over unit vectors the index score already is one -- no conversion, no assumption about whether the engine hands back the distance or its square, and no result landing outside [-1, 1] needing a clamp. `test_search_similar_nodes_ann` comes back with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL * Drop the constant aliases left over from configurable metrics Collapsing the metric enum left an instance attribute that only ever copied a class constant (self._space = self._SPACE) and, in the Nebula store, a local that copied a constant into a second local before use. Read the constants where they are used. The Nebula metric names move to module scope alongside the package's other fixed identifier constants; they are neither per-instance nor overridden. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit abf92a3)

…emMachine#1603) Require a vector on the record type nothing reads back `Record` is written and never returned, so `vector` being optional described nothing a caller could do. Every store rejected `None` at its own `upsert` with the same message, four copies of one rule that the type could state once. `properties` was optional in the same way, so Qdrant carried `record.properties if record.properties is not None else {}`. The vector is required and the properties default to `{}`. The four checks go, and rejection moves to the model, where a caller finds it at construction rather than at a write. Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit cdeef88)

…quest The event backend created a session's vector store collection and segment store partition on the first request that opened the session, so a search or a write for an unknown session created storage as a side effect, and the service locator was the only place that knew both stores' create paths. The owner is the session. Every path that creates a session row runs through EpisodicMemoryManager._create_session, which inserts the row and, when the row is new, creates the session's partitions in its segment store and its vector store (create_episodic_memory_storage); an equivalent re-create accepts the row and leaves the storage as it is. The request path binds handles with the stores' lookups and raises SessionPartitionMissingError when a partition is absent: a session without its storage is broken, not new. Deleting a session with no open instance deletes its partitions by key, so a session whose storage was never fully created can still be deleted. MemMachine.create_session goes through the manager for the same reason. The semantic manager owns its one collection and creates it, once, at the storage's first use. With that, nothing calls the stores' open-or-create. The API is unchanged: the manager's open-or-create still creates a session a memory request names, now through the same path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Nothing calls them since a session's storage is created with the session: `open_or_create_collection` and `close_collection` leave the vector store interface and its four backends, `open_or_create_partition` and `close_partition` leave the segment store interface and its implementation, and the two config-mismatch errors that only open-or-create raised go with them. A store creates on `create_*`, strictly, and looks up on `open_*`, answering None; create-if-absent is the owner's, where the key's provenance is known. Source changes are deletions only. The tests that exercised open-or-create as a fixture use a test-side create-if-absent instead, and the tests of its own semantics go. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…tization `QdrantConf` gains `hnsw_config`, `optimizers_config` and `quantization_config`, plain mappings mirroring qdrant-client's `HnswConfigDiff`, `OptimizersConfigDiff` and `QuantizationConfig`, so qdrant-client stays optional for configuration parsing; the store's params validate them against qdrant's own models. They apply to the store's data collection, never to its registry collection. `m` must be 0 or unset: the collection is multi-tenant and disables the global graph in favor of per-partition payload indexing, so a deployment tunes `payload_m`, which defaults to 16 as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…w them in the samples and the parameter table The option docstrings and descriptions still spoke of "native collections" and "registry collections": since MemMachine#1631 the store is one collection and its registry is relational tables. The sample configurations gain a commented block with the three keys, checked against qdrant-client's models, and the configuration page's parameter table gains their rows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

edwinyyyu mentioned this pull request Sep 14, 2026

[vector store 1/12] Remove per-project filterable properties (speedkick) #1606

Merged

edwinyyyu marked this pull request as draft September 14, 2026 20:03

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 4 times, most recently from 28884f8 to 6ef5caf Compare September 14, 2026 20:33

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 6ef5caf to 1364e65 Compare September 14, 2026 21:40

edwinyyyu changed the title ~~(Depends on #1606) Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)~~ [vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 14, 2026

edwinyyyu mentioned this pull request Sep 14, 2026

[vector store contract 6/6] Close the filter union, and make negation the complement on every backend #1616

Draft

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 5 times, most recently from 556eb26 to 2a1da2c Compare September 14, 2026 23:16

edwinyyyu changed the title ~~[vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)~~ [vector store 11/12] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 14, 2026

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 2 times, most recently from 556eb26 to 918e7cb Compare September 15, 2026 17:26

edwinyyyu mentioned this pull request Sep 15, 2026

[vector store scale-out 2/5] Mint an incarnation per collection life in a SQL-arbitrated registry, so any process may serve any Qdrant or Milvus collection #1631

Draft

edwinyyyu changed the title ~~[vector store 11/12] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick)~~ [vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 15, 2026

edwinyyyu mentioned this pull request Sep 15, 2026

Add session, source and expansion to EventMemory (speedkick) #1597

Closed

edwinyyyu and others added 29 commits September 21, 2026 15:25

edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 4cf0eff to e298be2 Compare September 21, 2026 22:45

Navigation Menu

[vector store contract 4/6] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization - #1618

[vector store contract 4/6] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization#1618
edwinyyyu wants to merge 38 commits into
MemMachine:mainfrom
edwinyyyu:feat/qdrant-collection-options-speedkick

edwinyyyu commented Sep 14, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

edwinyyyu commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the change

Stack

Verification

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

edwinyyyu commented Sep 14, 2026 •

edited

Loading