Skip to content

Example 29: hybrid agent-memory retrieval in one SQL statement (tenant pre-filter + full-text + graph) - #32

Open
tae898 wants to merge 1 commit into
mainfrom
example-29-hybrid-retrieval
Open

tae898 wants to merge 1 commit into
mainfrom
example-29-hybrid-retrieval

Conversation

@tae898

@tae898 tae898 commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator

What

Example 29, examples/29_hybrid_memory_retrieval.py (27 and 28 are taken by #30 and #29, so 29 is the next free number; the three README entries will conflict in the same spot and need a trivial merge). For an agent-memory or GraphRAG builder who today glues a vector store, a keyword index and a graph store together: the whole retrieval is one SQL statement, and everything goes through db.query / db.command (no new Python wrapper). Synthetic data only: six tenants, 900 Fact vertices (text plus a fixed-seed numpy embedding), 96 Entity vertices, MENTIONS and RELATES edges.

The point is the tenant pre-filter. A global top-k followed by a tenant test returns fewer rows than asked, or none, whenever the nearest neighbours belong to another tenant (Cypher db.index.vector.queryNodes(index, k, vector) has no filter, so WHERE node.tenant = ... runs on its k results: ArcadeData/arcadedb#10685). vector.neighbors takes the restriction as an option, so the index search only considers the tenant's vectors. The example runs the failing post-filter once for contrast (0 rows) and then the pre-filtered forms.

The statement

SELECT record.fid AS fid,
       record.text AS fact,
       score,
       record.out('MENTIONS').out('RELATES')[tenant = :tenant].name
         AS related_entities,
       difference(
         record.out('MENTIONS').out('RELATES').in('MENTIONS')[tenant = :tenant].fid,
         [record.fid]
       )[0..3] AS related_fids
FROM (
  SELECT expand(`vector.fuse`(
    `vector.neighbors`('Fact[embedding]', :qvec, :k, { where: { tenant: :tenant } }),
    (SELECT @rid, $score FROM Fact
      WHERE tenant = :tenant AND SEARCH_INDEX('Fact[text]', :keywords) = true
      ORDER BY $score DESC LIMIT :k),
    { fusion: 'RRF' }
  ))
)
ORDER BY score DESC
LIMIT :top

Setup is DDL only: CREATE INDEX ON Fact (embedding) LSM_VECTOR METADATA {...} and CREATE INDEX ON Fact (text) FULL_TEXT. One statement was enough, nothing is split. The [tenant = :tenant] guard on the hops matters: the memory holds one stray RELATES edge from an acme entity to a globex entity, and without the guard the hops of the acme facts also return globex facts (the example prints this, and a test asserts it). related_fids is a sample of 3 in no particular order, not a ranking.

What needs 26.11.1

Only the where option of vector.neighbors (ArcadeData/arcadedb#10169, #10185); it is the default --form. --form filter replaces it with filter: (SELECT @rid FROM Fact WHERE tenant = :tenant) and runs on 26.10.2 as well (checked on the published arcadedb-embedded==26.10.2.post1: vector.fuse, SEARCH_INDEX as a fuse source and the rest of the statement work there; where is rejected there with Unknown option 'where' for function 'vector.neighbors'). The CI examples job builds against arcadedata/arcadedb:latest, so the default form runs there; a release build pinned to 26.10.2 would need --form filter.

At the index defaults the search beam narrows from 10,000 vectors up (ArcadeData/arcadedb#10704); this memory has 900, so the defaults are fine, and the docstring says to pass efSearch in the options map for a larger one.

Measured

python 29_hybrid_memory_retrieval.py on the 26.11.1 build (arcadedb_embedded 26.11.1.dev0, this branch's Python over the 26.11.1-SNAPSHOT jars), copied from the run:

engine: arcadedb_embedded 26.11.1.dev0, form: where

requested tenant: acme, k = 10
global top-10, then WHERE tenant: 0 rows (the nearest neighbours belong to other tenants)
pre-filtered top-10 (where): 10 rows

hybrid result (vector + full-text + graph), top 5:
  fid   6  rrf 0.0320  acme user drinks grinder every morning and keeps a barista at home
           related entities: ['beans', 'barista']
           related facts:    [128, 0, 65]
  fid  99  rrf 0.0299  acme user drinks grinder every morning and keeps a barista at home
           related entities: ['beans', 'barista']
           related facts:    [128, 0, 65]
  fid  33  rrf 0.0288  acme user drinks espresso every morning and keeps a barista at home
           related entities: ['beans', 'grinder']
           related facts:    [128, 65, 99]
  fid 134  rrf 0.0164  acme user drinks espresso every morning and keeps a grinder at home
           related entities: ['barista', 'grinder']
           related facts:    [0, 65, 1]
  fid 149  rrf 0.0164  acme user asked the team to order more grinder because the barista ran out
           related entities: ['beans', 'barista']
           related facts:    [128, 0, 65]
check ok: pre-filtered rows are all the tenant's ['acme']
check ok: hybrid rows are all the tenant's ['acme']
check ok: hop results are all the tenant's ['acme']
without the [tenant = :tenant] hop guard the hops also reach: ['globex']

With --form filter on arcadedb-embedded==26.10.2.post1 the output after the engine line is identical, row for row. tests/test_example29_hybrid_memory.py (5 tests: both forms stay in the tenant, the post-filter loses the rows, the forms agree, the hop guard matters) passes on the 26.11.1 build. black, isort --profile black and bandit (the CI invocations) are clean. Nothing here is timed and no performance claim is made.

🤖 Generated with Claude Code

Tenant pre-filtered vector.neighbors fused with a full-text match by vector.fuse (RRF), then graph hops kept inside the tenant. The where form needs 26.11.1; the filter form runs on 26.10.2. Test runs both forms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant