Repository navigation
native: build_graph_now() rebuilds an existing vector graph - #163
Merged
Merged
Conversation
The native shim ran one vectorNeighbors search, which builds a graph only when there is none; on a built index the call returned in about 2 ms and the vectors written since stayed in the delta (agent-memory study: 26 calls over 13 runs, graphRebuildCount 0, graphNodeCount unchanged; the JVM wheel's calls took about 11 s each and grew the graph). A `build` op in arcadedb_n_vindex calls LSMVectorIndex.buildVectorGraphNow() on every bucket index, the call the JPype wrapper makes. warm_up() keeps its one search: it only loads a persisted graph. Regression test on both builds: an index built over 200 vectors, 100 more written with the automatic triggers off (high floor, inactivity timer 0), then build_graph_now() must leave graphNodeCount at 300 and raise graphRebuildCount. Closes #162 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine rebuilds a graph under 1000 nodes synchronously on the next search (ASYNC_REBUILD_MIN_GRAPH_SIZE), so at 200 vectors the old native search shim rebuilt too and the test passed on the pre-fix wheel. At 1000 + 100 vectors the pre-fix native wheel fails it (graphNodeCount 1000, expected 1100) and the JVM wheel passes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #162
On the native build,
build_graph_now()ran onevectorNeighborssearch. A search builds a graph only when there is none, so on an index whose graph was built the call returned in about 2 ms and every vector written since stayed in the delta (the issue has the timed-study figures:graphRebuildCount0 andgraphNodeCountunchanged after 26 calls on the native wheel, about 11 s per call and a growing graph on the JVM wheel).Change:
NativeExtras.java: abuildop inarcadedb_n_vindexthat callsLSMVectorIndex.buildVectorGraphNow()on every bucket index, the call the JPype wrapper makes._native/schema_objects.py: the shim'sbuildVectorGraphNow()calls that op.warmUp()keeps its one search, since it only has to load a persisted graph.tests/test_vector_index_introspection.py:TestBuildGraphNow, on both builds. An index is built over 1,000 vectors, 100 more are written with the automatic triggers off (mutations_before_rebuild1,000,000,inactivityRebuildTimeoutMs0), thenbuild_graph_now()must leavegraphNodeCountat 1,100 and raisegraphRebuildCount. The graph starts at 1,000 because the engine rebuilds a smaller graph synchronously on the next search (ASYNC_REBUILD_MIN_GRAPH_SIZE), which hid the defect: the old shim worked by accident on small indexes.The docs page (
how-to/vectors.adoc) already saysbuild_graph_now()rebuilds the graph at once; with this change that holds on the native build too.Verification: black, isort, bandit and pyflakes clean. The new test against the pre-fix 26.11.1.dev0 wheels on Linux x86-64: the native wheel fails it (
assert 1000 == 1100ongraphNodeCount), the JVM wheel passes it. A first version at 200 vectors passed on the pre-fix native wheel too, which is how the size threshold above was found. The fixed native image is built and tested by this PR'stest-native.ymlrun.🤖 Generated with Claude Code