Skip to content

fix(memory): find Chinese and Japanese words inside a sentence in sqlite memory - #7960

Open
Fruiterer333 wants to merge 2 commits into
google:mainfrom
Fruiterer333:fix/sqlite-memory-cjk-search
Open

Fruiterer333 wants to merge 2 commits into
google:mainfrom
Fruiterer333:fix/sqlite-memory-cjk-search

Conversation

@Fruiterer333

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
With FTS5 (the default, fts="auto"), SqliteMemoryService cannot find a Chinese or Japanese word inside a stored sentence. The unicode61 tokenizer indexes 我想买这辆车,价格是二十万。 as two tokens, 我想买这辆车 and 价格是二十万, so searching 价格 or 二十万 returns nothing. The same search works with fts="off" and with InMemoryMemoryService.

Solution:
When a query token contains a Chinese or Japanese character (CJK ideographs, hiragana, katakana), search_memory uses the existing LIKE path instead of FTS5. That path already matches tokens as substrings, so it is what fts="off" does today.

  • No schema change and no re-indexing: existing databases work as they are.
  • Queries without Chinese or Japanese characters still use FTS5 and bm25 ranking, unchanged.
  • Trade-off: these queries rank by the number of matched tokens, then by time (the LIKE path's ranking), instead of bm25, and they scan the user's rows instead of using the index.

I did not change other scripts written without spaces (e.g. Thai), since I have not tested them.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Added test_search_matches_cjk_word_inside_sentence in tests/unittests/memory/test_sqlite_memory_service.py. It runs in both fts="on" and fts="off" modes with four cases: 价格 and 二十万 in a Chinese sentence, and 使う and Python 使う in 私はPythonを使う. Before the fix, the three Chinese/Japanese-only cases fail in fts="on" mode.

$ pytest tests/unittests/memory/test_sqlite_memory_service.py
72 passed
$ pytest tests/unittests/memory/
208 passed

pre-commit run --files on the two changed files passes (ruff, isort, pyink, codespell, ADK compliance checks).

Manual End-to-End (E2E) Tests:

Ran the reproduction script from the issue.

Before:

SqliteMemoryService(fts='auto')    query='价格'       -> 0 memories
SqliteMemoryService(fts='auto')    query='二十万'      -> 0 memories

After:

InMemoryMemoryService              query='price'    -> 1 memories
InMemoryMemoryService              query='价格'       -> 1 memories
InMemoryMemoryService              query='二十万'      -> 1 memories
SqliteMemoryService(fts='auto')    query='price'    -> 1 memories
SqliteMemoryService(fts='auto')    query='价格'       -> 1 memories
SqliteMemoryService(fts='auto')    query='二十万'      -> 1 memories
SqliteMemoryService(fts='off')     query='price'    -> 1 memories
SqliteMemoryService(fts='off')     query='价格'       -> 1 memories
SqliteMemoryService(fts='off')     query='二十万'      -> 1 memories

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

I used Claude Code to help find the issue and draft the fix. I reproduced the bug and verified the fix locally myself.

…ite memory

FTS5's unicode61 tokenizer indexes a run of Chinese or Japanese text as one
token, so searching a word inside the run found nothing. Route queries that
contain Chinese or Japanese characters to the existing LIKE path, which
matches them as substrings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SqliteMemoryService with FTS5 finds nothing for a Chinese or Japanese keyword inside a sentence

2 participants