[Fruiterer333]

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

• Closes: load_web_page returns an empty string for Chinese and Japanese pages #7956

Problem:

load_web_page keeps only lines with more than three words and counted words with len(line.split()). Chinese and Japanese text has no spaces between words, so every line counted as one word, and a Chinese or Japanese page came back as an empty string.

Solution:

Lines are now measured with _count_words. Whitespace-separated tokens count as before. In a script written without spaces (Han, Hiragana, Katakana, Thai, Lao, Khmer, Myanmar), each letter counts as a word, and the text between those letters counts only if it holds a letter or a digit, so a full stop such as "。" is not a word. Word counts for space-separated text are unchanged.

One trade-off: counting by character keeps a short Chinese label of four or more characters, such as "关于我们" ("About us"), while its two-word English equivalent is still dropped. Character-level counting matches how final_response_match_v1 tokenizes these scripts.

Testing Plan

Unit Tests:

• I have added or updated unit tests for my change.
• All unit tests pass locally.

Added test_load_web_page_keeps_lines_of_scripts_written_without_spaces (Chinese and Japanese pages; both fail on main with an empty result) and test_count_words_counts_each_letter_of_an_unspaced_script.

$ pytest tests/unittests/tools/test_load_web_page.py
38 passed

Manual End-to-End (E2E) Tests:

Ran the reproduction script from #7956, which feeds the same page in three languages through load_web_page:

# before (main)
--- English: 136 characters returned
--- Chinese: 0 characters returned
--- Japanese: 0 characters returned

# after
--- English: 136 characters returned
--- Chinese: 60 characters returned
--- Japanese: 65 characters returned

Checklist

• I have read the CONTRIBUTING.md document.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes.
• I have manually tested my changes end-to-end.
• Any dependent changes have been merged and published in downstream modules. (N/A: no dependent changes)

Additional context

I used Claude Code to help find the issue and draft the change; I reproduced the bug and verified the fix and tests locally.

[mangeshraut712]

Very thoughtful fix for #7956.

A few positive observations from inspecting the logic:

1. Fast path: tokens without unspaced scripts take the early if not any(...) return, preserving identical behavior and performance for whitespace-delimited languages.
2. Mixed token boundary handling: in strings like 2024款Model Y, the Latin/digit runs are correctly separated from the ideographs, and purely punctuation runs (such as 。) do not trigger word increments because of any(c.isalnum() for c in run).
3. Script scope: keeping Korean (Hangul) on whitespace-delimited counting while including Thai, Lao, Myanmar, Khmer, Kana, and CJK ideographs matches real-world typographic word boundary conventions.
4. The test coverage validates both pure and mixed text lines against the > 3 threshold.

LGTM.