Repository navigation
normalize_utf32: keep grapheme-break sentinels when stripping control characters - #371
Merged
Merged
Conversation
… characters With UTF8PROC_STRIPCC, everything below 0x20 was treated as a control character, including the negative sentinels that UTF8PROC_CHARBOUND inserts before each grapheme. So CHARBOUND|STRIPCC returned the text without any 0xff boundary markers. Only strip non-negative codepoints, and when a control character is removed, drop the sentinel in front of it as well so that it doesn't leave an empty grapheme behind.
eschnett
reviewed
Oct 9, 2026
eschnett
reviewed
Oct 9, 2026
eschnett
requested changes
Oct 9, 2026
eschnett
reviewed
Oct 9, 2026
eschnett
approved these changes
Oct 11, 2026
stevengj
reviewed
Oct 11, 2026
stevengj
approved these changes
Oct 12, 2026
Member
|
Thanks! |
DomingoGP
added a commit
to DomingoGP/utf8proc_pascal
that referenced
this pull request
Oct 17, 2026
… control characters (JuliaStrings#371)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
UTF8PROC_CHARBOUND and UTF8PROC_STRIPCC don't work together: the STRIPCC pass in utf8proc_normalize_utf32() treats every value below 0x20 as a control character, and that includes the negative sentinels CHARBOUND puts before each grapheme. All the boundary markers get dropped:
This is the same kind of problem #355 fixed in the compose loop.
The fix only strips non-negative codepoints. When a control character is removed, it also drops the sentinel right before it, so the output doesn't get an empty grapheme (two 0xff bytes in a row). That works because a control character is always a grapheme of its own; CR LF is handled earlier as an NLF sequence. With the patch the second call gives "\xffa\xffb\xff \xffc\xffd".
Added a regression test to test/graphemetest.c next to the issue 356 one. It fails on master ("returned length 4, expected 8") and passes with the change. make check passes.