Repository navigation
make canonical ordering linear in the length of a combining-mark run - #376
Open
shamsManna wants to merge 1 commit into
Open
shamsManna wants to merge 1 commit into
shamsManna wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
the canonical-ordering pass at the end of utf8proc_decompose_custom bubbles each out-of-order combining mark back one slot at a time, so a run of marks in descending combining-class order costs a swap plus two property lookups per inversion, i.e. quadratic in the run length. that makes normalization an algorithmic-complexity DoS on untrusted input: every NFC/NFD/NFKC/NFKD call goes through this pass, and the same bytes cost nothing when the marks happen to be in ascending order. the numbers below are utf8proc_map with STABLE|DECOMPOSE on a single 'a' followed by U+0300/U+0323/U+0334 (classes 230, 220, 1) in descending blocks, explicitly sized input, -O2:
the pass now finds each run of non-starters (delimited by starters and the CHARBOUND sentinels) and sorts it stably by combining class: insertion sort in place for runs of up to 32 marks, a counting sort through a malloc'd copy of the run above that, falling back to the insertion sort if malloc fails. that is the same stable sort the bubble pass computed, so output is unchanged: 180k utf8proc_map calls over random mark/starter/sentinel mixes with DECOMPOSE, COMPOSE, CASEFOLD, COMPAT, CHARBOUND and STRIPCC give byte-identical output against master, and make check / ctest pass. the 64 KiB cap in test/fuzzer.c ("avoid timeout with long inputs") is this pass. the new misc test case covers both sort paths and checks stability within a class; it also passes on master, where the 65536-mark case takes about 6 s instead of a few ms.