Skip to content

make canonical ordering linear in the length of a combining-mark run - #376

Open
shamsManna wants to merge 1 commit into
JuliaStrings:masterfrom
shamsManna:canonical-order-linear
Open

shamsManna wants to merge 1 commit into
JuliaStrings:masterfrom
shamsManna:canonical-order-linear

Conversation

@shamsManna

Copy link
Copy Markdown
Contributor

the canonical-ordering pass at the end of utf8proc_decompose_custom bubbles each out-of-order combining mark back one slot at a time, so a run of marks in descending combining-class order costs a swap plus two property lookups per inversion, i.e. quadratic in the run length. that makes normalization an algorithmic-complexity DoS on untrusted input: every NFC/NFD/NFKC/NFKD call goes through this pass, and the same bytes cost nothing when the marks happen to be in ascending order. the numbers below are utf8proc_map with STABLE|DECOMPOSE on a single 'a' followed by U+0300/U+0323/U+0334 (classes 230, 220, 1) in descending blocks, explicitly sized input, -O2:

marks     bytes      before     after
32768     64 KiB     2.23 s     0.001 s
100000    195 KiB    16.4 s     0.002 s
1000000   1.9 MiB    ~27 min*   0.026 s
100000    ascending  0.002 s    0.002 s

* extrapolated from the quadratic fit, not run to completion

the pass now finds each run of non-starters (delimited by starters and the CHARBOUND sentinels) and sorts it stably by combining class: insertion sort in place for runs of up to 32 marks, a counting sort through a malloc'd copy of the run above that, falling back to the insertion sort if malloc fails. that is the same stable sort the bubble pass computed, so output is unchanged: 180k utf8proc_map calls over random mark/starter/sentinel mixes with DECOMPOSE, COMPOSE, CASEFOLD, COMPAT, CHARBOUND and STRIPCC give byte-identical output against master, and make check / ctest pass. the 64 KiB cap in test/fuzzer.c ("avoid timeout with long inputs") is this pass. the new misc test case covers both sort paths and checks stability within a class; it also passes on master, where the 65536-mark case takes about 6 s instead of a few ms.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant