Major performance degradation introduced with 10.3.3 (UTF8 char fetch) #1272

Activity

1. fdcastel commented on May 23, 2026

   

   fdcastel

   on May 23, 2026

   Member

   More actions

   Thanks @pl752 for the detailed report. I dug into this and can confirm both the regression and that your PR #1252 fixes it correctly. Here is what I found, with the supporting measurements.

   Root cause

   Commit bd0db88 (the fix for #1213) replaced the zero-allocation RuneCount() / Substring truncation with EnumerateRunesToChars(), which allocates one char[] per code point and is then materialized with .ToList() on the read hot path. This runs on every CHAR(n) fetch — even when no truncation is needed, which is the common case. Only DbDataType.Char goes through this path (VarChar returns the string directly), which is why a VARCHAR-based benchmark doesn't surface it: the regression only shows with CHAR(n) CHARACTER SET UTF8.

   The affected read sites are GdsStatement.ReadRawValue / ReadRawValueAsync and DbField.SetValue; the equivalent EnumerateRunesToChars().Count() is also used on the write/validate paths in DbValue and GdsStatement.WriteRawParameter* (10 call sites in total, all covered by your PR).

   Proof 1 — synthetic micro-benchmark (no DB, .NET 10)

   Isolating just the truncation/count operation per field value:

   Scenario	old (EnumerateRunesToChars)	new (PR #1252)
   CHAR(100), value + space padding, no truncation (the hot path)	691.7 ns / 4176 B	1.9 ns / 0 B
   120 chars → truncate to 100	1464 ns / 5536 B	15 ns / 224 B
   80 emoji → truncate to 60	1011 ns / 4016 B	69 ns / 264 B
   count-only (write/validate path)	332 ns / 3320 B	4.6 ns / 0 B

   Even when no truncation happens, the old code allocates ~4 KB per value while the new code allocates nothing.

   Proof 2 — end-to-end against a live Firebird 3.0 server (.NET 10)

   I built two trees, current master vs master + PR #1252, and ran the fetch benchmarks against the same server. Running your LargeFetchBenchmark (CHAR(255) CHARACTER SET UTF8, 100,000 rows):

   	current master	+ PR #1252
   Mean	1,445.6 ms	1,007 ms
   Allocated	4339.37 MB	369.8 MB

   That reproduces your exact figures and the 11.73× allocation ratio. A smaller CHAR(100) UTF8 / 100-row fetch shows the same shape: 1.76 MB → 192 KB (~9.4× less).

   Proof 3 — correctness (does not reintroduce #1213)

   • CountRunes and TruncateStringToRuneCount match the string.EnumerateRunes() reference across all edge cases plus a 200,000-string fuzz (0 mismatches), including lone/unpaired surrogates.
   • The Improper truncation when reading from UTF-8 CHAR(n) fields containing characters outside of the Basic Multilingual Plane #1213 scenarios behave correctly: a CHAR(1) holding 😊 returns the full surrogate pair (one rune), not a truncated high surrogate; a CHAR(2) holding 😊! returns 😊!.
   • The existing server tests HighLowSurrogatePassingTest and HighLowSurrogateReadingTest pass with the PR applied.
   • The PR compiles cleanly on current master. Note that Extensions.cs on master now uses the new extension(...) member syntax; the PR's classic this ReadOnlySpan<char> methods coexist with it without issue.

   The only behavioral difference I could find between old and new is on invalid UTF-16 (a lone surrogate) and only when truncation actually occurs: the old code normalizes it to U+FFFD while the new code preserves the original char. That is not reachable for valid UTF8 data and is unrelated to #1213.

   Verdict

   PR #1252 is correct and restores the pre-regression allocation behavior (~11.7× less for the UTF8 CHAR case) without reintroducing #1213. Two minor, non-blocking cleanups: after the PR, EnumerateRunesToChars is no longer used and could be removed; and the two new methods could be moved into an extension(...) block to match the current style of Extensions.cs.

   I'm happy to share the benchmark and correctness harness, and the exact patch I used, if that would be helpful.
2. fdcastel

   added a commit that references this issue on May 23, 2026

   Add CHAR UTF8 coverage to CommandBenchmark and document regression de…

   ...

   46db5a1
3. fdcastel commented on May 23, 2026

   

   fdcastel

   on May 23, 2026

   Last edited by fdcastel

   Member

   More actions

   @pl752 I’ve updated #1203 if you’d like to take a look. Feedback and suggestions are very welcome.
4. pl752 commented on May 24, 2026

   

   pl752

   on May 24, 2026

   ContributorAuthor

   More actions

   @fdcastel Added char utf-8 coverage, of course, reproduces the issue successfully too