Repository navigation
Conversation
BytesCommonBuiltins.DecodeNode, which backs bytes.decode,
bytearray.decode and str(obj, encoding), always went through the
codec, so b''.decode('bogus') and str(b'', 'bogus') raised
LookupError.
CPython's bytes and bytearray decode call PyUnicode_FromEncodedObject,
which returns '' for empty input without looking up the codec outside
dev mode. GraalPy's own PyUnicodeFromEncodedObject already does this
for the C API; DecodeNode now has the same shortcut.
Signed-off-by: Jeongseop Lim <jeongseop_lim@korea.ac.kr>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #1223
BytesCommonBuiltins.DecodeNode, which backsbytes.decode,bytearray.decodeandstr(obj, encoding), always looked up the codec, so an empty input with an unknown encoding raisedLookupError. CPython'sbytes_decodeandbytearray_decodego throughPyUnicode_FromEncodedObject, which returns''for empty input without a codec lookup outside dev mode (unicodeobject.c#L3311-L3317). GraalPy's ownPyUnicodeFromEncodedObjectalready has the same shortcut (PyUnicodeFromEncodedObject.java#L85-L87), so C extensions and Python code disagreed.AS-IS
TO-BE
(matching CPython)
Changes
T_EMPTY_STRINGfromDecodeNode.decodewhen the bytes storage is empty, behind anInlinedConditionProfile, asPyUnicodeFromEncodedObject.doBytesdoes.test_decode_empty_skips_codec_lookuptotests/test_bytes.py.Testing
mx python-jvmon linux-aarch64, thenmx graalpytest test_bytes.py test_codecs.py test_string.py(237 passed).test.test_codecs,test.test_strandtest.test_bytesfromlib-pythongive the same per-test results as on master (on thejvm-cebuilds, whereBytesAsStringTest.test_adaptive_findalone takes about 40 minutes on both). The new test fails on the unpatched build. Non-empty input still raisesLookupErrorfor an unknown encoding, and a non-buffer still raisesTypeError.For performance, I built the
jvm-cestandalone (with the compiler) from master and from this branch and timed 1M-call loops after warmup, alternating the two builds over five runs. On a 52-byte input,bytes.decode('utf-8'),bytes.decode('latin-1'),str(b, 'utf-8')andbytearray.decode('utf-8')stay within 3 ns of master at 30 to 45 ns per call, which is inside the run-to-run spread.b''.decode('utf-8')drops from 31 ns to effectively zero.