Skip to content

[Bug]: str(b'', encoding) raises LookupError for an unknown encoding instead of returning '' #1223

Closed

Description

@jseop-lim

Describe the bug

str() with an encoding or errors argument looks up the codec even when the input is empty, so str(b'', 'x') raises LookupError: unknown encoding x. A CPython release build returns '' for empty input without looking up the codec.

Operating system

Linux

CPU architecture

ARM64

GraalPy version

25.0.2 (Python 3.12.8); also reproduced on 25.4.4.1.1 (Python 3.13.14)

JDK version

No response

Context configuration

No response

Steps to reproduce

print(repr(str(b'', 'x')))

GraalPy 25.0.2:

LookupError: unknown encoding x

Expected behavior

CPython 3.12.13 (same on 3.13.16):

''

Stack trace

Additional context

str(b'', 'x', 'y'), str(bytearray(), 'x'), str(memoryview(b''), 'x'), b''.decode('x') and bytearray().decode('x') diverge the same way. Non-empty input agrees: str(b'a', 'x') raises LookupError on both.

CPython raises for the empty cases too under -X dev. Outside dev mode, devmode.rst#L80-L82 documents that "the encoding argument is sometimes ignored for empty strings".

Root Cause

StrNewNode.doBuffer (StringBuiltins.java#L278-L314) copies the buffer into a new PBytes and hands it to BytesCommonBuiltins.DecodeNode (#L302-L304), which looks up the codec without checking the length (BytesCommonBuiltins.java#L203-L214). GraalPy's own PyUnicodeFromEncodedObject already returns '' for empty input (#L84-L87, #L116-L119), but str() does not go through it. CPython's unicode_new_impl calls PyUnicode_FromEncodedObject (unicodeobject.c#L14723), which returns '' for an empty bytes or buffer (#L3311-L3317, #L3337-L3343) and validates encoding and errors only in dev mode or a debug build (#L584-L588):

// Objects/unicodeobject.c, PyUnicode_FromEncodedObject (v3.12.13, line 3310)
/* Decoding bytes objects is the most common case and should be fast */
if (PyBytes_Check(obj)) {
    if (PyBytes_GET_SIZE(obj) == 0) {
        if (unicode_check_encoding_errors(encoding, errors) < 0) {
            return NULL;
        }
        _Py_RETURN_UNICODE_EMPTY();
    }

Fix Suggestion

Return the empty string when bufferLib.getBufferLength(buffer) == 0 in doBuffer and in doNativeSubclassEncodeErr (#L339-L374), as PyUnicodeFromEncodedObject.doBuffer does. Putting the check in BytesCommonBuiltins.DecodeNode.decode instead also covers bytes.decode and bytearray.decode.

Activity

  1. msimacek commented on Oct 19, 2026

    @msimacek
    Contributor

    Does this cause any real problems or you're just fuzzing? I'd argue that GraalPy's behavior makes more sense here, especially if CPython's dev build has the same behavior.

  2. jseop-lim commented on Oct 19, 2026

    @jseop-lim
    ContributorAuthor

    Found by differential fuzzing against CPython, not from a user report. The closest real-world case I found is the stdlib email package, where get_text_content calls content.decode(charset):

    import email, email.policy
    msg = email.message_from_string("Content-Type: text/plain; charset=bogus\n\n", policy=email.policy.default)
    print(repr(msg.get_content()))

    CPython 3.12 prints '', GraalPy raises LookupError: unknown encoding bogus. A non-empty body raises on both.

    GraalPy's own PyUnicode_FromEncodedObject already returns '' for empty input, so C extensions and Python code disagree today. I have a small tested fix in BytesCommonBuiltins.DecodeNode if you want it, but I'm fine with closing this as intended behavior.

  3. msimacek commented on Oct 19, 2026

    @msimacek
    Contributor

    Feel free to open a PR with the small fix. If it's not too hacky and it doesn't look like it will regress on performance then I'd be fine with merging it.

  4. jseop-lim commented on Oct 20, 2026

    @jseop-lim
    ContributorAuthor

    Opened #1227. It's a single length check in the shared decode node, and I measured no performance difference on non-empty input.

  5. msimacek commented on Oct 23, 2026

    @msimacek
    Contributor

    Thank you, I merged all the PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions