Skip to content

JIT (interpreter): running hot code with PYTHON_LLTRACE=4 segfaults #139193

Closed
Closed
JIT (interpreter): running hot code with PYTHON_LLTRACE=4 segfaults#139193
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-JITtype-crashA hard crash of the interpreter, possibly with a core dump

Description

@devdanzin
devdanzin
opened on Sep 20, 2025

Crash report

What happened?

A debug JIT build will segfault when running the following MRE under PYTHON_LLTRACE=4:

def f1():
    pass

for x in range(4097):
    f1()

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x00005555558461ac in dump_item (item=...) at Python/ceval.c:173
173         printf("<%s at %p>", Py_TYPE(obj)->tp_name, (void *)obj);
(gdb) bt
#0  0x00005555558461ac in dump_item (item=...) at Python/ceval.c:173
#1  0x00005555558469df in dump_stack (frame=frame@entry=0x7ffff7fb0020, stack_pointer=stack_pointer@entry=0x7ffff7fb0088) at Python/ceval.c:200
#2  0x0000555555875e1b in _PyTier2Interpreter (current_executor=0x555555eb76d0, frame=0x7ffff7fb0020, stack_pointer=<optimized out>,
    tstate=0x555555d19fd8 <_PyRuntime+333720>) at Python/ceval.c:1190
#3  0x00005555558670d6 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555d19fd8 <_PyRuntime+333720>, frame=frame@entry=0x7ffff7fb0020,
    throwflag=throwflag@entry=0) at Python/generated_cases.c.h:7670
#4  0x0000555555875638 in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb0020, tstate=0x555555d19fd8 <_PyRuntime+333720>)
    at ./Include/internal/pycore_ceval.h:121
#5  _PyEval_Vector (tstate=tstate@entry=0x555555d19fd8 <_PyRuntime+333720>, func=func@entry=0x7ffff7a4a8d0, locals=locals@entry=0x7ffff7a47650,
    args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:1990
#6  0x0000555555875737 in PyEval_EvalCode (co=co@entry=0x7ffff7a40820, globals=globals@entry=0x7ffff7a47650, locals=locals@entry=0x7ffff7a47650)
    at Python/ceval.c:873
#7  0x000055555594923b in run_eval_code_obj (tstate=tstate@entry=0x555555d19fd8 <_PyRuntime+333720>, co=co@entry=0x7ffff7a40820,
    globals=globals@entry=0x7ffff7a47650, locals=locals@entry=0x7ffff7a47650) at Python/pythonrun.c:1365
#8  0x000055555594a453 in run_mod (mod=mod@entry=0x555555eb94c0, filename=filename@entry=0x7ffff7a96c40, globals=globals@entry=0x7ffff7a47650,
    locals=locals@entry=0x7ffff7a47650, flags=flags@entry=0x7fffffffdb08, arena=arena@entry=0x7ffff7ab7ca0, interactive_src=0x0,
    generate_new_source=0) at Python/pythonrun.c:1459
#9  0x000055555594a800 in pyrun_file (fp=fp@entry=0x555555d8f520, filename=filename@entry=0x7ffff7a96c40, start=start@entry=257,
    globals=globals@entry=0x7ffff7a47650, locals=locals@entry=0x7ffff7a47650, closeit=closeit@entry=1, flags=0x7fffffffdb08)
    at Python/pythonrun.c:1293
#10 0x000055555594cb16 in _PyRun_SimpleFileObject (fp=fp@entry=0x555555d8f520, filename=filename@entry=0x7ffff7a96c40, closeit=closeit@entry=1,
    flags=flags@entry=0x7fffffffdb08) at Python/pythonrun.c:521

Output running with PYTHON_LLTRACE=4:

Optimizing <module> (/mnt/c/Users/ddini/crashers/jit/crashy_interpreter.py:1) at byte offset 44
   1 ADD_TO_TRACE: _START_EXECUTOR (0, target=22, operand0=0x7fb4ed75491c, operand1=0)
   2 ADD_TO_TRACE: _MAKE_WARM (0, target=0, operand0=0, operand1=0)
22: JUMP_BACKWARD_JIT(12)
   3 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=22, operand0=0, operand1=0)
   4 ADD_TO_TRACE: _SET_IP (0, target=22, operand0=0x7fb4ed75491c, operand1=0)
   5 ADD_TO_TRACE: _CHECK_PERIODIC (0, target=22, operand0=0, operand1=0, error_target=0)
12: FOR_ITER_RANGE(10)
   6 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=12, operand0=0, operand1=0)
   7 ADD_TO_TRACE: _SET_IP (0, target=12, operand0=0x7fb4ed754908, operand1=0)
   8 ADD_TO_TRACE: _ITER_CHECK_RANGE (10, target=12, operand0=0, operand1=0)
   9 ADD_TO_TRACE: _GUARD_NOT_EXHAUSTED_RANGE (10, target=12, operand0=0, operand1=0)
  10 ADD_TO_TRACE: _ITER_NEXT_RANGE (10, target=12, operand0=0, operand1=0, error_target=0)
14: STORE_NAME(2)
  11 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=14, operand0=0, operand1=0)
  12 ADD_TO_TRACE: _SET_IP (0, target=14, operand0=0x7fb4ed75490c, operand1=0)
  13 ADD_TO_TRACE: _STORE_NAME (2, target=14, operand0=0, operand1=0, error_target=0)
15: LOAD_NAME(0)
  14 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=15, operand0=0, operand1=0)
  15 ADD_TO_TRACE: _SET_IP (0, target=15, operand0=0x7fb4ed75490e, operand1=0)
  16 ADD_TO_TRACE: _LOAD_NAME (0, target=15, operand0=0, operand1=0, error_target=0)
16: PUSH_NULL(0)
  17 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=16, operand0=0, operand1=0)
  18 ADD_TO_TRACE: _SET_IP (0, target=16, operand0=0x7fb4ed754910, operand1=0)
  19 ADD_TO_TRACE: _PUSH_NULL (0, target=16, operand0=0, operand1=0)
17: CALL_PY_EXACT_ARGS(0)
  20 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=17, operand0=0, operand1=0)
  21 ADD_TO_TRACE: _SET_IP (0, target=17, operand0=0x7fb4ed754912, operand1=0)
  22 ADD_TO_TRACE: _CHECK_PEP_523 (0, target=17, operand0=0, operand1=0)
  23 ADD_TO_TRACE: _CHECK_FUNCTION_VERSION (0, target=17, operand0=0x77e, operand1=0)
  24 ADD_TO_TRACE: _CHECK_FUNCTION_EXACT_ARGS (0, target=17, operand0=0, operand1=0)
  25 ADD_TO_TRACE: _CHECK_STACK_SPACE (0, target=17, operand0=0, operand1=0)
  26 ADD_TO_TRACE: _CHECK_RECURSION_REMAINING (0, target=17, operand0=0, operand1=0)
  27 ADD_TO_TRACE: _INIT_CALL_PY_EXACT_ARGS (0, target=17, operand0=0, operand1=0)
  28 ADD_TO_TRACE: _SAVE_RETURN_OFFSET (4, target=17, operand0=0, operand1=0)
Function: version=0x77e; new_func=0x7fb4ed75ea50, new_code=0x7fb4ed774a90
  29 ADD_TO_TRACE: _PUSH_FRAME (0, target=17, operand0=0x7fb4ed75ea50, operand1=0)
Continuing in f1 (/mnt/c/Users/ddini/crashers/jit/crashy_interpreter.py:1) at byte offset 0
0: RESUME_CHECK(0)
  30 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=0, operand0=0, operand1=0)
  31 ADD_TO_TRACE: _SET_IP (0, target=0, operand0=0x7fb4ed774b60, operand1=0)
  32 ADD_TO_TRACE: _RESUME_CHECK (0, target=0, operand0=0, operand1=0)
1: LOAD_CONST(0)
  33 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=1, operand0=0, operand1=0)
  34 ADD_TO_TRACE: _SET_IP (0, target=1, operand0=0x7fb4ed774b62, operand1=0)
  35 ADD_TO_TRACE: _LOAD_CONST (0, target=1, operand0=0, operand1=0)
2: RETURN_VALUE(0)
  36 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=2, operand0=0, operand1=0)
  37 ADD_TO_TRACE: _SET_IP (0, target=2, operand0=0x7fb4ed774b64, operand1=0)
  38 ADD_TO_TRACE: _RETURN_VALUE (0, target=2, operand0=0x7fb4ed75e8d0, operand1=0)
Returning to <module> (/mnt/c/Users/ddini/crashers/jit/crashy_interpreter.py:1) at byte offset 42
21: POP_TOP(0)
  39 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=21, operand0=0, operand1=0)
  40 ADD_TO_TRACE: _SET_IP (0, target=21, operand0=0x7fb4ed75491a, operand1=0)
  41 ADD_TO_TRACE: _POP_TOP (0, target=21, operand0=0, operand1=0)
  42 ADD_TO_TRACE: _JUMP_TO_TOP (0, target=0, operand0=0, operand1=0)
Created a proto-trace for <module> (/mnt/c/Users/ddini/crashers/jit/crashy_interpreter.py:1) at byte offset 44 -- length 42
Optimized trace (length 44):
   0 OPTIMIZED: _START_EXECUTOR (0, jump_target=30, operand0=0x5586f6f905b0, operand1=0)
   1 OPTIMIZED: _MAKE_WARM (0, target=0, operand0=0, operand1=0)
   2 OPTIMIZED: _SET_IP (0, target=22, operand0=0x7fb4ed75491c, operand1=0)
   3 OPTIMIZED: _CHECK_PERIODIC (0, jump_target=0, operand0=0, operand1=0, error_target=31)
   4 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=32, operand0=0, operand1=0)
   5 OPTIMIZED: _ITER_CHECK_RANGE (10, jump_target=33, operand0=0, operand1=0)
   6 OPTIMIZED: _GUARD_NOT_EXHAUSTED_RANGE (10, jump_target=34, operand0=0, operand1=0)
   7 OPTIMIZED: _ITER_NEXT_RANGE (10, jump_target=0, operand0=0, operand1=0, error_target=35)
   8 OPTIMIZED: _SET_IP (0, target=14, operand0=0x7fb4ed75490c, operand1=0)
   9 OPTIMIZED: _STORE_NAME (2, jump_target=0, operand0=0, operand1=0, error_target=36)
  10 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=37, operand0=0, operand1=0)
  11 OPTIMIZED: _SET_IP (0, target=15, operand0=0x7fb4ed75490e, operand1=0)
  12 OPTIMIZED: _LOAD_NAME (0, jump_target=0, operand0=0, operand1=0, error_target=38)
  13 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=39, operand0=0, operand1=0)
  14 OPTIMIZED: _PUSH_NULL (0, target=16, operand0=0, operand1=0)
  15 OPTIMIZED: _SET_IP (0, target=17, operand0=0x7fb4ed754912, operand1=0)
  16 OPTIMIZED: _CHECK_FUNCTION_VERSION (0, jump_target=40, operand0=0x77e, operand1=0)
  17 OPTIMIZED: _CHECK_FUNCTION_EXACT_ARGS (0, jump_target=40, operand0=0, operand1=0)
  18 OPTIMIZED: _CHECK_STACK_SPACE_OPERAND (0, jump_target=41, operand0=0xb, operand1=0)
  19 OPTIMIZED: _CHECK_RECURSION_REMAINING (0, jump_target=41, operand0=0, operand1=0)
  20 OPTIMIZED: _INIT_CALL_PY_EXACT_ARGS_0 (0, target=17, operand0=0, operand1=0)
  21 OPTIMIZED: _SAVE_RETURN_OFFSET (4, target=17, operand0=0, operand1=0)
  22 OPTIMIZED: _PUSH_FRAME (0, target=17, operand0=0x7fb4ed75ea50, operand1=0)
  23 OPTIMIZED: _RESUME_CHECK (0, jump_target=42, operand0=0, operand1=0)
  24 OPTIMIZED: _LOAD_CONST_INLINE_BORROW (0, target=1, operand0=0x5586b82b47c0, operand1=0)
  25 OPTIMIZED: _SET_IP (0, target=2, operand0=0x7fb4ed774b64, operand1=0)
  26 OPTIMIZED: _RETURN_VALUE (0, target=2, operand0=0x7fb4ed75e8d0, operand1=0)
  27 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=43, operand0=0, operand1=0)
  28 OPTIMIZED: _POP_TOP_NOP (0, target=21, operand0=0, operand1=0)
  29 OPTIMIZED: _JUMP_TO_TOP (0, jump_target=1, operand0=0, operand1=0)
  30 OPTIMIZED: _DEOPT (0, target=22, operand0=0, operand1=0)
  31 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0x16, operand1=0)
  32 OPTIMIZED: _DEOPT (0, target=12, operand0=0, operand1=0)
  33 OPTIMIZED: _EXIT_TRACE (0, target=12, operand0=0x5586f6f90628, operand1=0)
  34 OPTIMIZED: _EXIT_TRACE (0, target=25, operand0=0x5586f6f90638, operand1=0)
  35 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0xc, operand1=0)
  36 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0xe, operand1=0)
  37 OPTIMIZED: _DEOPT (0, target=15, operand0=0, operand1=0)
  38 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0xf, operand1=0)
  39 OPTIMIZED: _DEOPT (0, target=16, operand0=0, operand1=0)
  40 OPTIMIZED: _EXIT_TRACE (0, target=17, operand0=0x5586f6f90648, operand1=0)
  41 OPTIMIZED: _DEOPT (0, target=17, operand0=0, operand1=0)
  42 OPTIMIZED: _DEOPT (0, target=0, operand0=0, operand1=0)
  43 OPTIMIZED: _DEOPT (0, target=21, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
   0 uop: _START_EXECUTOR (0, jump_target=30, operand0=0x5586f6f905b0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
   1 uop: _MAKE_WARM (0, target=0, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
   2 uop: _SET_IP (0, target=22, operand0=0x7fb4ed75491c, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
   3 uop: _CHECK_PERIODIC (0, jump_target=0, operand0=0, operand1=0, error_target=31)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
   4 uop: _CHECK_VALIDITY (0, jump_target=32, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
   5 uop: _ITER_CHECK_RANGE (10, jump_target=33, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
   6 uop: _GUARD_NOT_EXHAUSTED_RANGE (10, jump_target=34, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
   7 uop: _ITER_NEXT_RANGE (10, jump_target=0, operand0=0, operand1=0, error_target=35)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <int at 0x7fb4ed753280>]
   8 uop: _SET_IP (0, target=14, operand0=0x7fb4ed75490c, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <int at 0x7fb4ed753280>]
   9 uop: _STORE_NAME (2, jump_target=0, operand0=0, operand1=0, error_target=36)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
  10 uop: _CHECK_VALIDITY (0, jump_target=37, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
  11 uop: _SET_IP (0, target=15, operand0=0x7fb4ed75490e, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>]
  12 uop: _LOAD_NAME (0, jump_target=0, operand0=0, operand1=0, error_target=38)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <function at 0x7fb4ed75ea50>]
  13 uop: _CHECK_VALIDITY (0, jump_target=39, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <function at 0x7fb4ed75ea50>]
  14 uop: _PUSH_NULL (0, target=16, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <function at 0x7fb4ed75ea50>, <NULL>]
  15 uop: _SET_IP (0, target=17, operand0=0x7fb4ed754912, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <function at 0x7fb4ed75ea50>, <NULL>]
  16 uop: _CHECK_FUNCTION_VERSION (0, jump_target=40, operand0=0x77e, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <function at 0x7fb4ed75ea50>, <NULL>]
  17 uop: _CHECK_FUNCTION_EXACT_ARGS (0, jump_target=40, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <function at 0x7fb4ed75ea50>, <NULL>]
  18 uop: _CHECK_STACK_SPACE_OPERAND (0, jump_target=41, operand0=0xb, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <function at 0x7fb4ed75ea50>, <NULL>]
  19 uop: _CHECK_RECURSION_REMAINING (0, jump_target=41, operand0=0, operand1=0)
    locals=[]
    stack=[<range_iterator at 0x7fb4ed7517c0>, <NULL>, <function at 0x7fb4ed75ea50>, <NULL>]
  20 uop: _INIT_CALL_PY_EXACT_ARGS_0 (0, target=17, operand0=0, operand1=0)
    locals=[]
Segmentation fault (core dumped)

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.15.0a0 (heads/main:69c6b438e84, Sep 20 2025, 17:42:17) [GCC 11.4.0]

Linked PRs

Activity

  1. devdanzin
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    type-crashA hard crash of the interpreter, possibly with a core dump
    topic-JIT
    on Sep 20, 2025

  2. Fidget-Spinner commented on Sep 21, 2025

    @Fidget-Spinner
    Fidget-Spinner
    on Sep 21, 2025
    Member

    Yeah this is a known issue. The problem is that everything between _INIT_CALL_PY_EXACT_ARGS and _PUSH_FRAME pushes a non-PyObject to the stack (they push a frame struct directly). dump_stack then tries to read that and segfaults.

  3. mowmanminer commented on Sep 27, 2025

    @mowmanminer
    mowmanminer
    on Sep 27, 2025 · Hidden as spam

  4. sergey-miryanov commented on Sep 27, 2025

    @sergey-miryanov
    sergey-miryanov
    on Sep 27, 2025
    Contributor

    There are two problems:

    1. dump_stack/dump_item should ignore items that pushed on the stack with PyStackRef_Wrap
    2. PyStackRef_IsError should test only Py_TAG_BITS of .bits not whole .bits value

    Also, PyStackRef_IsValid should be fixed accordingly but GIL-enabled version doesn't have PyStackRef_IsNULL.

    I have made a PR, please take a look.

  5. markshannon
    added a commit that references this issue on Oct 22, 2025

    GH-139193: Fix dump_stack when PYTHON_LLTRACE=4 (GH-139384)

    a4709e5

  6. sergey-miryanov commented on Oct 22, 2025

    @sergey-miryanov
    sergey-miryanov
    on Oct 22, 2025
    Contributor

    @devdanzin The fix has been merged to main. I believe we can close this now.

  7. devdanzin
    closed this as completedon Oct 22, 2025
  8. StanFromIreland
    added a commit that references this issue on Dec 6, 2025

    pythonGH-139193: Fix dump_stack when PYTHON_LLTRACE=4 (pythonGH-139384)

    a5a5687
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-JITtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions