Skip to content

profiling.sampling incorrectly handles base_frame and native frame as the first frame #157605

Closed
Closed
profiling.sampling incorrectly handles base_frame and native frame as the first frame#157605
Labels
extension-modulesC modules in the Modules dirtopic-profilingtype-bugAn unexpected behavior, bug, or error

Description

@liuzhijie-0614
liuzhijie-0614
opened on Sep 16, 2026

Bug description

profiling.sampling does not correctly handle some frame chains whose first frame is not a Python frame.

There are two cases:

  1. The first frame is base_frame

    A native thread can retain a Python thread state while waiting in C code, with no Python functions executing. Its frame chain contains only the base_frame sentinel.

    The sampler skips this sentinel and then raises:

    RuntimeError: Failed to parse initial frame in chain

  2. The first frame is a native frame

    When the first frame in the chain is a native frame, the sampler does not correctly handle it as the initial frame and may report the frame chain as invalid instead of adding a <native> frame.

In both cases, the failure can cause the entire sampling call to fail, so valid Python stacks from other threads are also lost.

Reproduction

Case 1: base_frame as the first frame

Create a pthread that registers and detaches a Python thread state:

PyGILState_STATE state = PyGILState_Ensure();
PyThreadState *saved = PyEval_SaveThread();

/* Wait on a pthread condition variable while the parent samples. */

/* Cleanup after sampling. */
PyEval_RestoreThread(saved);
PyGILState_Release(state);

Keep the main thread inside a Python function and sample the child process from its parent:

unwinder = _remote_debugging.RemoteUnwinder(
    pid, all_threads=True, cache_frames=False,
)
unwinder.get_stack_trace()

tachyon-empty-native-repro.zip

  • Reproduces with frame caching both enabled and disabled.

Case 2: Native frame as the first frame

Construct a frame chain whose current frame is a native frame and whose previous frame leads to base_frame.

When native-frame collection is enabled, sampling this thread currently fails to correctly handle the native frame as the first frame.

Expected behavior

The sampler should correctly handle both cases:

  1. If the first frame is base_frame, treat it as a valid empty Python stack.
  2. If the first frame is a native frame, add a <native> frame to the returned stack and continue walking the frame chain.

A special frame state in one thread should not cause the entire sampling operation to fail or prevent valid stacks from other threads from being returned.

Suspected cause

process_frame_chain() does not correctly handle cases where the first frame does not contain a parseable Python frame, including base_frame and native frames.

These cases can be incorrectly treated as a broken initial frame chain.

Environment

Linux, CPython 3.15.0rc2+dev free-threaded build and current main.

Linked PRs

Activity

  1. aisk
    added
    type-bugAn unexpected behavior, bug, or error
    extension-modulesC modules in the Modules dir
    topic-profiling
    on Sep 16, 2026
  2. aidaodedjl
    added 5 commits that reference this issue on Sep 17, 2026

    pythongh-157605: Fix regression test include dirs for out-of-tree builds

    1a923f1

    pythongh-157605: Handle a native first frame like a mid-chain native …

    c39b4bc

    pythongh-157605: Cover native-first frames with a regression test

    ddcd747

    pythongh-157605: Discriminate code-less frames by owner field

    c3a0016

    pythongh-157605: Skip dropped frames and keep walking the live caller…

    2c58f57
  3. liuzhijie-0614
    changed the title [-]profiling.sampling fails when a native thread has an empty Python stack[/-] [+]profiling.sampling incorrectly handles base_frame and native frame as the first frame[/+] on Sep 19, 2026

  4. liuzhijie-0614 commented on Sep 19, 2026

    @liuzhijie-0614
    liuzhijie-0614
    on Sep 19, 2026
    ContributorAuthor

    I've updated the issue description to clarify the two cases I've identified:

    1. the first frame is base_frame, which should be treated as a valid empty Python stack;
    2. the first frame is a native frame, which should produce a <native> entry when native frame collection is enabled.

    Since gh-157657 was closed, I'd like to confirm the preferred direction for this issue.

    @pablogsal , would it make sense to continue investigating these cases and prepare a fresh PR with focused regression tests, or would you prefer a different approach here? Thanks!

  5. pablogsal commented on Sep 19, 2026

    @pablogsal
    pablogsal
    on Sep 19, 2026
    Member

    Since #157657 was closed, I'd like to confirm the preferred direction for this issue.

    Was closed as it violates https://devguide.python.org/getting-started/ai-tools/. Since this issue was also AI generated I don't trust right now more PRs for it. I will probably take of it myself.

  6. liuzhijie-0614 commented on Sep 19, 2026

    @liuzhijie-0614
    liuzhijie-0614
    on Sep 19, 2026
    ContributorAuthor

    Since #157657 was closed, I'd like to confirm the preferred direction for this issue.

    Was closed as it violates https://devguide.python.org/getting-started/ai-tools/. Since this issue was also AI generated I don't trust right now more PRs for it. I will probably take of it myself.

    Alright, this is a real issue discovered in a production environment, and we genuinely want to resolve it. All examples come from actual environments.

  7. pablogsal commented on Sep 19, 2026

    @pablogsal
    pablogsal
    on Sep 19, 2026
    Member

    Since #157657 was closed, I'd like to confirm the preferred direction for this issue.

    Was closed as it violates devguide.python.org/getting-started/ai-tools. Since this issue was also AI generated I don't trust right now more PRs for it. I will probably take of it myself.

    Alright, this is a real issue discovered in a production environment, and we genuinely want to resolve it. All examples come from actual environments.

    Thanks for letting me know

  8. pablogsal commented on Sep 19, 2026

    @pablogsal
    pablogsal
    on Sep 19, 2026
    Member

    All examples come from actual environments.

    Also, what examples are you talking about? There is no "examples|

  9. liuzhijie-0614 commented on Sep 19, 2026

    @liuzhijie-0614
    liuzhijie-0614
    on Sep 19, 2026
    ContributorAuthor

    All examples come from actual environments.

    Also, what examples are you talking about? There is no "examples|

    this case：

    PyGILState_STATE state = PyGILState_Ensure();
PyThreadState *saved = PyEval_SaveThread();

/* Wait on a pthread condition variable while the parent samples. */

/* Cleanup after sampling. */
PyEval_RestoreThread(saved);
PyGILState_Release(state);

    I have created a base frame case locally. The native frame case was discovered during the analysis of the stack, but it hasn't been constructed yet. I am currently attempting to do so.

    tachyon-empty-native-repro.zip

  10. pablogsal
    added a commit that references this issue on Sep 19, 2026

    pythongh-157605: Accept empty native-thread stacks when sampling

    2c982a2

  11. liuzhijie-0614 commented on Sep 20, 2026

    @liuzhijie-0614
    liuzhijie-0614
    on Sep 20, 2026
    ContributorAuthor

    @pablogsal Thanks for the fix! This looks like it resolves the issue.

    While testing this PR, I noticed that a cleared Python frame could also be mistaken for a native frame. In my test run, about 0.288% of samples contained an unexpected frame.

    This seems like a separate issue. With the initial-frame check removed, this case now produces a frame instead of an error. Do you have any suggestions on how we could handle it? I’ve included a small reproducer below.

    flamegraph.html

    import os
import signal
import time
from _remote_debugging import RemoteUnwinder


def leaf():
    pass


def parent():
    while True:
        leaf()
        n = 0
        while n < 100:
            n += 1


pid = os.fork()
if pid == 0:
    parent()

try:
    time.sleep(0.1)
    unwinder = RemoteUnwinder(pid, native=True, gc=False, cache_frames=False)
    for _ in range(200_000):
        frames = unwinder.get_stack_trace()[0].threads[0].frame_info
        names = [f.funcname for f in frames]
        if names[:2] == ["<native>", "parent"]:
            print("Reproduced:", names)
            break
    else:
        print("Race not observed; run again.")
finally:
    os.kill(pid, signal.SIGKILL)
    os.waitpid(pid, 0)

  12. liuzhijie-0614 commented on Sep 20, 2026

    @liuzhijie-0614
    liuzhijie-0614
    on Sep 20, 2026
    ContributorAuthor

    Both a real first frame and a cleared frame incorrectly recognized as seem related to frame cleanup:

                // GH-99729: We need to unlink the frame *before* clearing it:
            _PyInterpreterFrame *dying = frame;
            frame = tstate->current_frame = dying->previous;
            _PyEval_FrameClearAndPop(tstate, dying);
    1. The first frame is really a frame. In the following example, the call chain is parent (Python) → sorted (C) → key (Python). When key returns, tstate->current_frame points to the trampoline while the Python frame is being cleared.
    def key(value):
    return value

def parent():
    while True:
        sorted((2, 1), key=key)

parent()
    1. The first frame is a cleared Python frame, caused by a read race. The sampler still has the address of dying, although tstate->current_frame has already moved to the caller. _PyEval_FrameClearAndPop() clears the code pointer, but previous can still be non-NULL, so the frame is mistakenly recognized as a frame.
  13. pablogsal
    added 6 commits that reference this issue on Sep 22, 2026

    pythongh-157605: Accept empty native-thread stacks when sampling

    fa19750

    pythongh-157605: Accept empty native-thread stacks when sampling

    8aec82f

    pythongh-157605: Distinguish C frames from cleared Python frames

    d265973

    pythongh-157605: Distinguish C frames from cleared Python frames

    6cc82bb

    pythongh-157605: Distinguish C frames from cleared Python frames

    e37e929

    gh-157605: Accept empty native-thread stacks when sampling

    80f9aa0
  14. liuzhijie-0614
    closed this as completedon Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtopic-profilingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions