Add initial draft of Rust API design document

[emmatyping]

Okay, it took a while to play with these API design ideas and get them to a state where I was happy with them. I have a rough prototype similar to #41 that is based on this, but I figured it would make more sense to discuss the API design and make sure I haven't missed things or there isn't unsoundness in this design. It turns out it is just darn difficult to get thread-state passing right when you consider re-entrancy and multi-threading.

As mentioned in the document, my overall priorities in this design were:

1. be sound on GIL and free-threaded builds;
2. support every PEP 11 platform;
3. approach equivalent C performance where safety permits;
4. feel familiar to PyO3 and C API users; and
5. use modern APIs such as PyModExport.

I think 3 is probably the furthest from achievable with this current design, we query the TLS quite frequently with this design, but I believe a lot of those queries could be cleaned up. 4 is really a matter of taste, but I tried to be somewhat similar while also trying to have as small an API surface as possible.

I also wasn't totally sure where to put this, perhaps it would go better in InternalDocs?

AI disclosure: I worked with Codex and Claude when iterating on this API and the content is a mix of writing me and the LLM. I've reviewed every word, but please let me know if there is anything unclear!

[ngoldbaum]

Did a read-through and had a few comments questions and suggestions

[emmatyping]

Okay, big update! I have revised the API design doc significantly, re-working how thread states/contexts work, how exceptions work, and adding support for what I believe is a safe subinterpreters API. I've also responded to the great feedback so far (thank you to everyone who has commented!)

I'd greatly appreciate re-review as folks have time. Barring major issues I hope to bring this as the design to the core sprints to discuss with folks there.

[willingc]

@emmatyping I made some suggestions on the intro for greater clarity. Overall, the doc as you presented the other day looks sound.

[dimaqq]

I've played this spec back and forth against AI. Here's what it considers a possible race, though I feel this is a bit beyond my level of comprehension when it comes to the details:

sequenceDiagram
    participant E as Rust entry thread
    participant F as Finalizer

    alt Entry acquires first
        E->>E: Read admission = open
        E->>E: Increment its thread-state lease count<br/>without a safepoint in between
        E->>E: Publish ThreadContext
        F->>F: Stop the world (or hold the GIL)<br/>and recheck lease counts
        F-->>F: Sees lease > 0. Waits for entry to finish
    else Finalizer closes first
        F->>F: Stop the world (or hold the GIL)<br/>and recheck counts = 0
        F->>F: Close admission
        E->>E: Read admission = closed
        E-->>E: Reject entry. Publish no context
    end

Loading

[dimaqq]

Explanation for the above: proposed faster per-thread lease scheme needs to prove it preserves the same exclusion as the public PyInterpreterGuard.

"acquisition reads the flag and increments the lease count without a safepoint between, while finalization stops the world and rechecks counts before closing admission" could be sound, not only if CPython guarantees that finalisation cannot pass it's check while an entry is getting paused.

The AI friend proposes that "the document should state explicitly that an attached thread cannot be counted as stopped midway through the lease-acquisition sequence, and that the finalizer closes admission only after that stop-the-world rendezvous and its final count recheck." Or in other words, stating "without not safepoint between" is good, but now please prove that read and increment is really an atomic operation or specify how it can be implemented so that it's really an atomic operation.

I've also debated the possible approaches, here are the three alternatives:

1. use PyInterpreterGuard for each Rust entry: simple but there's a hot counter
2. put closed and active lease count into a single atomic: not sure if that's available on all platforms
3. the proposed per-thread lease counts: correctness depends on stop-the-world and resume procol

Now... it bugs me a little that the core spec gets so complicated, but then again so is free-threading. So maybe that's OK.

I do wonder though, if the spec implies "enhancing" cpython core thread state tracking for the first-party Rust code, what about the third-party Rust modules, or modules written in other languages? Should we export something here? Would that be a good idea?

[emmatyping]

I've played this spec back and forth against AI. Here's what it considers a possible race

Thanks for poking at this!

acquisition reads the flag and increments the lease count without a safepoint between

Ah! The acquisition occurs while attached to the interpreter, so a stop-the-world pause cannot run until that thread detaches or finishes, and another thread cannot run a finalizer in the GIL-ful build.

I agree however that rephrasing the invariants there would be quite useful. I've pushed db9f214 with the wording change.

[encukou]

Thank you! This looks as sound as the C API will allow :)

[encukou]

For info: I've started on this but not a very high priority for me. If someone wants to take this, let's coordinate.

[emmatyping]

Absolutely! Would love to coordinate on this, as it's obviously a blocker for Rust. The PyModExport_ interface is way more ergonomic for Rust.

[encukou]

Into the sprint list it goes :)

[encukou]

_PyInterpreterState_Enable resets next_id and the main interpreter is always ID zero.

Is that an issue to fix in CPython? I'm not sure if we baked this into public API.

[emmatyping]

TBH I haven't checked. I feel like I spoke to someone who mentioned that this was intentional, but I should follow up on that. Changing this would be much better :)

[encukou]

That would probably be Eric Snow, I'd like to hear the reasons. (In person if he's at the sprint, so I'll not tag him here now.)