diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 13e74a486cb..2007070652d 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -722,6 +722,8 @@ peps/pep-0844.rst @warsaw peps/pep-0845.rst @ethanfurman peps/pep-0846.rst @JelleZijlstra @johnslavik peps/pep-0847.rst @dstufft +# peps/pep-0848.rst +peps/pep-0849.rst @JelleZijlstra # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-0849.rst b/peps/pep-0849.rst new file mode 100644 index 00000000000..e6bd3434403 --- /dev/null +++ b/peps/pep-0849.rst @@ -0,0 +1,462 @@ +PEP: 849 +Title: More Expressive Type Expressions +Author: Imogen Hergeth +Sponsor: Jelle Zijlstra +Discussions-To: Pending +Status: Draft +Type: Standards Track +Topic: Typing +Created: 20-Sep-2026 +Python-Version: 3.16 +Post-History: `05-Nov-2025 `__ + `02-Sep-2026 `__ + + +******** +Abstract +******** + +Currently, the Python interpreter allows any expression to be used as an +annotation. But their most popular usage, type annotations, are restricted to +only a small subset of possible expressions. This is because type annotations +need to be introspectable at runtime and many valid expressions produce +values where that is not possible. This greatly limits which expressions can +be assigned a meaning within the typing spec and limits the expressiveness and +ease of use of type annotations. + +This PEP proposes to expand annotate functions such that +any Python expression can be used in type annotations. It does this by creating +a new annotation format which instructs annotate functions to return the +annotation's AST and namespaces. Runtime consumers of type annotations can then +evaluate these into typing objects. + +.. _motivation: + +********** +Motivation +********** + +Python type annotations are expressions attached to variable names that denote +which values can be assigned to a variable. They are not only available to +linters and type checkers via the source code directly but also to runtime +reflection uses because the interpreter creates special annotate functions that +compute the objects the annotation expressions evaluate to. + +While this approach works well for many existing annotations and is easy to +understand, it also greatly limits the kinds of expressions that can be used +as type annotations. In the abstract, this is because the annotate functions +evaluate such expressions using the same mechanism that Python uses for regular +value expressions, and in many cases those semantics are not compatible with +runtime reflection needs. + +We will now provide several more concrete examples of expressions that could +see use in type annotations but are currently not feasible. However, we are not +directly advocating for any specific such usage and this PEP does not implement +any of them. Rather, we are providing the groundwork that would make them and +similar future proposals possible. + +:pep:`586` introduced literal types, which enumerate a concrete list of possible +literal values, e.g. the numbers ``1``, ``2`` and ``3``. This is currently spelled +as ``Literal[1, 2, 3]``. Many users intuitively want to instead spell this as +``1 | 2 | 3``, writing out the bare literals and using the union operator to join +them, which also is what other languages such as TypeScript use. This is +currently not possible in Python because the expression ``1 | 2 | 3`` evaluates +simply to ``3`` because ``|`` is interpreted as the binary-or operation rather than +the union of types. Further, constant folding eliminates this +expression from appearing anywhere in the generated bytecode, and it thus is +impossible to retrieve the actual type annotation at runtime. + +The above example only prevents us from using slightly shorter spellings for +already existing types. There also are many type annotations that either +are reasonably possible or even currently being discussed, but which are +infeasible to implement currently. The currently open draft of :pep:`827` +introduces many such types. For example, it proposes conditional types, which +are type expressions that evaluate to one of two types depending on some +condition. Intuitively, these would be written as +``FirstType if TypeCondition else OtherType``, just like ternary expressions. +But that is not possible because no matter what ``TypeCondition`` evaluates to +at runtime, one of the other type expressions will never be evaluated and thus +be invisible to runtime introspection. + +Similar issues arise when defining other types that are defined using other +existing types. For example, one might want to write +``{K: NotRequired[T] for K, T in SomeTypedDict}`` to define a typed dict that has +the same definition as an existing typed dict, but where every key is optional. +This is currently not possible because type objects cannot be iterated over in +this way. This has also already been discussed in e.g. `this thread +`__. + + +********* +Rationale +********* + +Overview +======== + +We propose to make it possible to use arbitrary expressions in type expressions +while maintaining full runtime introspection capabilities by introducing a new +format for annotate functions. It will instruct them to return objects that +contain both the annotations' ASTs and their namespace. These can then be used +to construct the actual type objects that runtime introspection users are +interested in. + +For example, consider the following class: + +.. code-block:: python + + class MyClass: + a: int + b: list[str] + +When calling ``MyClass.__annotate__(Format.VALUE)`` it will still return the +usual annotation dict ``{"a": int, "b": list[str]}``. But when called as +``MyClass.__annotate__(Format.AST)`` we receive this dictionary: + +.. code-block:: python + + { + "a": AnnotationAST(ast.Name("int"), {"int": int}), + "b": AnnotationAST(ast.Subscript(ast.Name("list"), ast.Name("str")), {"list": list, "str": str), + } + +However, most users will never see these objects directly. Rather, we propose +to add a new function ``get_type_annotations`` to the :py:mod:`typing` +module, which will internally perform the above call and then return the +familiar ``{"a": int, "b": list[str]}`` annotation dictionary. + +The power of this approach is that it lets us implement new type expressions +such as the ones mentioned above by simply extending the evaluation logic in +``get_type_annotations``. This logic can then be completely decoupled from the +usual Python expression semantics and can instead create typing objects that +allow complete runtime introspection. + +We also propose to add some additional utility functionality related to +annotate functions and these AST objects. In particular, a new +``create_annoate_function`` in the :py:mod:`annotationlib` module to easily +synthesize an annotate function. The core of this functionality is already +implemented in the :py:mod:`dataclasses` module, and we foresee that many +users will need this in order to create annotate functions that support +this somewhat more complex format. + + +Implementation +============== + +While the exact mechanism annotate functions use is an implementation detail, +it may be useful to look at what that might look like. Consider, for example, +the class from above: + +.. code-block:: python + + MyClass: + a: int + b: list[str] + +Under this proposal its annotate function may behave similar to this Python code: + +.. code-block:: python + + def __annotate__(format): + namespace = { + "int": int, + "list": list, + "str": str, + } + asts = { + "a": "\x1a\x03int", + "b": "\x18\x1a\x04list\x1a\x03str", + } + return _make_annotate_asts(namespace, asts) + +It functions in essentially three steps. It populates the namespace dictionary +by performing ordinary variable name lookups. This is important to ensure that +we can later evaluate the returned AST objects using the correct bindings for +each variable. It also loads some string constants that contain binary data +defining each annotation's AST. This is a compact format that is easily +implemented with existing compiler functionality. It also is much faster to +parse than storing the source code of the annotations directly. The annotate +function then uses a new intrinsic to actually build the required annotation +AST objects. + +The proposed ``get_type_annotations`` function would then compute the type objects +using a function like this: + +.. code-block:: python + + def eval_type_annotation_AST(ast, namespace): + match ast: + case ast.Name: + return namespace[ast.id] + case ast.Attribute: + value = eval_type_annotation_AST(ast.value, namespace) + return getattr(value, ast.attr) + ... + + +Performance Impact +================== + +While every new feature has to be weighed against its impact on performance and +complexity, typing related features deserve additional scrutiny because type +annotations are an entirely optional part of the Python language. We thus need +to consider three separate groups of users and its impact on them: users who +do not use type annotations at all, users who annotate their code for type +checkers and/or linters but don't use runtime introspection and finally, users +who also evaluate their type annotations at runtime. + +For this, we considered three metrics: the time it takes to import modules both +without and with type annotations, the annotate functions' size in memory and +the time it takes to actually evaluate the annotate functions. The +import time is most important for the first group of users; the second group is +also affected by the annotate functions' memory footprint and the time to +evaluate the annotate functions only impacts the last group of users. + +Using our reference implementation, we have found no significant difference in +import times of modules that do not use type annotations. For modules that do +use annotations, import was moderately faster and the memory footprint slightly +smaller using the proposed annotate functions, both by a few percent. But +unfortunately, evaluation time can increase significantly. In the worst case, +when annotations are requested in the value format and every used name is +defined, the increase is about sevenfold. However, when some names are not +defined and the ``STRING`` or ``FORWARDREF`` formats have to be used, the +current approach is also significantly slower and results in comparable times +to the proposed annotate functions. + +A common situation where type annotations are evaluated is when tools like +:py:mod:`dataclasses` or similar ORM packages analyze class or function +definitions to synthesize additional behavior. For these tools, the time to, +e.g., create a dataclass will be impacted by this proposal. But as mentioned +above, there already are many situations where inspecting annotations takes +a similar amount of time. + +In total, since the negative performance impacts only affect the smallest group +of users, who also benefit from the newly possible type annotations, we consider +this a worthwhile trade-off. The specification of the proposed format is also +open enough that many optimizations are possible should they be deemed necessary +in the future. For many users, this proposal will even be a slight performance +increase since their code never evaluates any annotate functions. + +.. _outside-annos: + +Usage Outside of Annotations +============================ + +While this proposal enables the usage of future type expressions in annotations, +there also are other places where users want to write type expressions. For +example in ``cast(, value)``. Since the interpreter +cannot differentiate these cases from other function calls, it is impossible +for it to infer that it should use a mechanism like we suggest. + +This problem can be avoided using an intermediate type alias: + +.. code-block:: python + + type _TargetType = + cast(_TargetType, value) + +This lets you use any new type expression within ```` +since type aliases also are implemented using annotate functions. + +While this solution only presents a workaround to this problem, a more +comprehensive fix would require adding a new keyword to the Python language, +which we do not think is necessary. This decision can be reconsidered in the +future if using intermediate type aliases like this does present a significant +problem in real-world code. + + +************* +Specification +************* + +Throughout this PEP, when we talk about *annotate functions/methods*, we are +referring to both the ``__annotate__`` special methods found on some objects +and the following methods found on typing objects: + +* ``evaluate_value`` on :py:class:`typing.TypeAliasType` +* ``evaluate_bound``, ``evaluate_constraints``, and ``evaluate_default`` on :py:class:`typing.TypeVar` +* ``evaluate_default`` on :py:class:`typing.ParamSpec` +* ``evaluate_default`` on :py:class:`typing.TypeVarTuple` + +When referring to their return values, we mean either the objects contained in +the dictionary returned by ``__annotate__`` methods or the single object +returned by the other methods. + +The ``AST`` Format +================== + +A new value called ``AST`` is added to the ``annotationlib.Format`` enum with +value 5. Annotate functions do not have to support this format. If an +annotate function is called with this format, it must return a +``AnnotationAST`` object. These are instances of a proposed new class that +hold the annotation's AST as an ``ast.expr`` object and a namespace that is +used to evaluate them. + +Compiler-generated annotate functions will always support this format. +They will store the necessary AST data stored as string constants and then +construct a new ``AnnotationAST`` objects each time they are called. The +contained AST objects will be identical to the objects created by parsing the +annotation source code directly. + + +Helper Functions +================ + +A new function ``typing.get_type_annotations`` is added that functions similarly +to the existing ``annotationlib.get_annotations``, but instead calls the +underlying annotate function with ``Format.AST`` and then constructs typing +objects using the new ``typing.evaluate_type_ast`` helper. + +The existing ``typing.get_type_hints`` function will be deprecated. It has +slightly different semantics to both ``annotationlib.get_annotations`` and the +proposed function, which make it impossible to instead modify it to support the +new functionality. It also will be completely superfluous since users of type +annotations will need to call ``get_type_annotations`` instead to properly +resolve any type annotations that contain new typing features. Leaving this +function as-is will only create confusion about which function should be used. + +The AST format also provides a simpler and more reliable method to create +annotations in the ``STRING`` and ``FORWARDREF`` formats. Currently, +compiler-generated annotate functions do not support these directly; rather, the +helper functions in :py:mod:`annotationlib` try to create a best-effort AST and +then unparse it into the requested format. Under this PEP these helper functions +will instead use the AST format and unparse the result. This creates more +accurate results for these formats in many cases. + +In order to simplify the creation of synthesized annotate functions, a new +helper function ``annotationlib.create_annotate_function`` will be added to +:py:mod:`annotationlib`. It accepts a mapping from variable names to annotation +objects in one of the existing annotation formats. Using these, it then returns +an annotate function that supports all annotation formats by converting the +passed-in values appropriately. + + +*********************** +Backwards Compatibility +*********************** + +The new annotation format and helper functions only add new functionality and +thus do not have backwards compatibility concerns. The deprecation of +``typing.get_type_hints`` means that existing code that uses this function will +break once it is removed from the standard library. We recommend that users +migrate to ``annotationlib.get_annotations`` or ``typing.get_type_annotations``, +depending on which semantics they want to use. While these functions have +slightly different behavior in some cases, they are a drop-in replacement +most of the time. + +We further want to point out that future additions +like the ones outlined in the :ref:`motivation` do not create backwards +compatibility problems, even if adopted gradually. + +Consider, for example, a change to the typing spec that makes it so that +``var: 1`` is interpreted as a literal type ``Literal[1]``. Users that evaluate +the annotation method directly or with one of the existing helper methods +will still observe the result as the plain integer ``1``. The changed semantics +are only considered when the user opts into them by calling the annotation +function with ``Format.AST`` or using ``typing.get_type_annotations``. + +In the most common use case, annotations are not consumed by the user directly +but by some library code that introspects user-defined objects. Thus, adoption +of new annotation semantics could be stalled if library authors have to worry +about existing user annotations being reinterpreted to different objects when +the library is updated. But this also is not an issue. The typing spec already +is covered by backwards compatibility concerns, which means that any currently +valid type annotations will not be changed to mean something different. +Potential future changes can only define new semantics to syntax constructs that +currently are not valid type annotations and thus do not occur in user code. + + +********************* +Security Implications +********************* + +There are no known security implications for this change. Calling annotation +functions already could execute arbitrary code defined in the annotated +object. Python also already offers the capability to access the source code and +compiled bytecode of introspected objects. The AST objects returned in the +new format thus do not contain any previously unavailable information. + + +***************** +How to Teach This +***************** + +Users of annotations are not directly affected by this proposal. It intends to +enable the simplification of the way type annotations are spelled; any future +changes based on this PEP will need to be evaluated on their own merits. +Documentation will inform users that any new semantics are only natively +supported in annotations. Since this is by far the most common place for types +to be spelled, we expect this to not be a big limitation. Other places where +types can occur already require users to wrap forward references in string +literals, so this is a known practice. Type checkers should warn users if they +do not wrap a type form using syntax that would be evaluated incorrectly in +a place where it is statically known that a type form is expected. + +The new format and changes to helper functions will be documented as part of the +language standard. Libraries that introspect type annotations will be able to +easily support any new type syntax by calling the provided utility functions +and should document this behavior so that their users are made aware of any +potential future changes. + +One potential issue is the :ref:`outside-annos` discussed in the +beginning. Since most users of type annotations will also use type checkers +and/or linters, we recommend that these tools implement checks for these errors +and suggest the fix via an intermediate type alias. + + +************************ +Reference Implementation +************************ + +This proposal is prototyped in `a CPython fork +`__. + +************** +Rejected Ideas +************** + +Creating Typing Objects Within Annotate Functions +=================================================== + +An initial idea was to modify annotate functions such that they create the +relevant typing objects themselves. This can be achieved in several ways, for +example via a new optional argument to signal typing-specific semantics or a new +format. These approaches were rejected because they force the type-specific +semantics to be defined in the interpreter itself. This not only limits the +semantics to e.g. not require namespace lookups and also locks the usage of +typing features to the Python version that is being used, rather than allowing +the currently possible backporting via ``typing_extensions``. + +Storing Annotation Source Code +============================== + +Instead of storing binary data that defines the annotations' ASTs an alternative +is to simply store the annotations source code directly. This also was presented +as a possibility all the way back in :pep:`649`. The two approaches are largely +equivalent since one can create the AST from the source code and vice versa. +While the unparsed AST is not necessarily the exact string that occurred in +the source code since the AST does not preserve the precise formatting of the code, it is semantically +equivalent and, in particular, is not affected by compiler optimizations. + +In a performance comparison, the two data representations also are largely +equivalent concerning import times and memory usage. However, parsing the source +code to then create typing objects is significantly slower than working with the +AST data, by a factor of 3. Storing the source code also greatly limits future +optimizations since the data representation is directly exposed as the API. + +Not Returning AST Namespaces +============================ + +In order to properly evaluate an AST into the correct typing objects, the +evaluating function needs to have access to the namespace the annotation was +defined in. Initially it seems like this namespace can be reconstructed from +the annotate function's object since they contain the globals and cellvars used. +However, this is not sufficient since namespaces can use more complex lookup +logic when ``global`` statements or name mangling are used. + +********* +Copyright +********* + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.