From c5f1747af6354a68b6c5500396e82d63ac83acd6 Mon Sep 17 00:00:00 2001 From: Digant Desai Date: Sat, 17 Oct 2026 00:25:01 -0500 Subject: [PATCH] [native] Add a single ExecuTorch Native README Replace exir/native/README.md with a shorter backends/native/README.md, next to the other backend READMEs and most of the Native code. It covers exporting with to_native, what the partitioner accepts, the .ptn layout, and the runtime engines that exist today (Vulkan and ggml). The pointers in exir/native/native.py now reference the new location. Authored with Claude Code. --- backends/native/README.md | 61 ++++++++++++++++ exir/native/README.md | 144 -------------------------------------- exir/native/native.py | 6 +- 3 files changed, 64 insertions(+), 147 deletions(-) create mode 100644 backends/native/README.md delete mode 100644 exir/native/README.md diff --git a/backends/native/README.md b/backends/native/README.md new file mode 100644 index 00000000000..a29bff270b2 --- /dev/null +++ b/backends/native/README.md @@ -0,0 +1,61 @@ +# ExecuTorch Native + +> [!NOTE] +> Experimental. The Python API, the `.ptn` format, and operator coverage may change. + +ExecuTorch Native lowers a PyTorch model to a single `.ptn` file and runs it on +one of several engines. In a `.pte`, delegates lower the graph ahead of time for +specific backends. A `.ptn` keeps the whole graph without backend-specific +information, and each engine lowers it when compiling a method. + +``` +.pte: export -> [quantize] -> to_edge -> to_backend -> to_executorch -> .pte (+ .ptd) +.ptn: export -> [quantize] -> to_native -> .ptn +``` + +## Export + +```python +import torch +from executorch.exir.native import to_native + +ep = torch.export.export(model, example_inputs) +to_native(ep).save("model.ptn") + +# Several methods in one file. +to_native({"forward": ep_forward, "prefill": ep_prefill}).save("model.ptn") +``` + +The partitioner accepts core ATen ops plus `linear`, `matmul`, `addmm`, +`scaled_dot_product_attention`, `rms_norm`, `native::rope`, and TorchAO affine +quantize/dequantize ops. TorchAO int4 group-wise weights of `linear` and +`embedding`, and GGUF k-quant weights from `executorch.extension.llm.export.gguf`, are +stored packed with their scheme in tensor metadata. Each method must lower wholly +to one delegate; `to_native` raises otherwise. + +A `.ptn` is an uncompressed zip with fixed entry names: + +| Entry | Contents | +| --- | --- | +| `program.ptg` | Methods as graphs; FlatBuffer defined by `serialization/native_graph.fbs`. | +| `program.safetensors` | Constants. Byte-identical immutable tensors are stored once. | +| `aliases.json` | Deduplicated key to owner key; present only when needed. | + +## Runtime + +The C++ runtime reads a `.ptn` into an in-memory graph and hands each method to +an engine through the interfaces in `runtime/engine/Engine.h`: + +- `EngineHost`: process-wide device resources. +- `EngineContext`: state for one loaded program, including mutable buffers. +- `EngineExecutable`: one compiled method (`set_input`, `execute`, `get_output`). + +An engine rejects a method whose semantics it cannot fully honor. + +| Engine | Source | Notes | +| --- | --- | --- | +| Vulkan | `runtime/vulkan/` | Builds on the ET-VK `ComputeGraph` and shaders. | +| ggml | `runtime/engine/ggml/` | CPU, CUDA, HIP, and Metal variants. | + +The runtime currently builds only with Meta's internal Buck setup; there is no +open-source build yet. diff --git a/exir/native/README.md b/exir/native/README.md deleted file mode 100644 index 1f461b1bfce..00000000000 --- a/exir/native/README.md +++ /dev/null @@ -1,144 +0,0 @@ -# Direct native lowering (`to_native`) - -`to_native` lowers exported PyTorch programs for the native portable runtime and -produces a `.ptn` package. Where the ExecuTorch flow is - -``` -export -> [quantize] -> to_edge -> to_backend -> to_executorch -> .pte (+ .ptd) -``` - -this flow is - -``` -export -> [quantize] -> to_native -> .ptn -``` - -## Usage - -```python -import torch -from executorch.exir.native import to_native - -ep = torch.export.export(model, example_inputs) -program = to_native(ep) -program.save("model.ptn") -``` - -Multiple methods are lowered together by passing a dict: - -```python -program = to_native({"forward": ep_forward, "prefill": ep_prefill}) -print(program.methods) # {"forward", "prefill"} -program.save("model.ptn") -``` - -## API - -### `to_native(programs) -> NativeProgramManager` - -`programs` is a single `ExportedProgram`, lowered as the sole `forward` method, or -a non-empty dict mapping non-empty string method names to `ExportedProgram`. - -Nothing about lowering is configurable yet, intentionally. We will open if -needed in the future like `constant_methods` and ETRecord integration. - -Raises `TypeError` for the wrong input or method-value types. Raises `ValueError` -for an empty method mapping or method name, if a method does not fully delegate to -the native backend, if a constant differs between methods, or if a data key the -graph references has no backing tensor. - -## Current limitations: one complete delegate per method - -Each method must lower to **exactly one** native delegate that covers the whole -method, and the top-level graph must be an identity wrapper around it. This is not -"every `ExportedProgram` the native serializer can represent" — these cases are -outside the contract and fail loudly rather than producing a partial package: - -- **Identity methods** (`return x`) delegate nothing, so there are zero delegates. -- **Disconnected supported regions** become several partitions. -- **Graph breaks** leave an unsupported op in the outer graph. In the ExecuTorch - path that op would run outside the delegate; here there is no outer runtime. -- **Outer-graph plumbing** that drops, reorders, duplicates, or injects values — - including a pass-through like `return x, x + 1` — changes the method's contract - and cannot be represented by packaging the delegate alone. - -Validation happens before a `NativeProgramManager` exists, and `save` is a separate -call, so a rejected program never writes a file. - -TODO: compare the merged method's complete output arity, mutation targets and -ordering, and dynamic-shape constraints. The current wrapper check rejects outer-graph -plumbing it can observe, but these signature details need merged-program metadata. - -### `NativeProgramManager` - -The counterpart to `ExecutorchProgramManager`. `methods` and `save` mirror it -deliberately, so the two pipelines read alike. - -| Member | Description | -| --- | --- | -| `methods` | Names of the methods in the program. | -| `save(path)` | Write the program and its constants to `path` as a `.ptn`. | - -The constructor takes the graph blob and the constants, and nothing else. Method -names and the package's mutable-buffer set are read back out of the blob on demand -rather than passed alongside it, so a manager cannot describe a package it does not -hold. - -The surface is intentionally small. TODO: add `buffer`, `exported_program`, -`write_to_file`, and debug-map access only when concrete callers establish the -required contracts; these additions can remain additive. - -Constants are held as tensor references returned by lowering. `save` may normalize -device or layout before writing them. - -## The `.ptn` package - -An uncompressed zip (TODO: revisit the packaging format later) with fixed member names: - -| Entry | Contents | -| --- | --- | -| `program.ptg` | Native Program flatbuffer, file identifier `NPTG`. | -| `program.safetensors` | Constants the graph references; immutable values may be content-deduped. Present only when the graph references constants. (TODO: revisit this format later) | -| `aliases.json` | Duplicate data key to owner key. Present only when duplicates exist. | - -Byte-identical immutable tensors are stored once: the first key owns the -`safetensors` entry and the rest alias to it. An owner is always a real -`safetensors` key and never appears in the alias map, so resolution is -`owner = aliases.get(key, key)`. - -`save` writes each `safetensors` owner payload straight into the zip instead of -materializing the whole model as Python bytes. Owner tensors remain live until the -write completes, and non-CPU or non-contiguous inputs may require normalized tensor -copies. - -Only **immutable** constants are deduplicated. An alias means "these keys held -identical bytes at save time, so one copy was stored" — never that two keys share -runtime state. Mutable buffers are excluded, because two independently mutable -buffers can be zero-initialized to identical bytes and must not end up aliased. -Distinct data keys that share source storage are rejected when either is mutable: -The current PTN format cannot encode that alias topology, and storing the keys -independently would silently change mutation semantics. TODO: add explicit storage -groups and view metadata if distinct-key mutable aliasing becomes necessary. - -TODO: save atomically — write beside the destination and replace it once complete, -so a failure cannot truncate an existing valid package. - -## Relationship to the ExecuTorch path - -`NativePartitioner` remains usable directly through -`to_edge_transform_and_lower`, in which case lowering behaves like any other -ExecuTorch delegate: the graph is a delegate blob and constants are shipped via the -`NamedDataStore`, honoring `external_constants_tag`. That path is unchanged and -still produces a `.pte`. - -`to_native` opts the partitioner into handing constants back out of band instead, -which is why it returns a terminal `NativeProgramManager` rather than an -`EdgeProgramManager`: such a program has no constants to serialize into a `.pte`. - -## Layout - -| Path | Contents | -| --- | --- | -| `exir/native/` | This API: `to_native`, `NativeProgramManager`. | -| `exir/_serialize/_ptn.py` | The `.ptn` format, independent of any backend. | -| `backends/native/` | The backend: partitioner, preprocess, passes, and the `NPTG` graph serializer. | diff --git a/exir/native/native.py b/exir/native/native.py index fe0df625da7..ad85f110734 100644 --- a/exir/native/native.py +++ b/exir/native/native.py @@ -10,7 +10,8 @@ Direct native lowering. Where the ExecuTorch flow is export -> [quantize] -> to_edge -> to_backend -> to_executorch, this -is export -> [quantize] -> to_native, and yields a NativeProgramManager to save as a .ptn. See README.md. +is export -> [quantize] -> to_native, and yields a NativeProgramManager to save as a .ptn. +See backends/native/README.md. Lowering runs the same to_edge_transform_and_lower pipeline as the ExecuTorch path (the default native passes plus NativePartitioner), but opts the partitioner into @@ -425,8 +426,7 @@ def to_native( f"wholly supported by the native backend: an identity method " f"produces none, and disconnected supported regions produce " "several. Residual operator targets: " - f"{residual_targets}. See the README for this limitation of " - "the current implementation." + f"{residual_targets}." ) module = lowered[0][1] _check_delegate_contract(