ExecuTorch Documentation — On-Device LLM and Model Inference with PyTorch
by PyTorch Foundation
PyTorch's first-party path from a Llama or Qwen checkpoint to a quantized .pte running on a phone, laptop or microcontroller.
Overview
ExecuTorch reached 1.0 general availability on 22 October 2025, so the export API and the .pte artifact format are now stable rather than moving targets. The documentation is split into Intro, Quick Start, Edge (Android, iOS, desktop, embedded), Backends, LLMs, Advanced, Tools, API and Support, with three suggested pathways for beginner, quick-start and advanced readers. The mental model it teaches is ahead-of-time compilation: export the PyTorch graph with PT2 export — not TorchScript — compile and delegate it to a hardware backend, then execute through a lightweight C++ runtime. The LLM section is a concrete seven-page path rather than an overview: exporting with the native export_llm API, exporting via Hugging Face Optimum, exporting a custom architecture, running with C++, and running on Android with XNNPACK, on Android with Qualcomm AI Engine Direct, and on iOS. export_llm is driven by a YAML file whose fields are defined in LlmConfig, and supports Llama 2/3/3.1/3.2, Qwen 2.5/3, Phi 3.5/4-mini and SmolLM2 out of the box, with the Optimum path for architectures like Gemma and Mistral. Quantization is documented two ways: TorchAO int8-dynamic-activation plus int4-weight (8da4w) or int8 weight-only for XNNPACK CPU targets, and pt2e post-export graph quantization when targeting QNN, CoreML or Vulkan. Twelve-plus backends are documented individually — XNNPACK, Vulkan, Qualcomm, MediaTek, Samsung Exynos, CoreML, MLX, OpenVINO, experimental CUDA, ARM Ethos-U, NXP and Cadence DSP. The project is BSD-licensed with roughly 5,000 GitHub stars and over 13,500 commits, and Arm reports the runtime already ships inside Facebook, Instagram, Messenger and WhatsApp.
At a Glance
- Topic
- Frameworks
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~3-5 hours for the quick start plus the full LLM export-and-run path; reference thereafter
- Provider
- PyTorch Foundation
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Export a PyTorch model to a portable .pte artifact using PT2 export rather than TorchScript
- ✓Drive LLM export through a YAML LlmConfig instead of writing bespoke export scripts
- ✓Quantize with TorchAO 8da4w or int8 weight-only for CPU-class XNNPACK targets
- ✓Apply pt2e post-export quantization when targeting Qualcomm, CoreML or Vulkan backends
- ✓Invoke the exported model from the C++ runtime or the Swift and Java bindings
- ✓Deploy the same artifact to Android via XNNPACK or Qualcomm AI Engine Direct, and to iOS
- ✓Bring unsupported architectures such as Gemma or Mistral through the Hugging Face Optimum path
- ✓Profile and debug on-device execution using the ExecuTorch developer tooling section
Highlights
- •First-party PyTorch route to on-device inference — no ONNX, GGUF or TorchScript conversion detour
- •Reached 1.0 GA on 22 October 2025, so the export API and .pte format are stable, not experimental
- •Each of the twelve-plus hardware backends gets its own page, from Apple CoreML and MLX down to ARM Ethos-U microcontrollers
- •The LLM section is an end-to-end path from checkpoint to a running Android or iOS app, with the quantization choice made explicit per backend
- •Arm reports the runtime already serves production traffic inside Facebook, Instagram, Messenger and WhatsApp
Who It’s For
Best For
- ✓Mobile and edge engineers shipping an LLM inside an Android or iOS application
- ✓AI engineers who need offline or privacy-preserving inference with no server round trip
- ✓Embedded developers targeting NPUs, DSPs or microcontrollers with PyTorch models
- ✓Teams evaluating on-device alternatives to hosted inference on cost or latency grounds
Prerequisites
- •Solid PyTorch: model loading, torch.export, and the difference between eager and compiled graphs
- •Comfort with a native toolchain — C++, Gradle/Android Studio or Xcode — to actually run on hardware
- •Basic understanding of the accuracy tradeoffs between int4, int8 and full-precision inference
FAQ
What is ExecuTorch Documentation — On-Device LLM and Model Inference with PyTorch?
ExecuTorch is PyTorch's runtime for running models directly on phones, laptops, embedded boards and microcontrollers, and this is its official documentation. It is aimed at engineers who need an LLM or vision model to run locally — for privacy, offline use, or latency — rather than behind a hosted API. Work through it and you can export a Llama or Qwen checkpoint to a quantized .pte artifact and call it from C++, Swift or Java on real hardware.
Is ExecuTorch Documentation — On-Device LLM and Model Inference with PyTorch free?
ExecuTorch Documentation — On-Device LLM and Model Inference with PyTorch is free to access.
What level is ExecuTorch Documentation — On-Device LLM and Model Inference with PyTorch for?
ExecuTorch Documentation — On-Device LLM and Model Inference with PyTorch is aimed at a advanced audience. Recommended background: Solid PyTorch: model loading, torch.export, and the difference between eager and compiled graphs, Comfort with a native toolchain — C++, Gradle/Android Studio or Xcode — to actually run on hardware, Basic understanding of the accuracy tradeoffs between int4, int8 and full-precision inference.
How long does ExecuTorch Documentation — On-Device LLM and Model Inference with PyTorch take?
Expect roughly ~3-5 hours for the quick start plus the full LLM export-and-run path; reference thereafter. Most learners work through it at their own pace.
What will I learn from ExecuTorch Documentation — On-Device LLM and Model Inference with PyTorch?
You'll learn: Export a PyTorch model to a portable .pte artifact using PT2 export rather than TorchScript; Drive LLM export through a YAML LlmConfig instead of writing bespoke export scripts; Quantize with TorchAO 8da4w or int8 weight-only for CPU-class XNNPACK targets; Apply pt2e post-export quantization when targeting Qualcomm, CoreML or Vulkan backends; Invoke the exported model from the C++ runtime or the Swift and Java bindings; Deploy the same artifact to Android via XNNPACK or Qualcomm AI Engine Direct, and to iOS; Bring unsupported architectures such as Gemma or Mistral through the Hugging Face Optimum path; Profile and debug on-device execution using the ExecuTorch developer tooling section.
Topics
Sources
This page was written from 4 sources, 2 on domains other than docs.pytorch.org.