For a small model on twenty thousand corporate laptops, ship your own pinned model on llama.cpp, and add Microsoft's Foundry Local only on the Windows SKUs where a pilot proves the NPU path works. Do not build a fleet feature on the model the operating system provides. Apple updates its on-device model with OS releases, and Microsoft removes Phi Silica from retail Windows in January 2027. Go in knowing that the token bill you are replacing may be smaller than the fleet you are about to run. Data that cannot leave the device, or work that has to happen offline, are the stronger reasons to do this.
| Option | What you get | Silicon it reaches today | Can you pin the model? | Verdict |
|---|---|---|---|---|
| llama.cpp + your own GGUF | MIT runtime, any GGUF model you choose | CPU, Apple Metal, CUDA, Vulkan, Snapdragon Hexagon; Intel NPU via OpenVINO still "in progress" | Yes, you ship the file | Default pick |
| Foundry Local (ONNX Runtime GenAI) | Microsoft's local runtime and catalog, on an MIT engine | CPU, CUDA, DirectML, QNN, OpenVINO, WebGPU execution providers | Yes, by model ID; aliases move | Pick for Windows NPU battery life |
| Apple Foundation Models | Free ~3B model built into Apple Intelligence | Apple silicon, OS-managed | No, it updates with the OS | Mac-only features you never need to reproduce |
| Windows AI APIs (Phi Silica) | Built-in model on Copilot+ PCs | NPU on Copilot+ PCs; GPU in experimental builds | No, and it is being replaced | Loser: do not start here |
| LiteRT-LM (MediaPipe successor) | Google's Apache 2.0 edge runtime | GPU and NPU on Android, iOS, web and desktop | Yes | Mobile apps, not laptop fleets |
| ExecuTorch | PyTorch's on-device runtime, 1.0 | XNNPACK on desktop CPU; Qualcomm NPU on Android | Yes | Mobile apps you build in PyTorch |
| Gemini 2.5 Flash-Lite (cloud baseline) | Per-token API | Google's datacentre | Until Google retires it | The number to beat: about $48K a year at our workload |
Prices and docs checked October 12, 2026.
The Workload, and What the Cloud Meter Actually Costs
The case for local inference usually starts with the token bill, so price it first. Our reference workload is a low-stakes task on 20,000 laptops: 40 requests per user per working day, 1,500 input tokens and 200 output tokens each, 260 working days a year. That is 800,000 requests a day, 1.2 billion input tokens and 160 million output tokens.
At Gemini 2.5 Flash-Lite's published rate of $0.10 per million input tokens and $0.40 per million output tokens (checked October 12, 2026), that is $184 a day, or about $47,840 a year for the whole fleet. Google's deprecations page lists no shutdown date for 2.5 Flash-Lite, but it limits access to customers who already used the 2.5 models and points new projects at 3.5 Flash-Lite. At 3.5 Flash-Lite's $0.30 input and $2.50 output, the same workload costs $760 a day, about $197,600 a year. Our Gemini Flash tool page tracks the family.
That range is the entire prize, before you pay anyone to run a model fleet, and before you subtract the share of requests that still escalate to a cloud model. A team that expects on-device inference to "end the meter" on a cheap model is chasing at most a few hundred thousand dollars a year, and the operating cost below can eat it.
The cases where on-device clearly wins are the ones the meter does not price:
- Data that cannot leave the device. Prompts and documents stay on the laptop, so the cloud model provider is not a processor for the local share. Your DPA conversation shrinks to the escalated share, and the model licence and your own on-device logs come under scrutiny instead.
- Offline or degraded-network work: field staff, travel, secure rooms.
- A task you would otherwise send to a mid-tier model, where per-token prices are many times Flash-Lite's.
Which Small Models Are Safe to Ship?
Ship a model with a permissive licence and an official quantised build from the model's own publisher. A community quantisation from an anonymous uploader is a dependency nobody owns: if it is subtly broken, no one is obliged to fix it, and you cannot prove which weights you shipped. That rules out more than you would expect.
| Model | Licence | Size | Context | Official quantised build? |
|---|---|---|---|---|
| Qwen3-4B | Apache 2.0 | 4B | 32,768 native | Yes: Qwen-published GGUF, Q4_K_M is 2.5 GB |
| Phi-4-mini-instruct | MIT | 3.8B | 128K | Yes: Microsoft's int4 ONNX builds for CPU and GPU |
| Gemma 3 4B | Gemma terms | 4B | 128K | Yes: Google's QAT Q4_0 GGUF, 3.16 GB |
| Gemma 4 E4B | Apache 2.0 | 4.5B effective, 8B with embeddings | 128K | Not stated on Google's model card; 386 community quantisations |
| Llama 3.2 3B | Llama 3.2 Community License | 3B | 8K for the quantised builds | Yes: Meta's SpinQuant and QLoRA builds |
Qwen3-4B is the default for a llama.cpp fleet: Apache 2.0, and the publisher ships the GGUF files you would otherwise download from a stranger. Phi-4-mini is the default for a Foundry Local fleet, because Microsoft publishes the ONNX builds and the Foundry Local quickstart uses it as its reference model. Gemma 3's QAT build is quantisation-aware trained by Google, so it should lose less accuracy at 4-bit than a build quantised after training, but the Gemma terms carry a use policy your legal team has to read. Gemma 4 moved to Apache 2.0 (our take on what that changed), and its model card does not say which quantisation is Google's own. Llama 3.2 carries a custom licence with a 700-million-MAU clause and an EU carve-out for its multimodal models; the 3B text model is outside that carve-out, but you will still spend a legal review on it that the Apache models do not need.
Who should not ship any of these: a team that cannot run its own evaluation on the target task. A 4B model at 4-bit is a different model from the benchmark card's BF16 numbers, and our quantisation format comparison shows how much the format alone moves results.
The Runtime Decides More Than the Model
The runtime determines which silicon you actually use. Each option below is judged on the same question: can you pin it, and what does it reach on a standard corporate laptop?
llama.cpp: The Default
llama.cpp is MIT-licensed C++ that runs GGUF models on CPU, Apple Metal, NVIDIA CUDA, AMD HIP, Vulkan and a Snapdragon Hexagon backend. Its README lists Intel NPU support through OpenVINO as "In Progress". You choose the model file, you ship it with your own software distribution tool, and the hash on disk is the hash you tested. Nothing about it changes unless you push a change.
Do not pick it if battery life on an Intel or AMD NPU is the requirement: on those chips llama.cpp runs on the CPU or GPU. Do not pick it if nobody on your team will own a fast-moving native dependency through its security updates. Tools built on it inherit the gap. Ollama's request for AMD Ryzen NPU support has been open since June 2024, and a maintainer folded a duplicate into it in July 2026.
Foundry Local: The Windows NPU Path
Foundry Local is Microsoft's local runtime, built on ONNX Runtime GenAI (MIT), whose execution providers cover CPU, CUDA, DirectML, QNN for Qualcomm, OpenVINO for Intel, and WebGPU. It needs Windows 11 24H2 or later, and it downloads the right hardware variant per device: per Microsoft's docs, a QNN NPU variant on Snapdragon, CUDA on NVIDIA, CPU otherwise. Microsoft's Foundry team added Intel and AMD NPU support in v0.7, and AMD has documented phi-4-mini running on a Ryzen AI NPU through Windows ML. It ties into the wider Microsoft Foundry catalog.
The pinning catch: the quickstart warns that "Foundry Local catalog aliases and recommended models can change as the catalog evolves." Pin the full model ID and cache the files yourself. The deeper catch is in the Copilot+ PC developer guide: Windows ML execution providers such as QNN and OpenVINO "are now bundled with Windows or delivered via Windows Update." You can pin the model, but on the NPU path you do not fully control the runtime layer underneath it. Do not pick Foundry Local for a Mac-heavy fleet, or if your change board needs every inference-path component under its own release control.
Apple Foundation Models: Free but Not Yours to Pin
Apple's framework gives Swift apps the on-device model behind Apple Intelligence. Apple describes it as an approximately 3-billion-parameter model compressed to 2 bits per weight, with inference free of charge. Its context window is 4,096 tokens per session, shared between prompt and response, which our reference workload fits but a long document does not.
The fleet problem is version control. Apple's research post says custom adapters must be retrained for each new base model version, and the adapter toolkit page says version 26.0.0 is the last release and is not compatible with OS 27. The model you tested in March is not guaranteed to be the model a user has after an OS update. Your MDM settings also decide whether it runs at all: Apple Intelligence can be restricted per device, and Addigy's guidance notes Apple spread new controls across several payloads from 26.4. We have covered how confidently this model is wrong on audit-style tasks.
Pick it for a Mac-only convenience feature where output can drift between OS versions without anyone filing a ticket. Do not pick it for anything you must reproduce, audit, or run on Windows.
Windows AI APIs and Phi Silica: The Loser
Phi Silica is the model Windows ships on Copilot+ PCs, the class Microsoft defines by an NPU of more than 40 TOPS. It is the cheapest-looking option on paper, because the model is already on the laptop. It loses on every fleet criterion. Per Microsoft's own page, the APIs are a Limited Access Feature that needs an unlock token, they are not available in China, and Phi Silica is being replaced by Aion Instruct: Insider rollout in November 2026, then "January 2027: Aion Instruct rolls out to retail devices and Phi Silica is removed." Any LoRA you trained has to be retrained. Running it on a GPU instead needs Developer Mode, an Insider build and a driver from the GPU maker, because "default drivers from Windows Update or OEM installations may not be sufficient."
Nobody should start a fleet project on Phi Silica in October 2026. Re-evaluate Aion Instruct when it reaches retail, and expect the same version-control problem Apple has.
LiteRT-LM and ExecuTorch: Right Tools, Wrong Device
Google's MediaPipe LLM Inference API is now in maintenance-only mode, and Google recommends migrating to LiteRT-LM, which is Apache 2.0 and targets Android, iOS, web, desktop and IoT with GPU and NPU acceleration. Its Swift and JavaScript APIs are still early preview. PyTorch's ExecuTorch 1.0 lists XNNPACK as its CPU backend on all platforms and a Qualcomm NPU backend for Android SoCs.
Both are strong choices for a mobile app your team builds. Neither is the shortest path to a managed Windows and macOS laptop fleet, and starting a new project on MediaPipe LLM Inference today means planning a migration on day one.
Fleet Management Is the Cost Nobody Models
Distributing, pinning and supporting a free model on 20,000 machines is where the money goes. Budget for four things:
- Distribution and caching. A Qwen3-4B Q4_K_M file is 2.5 GB. One version pushed to 20,000 laptops is 50 TB over your network, and every model update repeats it. Microsoft's Phi Silica docs tell developers to ask for the user's consent before a multi-gigabyte model download, and the GPU build of Phi Silica arrives "in the background through Windows Update."
- Version pinning and rollback. Keep two versions on disk (the current one plus the last known good), roll forward in rings the way you do OS patches, and record the model hash and runtime version on every request.
- Disk and RAM budget. A 4B model at 4-bit needs roughly its file size in memory plus the KV cache, on a laptop already running Teams, a browser and your security agents. Put it in the standard image spec, or users will find the out-of-memory case for you.
- Battery and thermals. This is where the NPU earns its keep. A June 2026 preprint profiling a Snapdragon X Elite Dell XPS 13 measured 4.0x less system energy on the NPU than on the CPU for a 120-query retrieval-augmented generation workload, while the integrated GPU used 6.5x more energy than the NPU. That benefit only exists if your runtime reaches the NPU on that exact chip.
You Lose the Gateway, So Decide the Telemetry Now
On-device inference removes the gateway, and with it your default record of what the model did. In the cloud, every request crosses a proxy you can log. On a laptop, nothing is recorded unless the client is built to record it, a gap we saw when Meta's agent model fit on a laptop and nothing logged it.
Decide before the pilot what each request emits: model ID and file hash, runtime version, prompt template version, the route decision (local or escalated), latency, token counts and a task outcome signal. Prompt and response content stays off by default, because shipping it home rebuilds the data flow you went local to avoid. On Windows, ONNX Runtime already emits ETW events that the Windows Performance Toolkit can read for load time and per-inference latency, which helps a pilot but is no substitute for fleet telemetry.
If you skip this, you cannot answer the first audit question about the feature: which model produced this output, and was it the one you approved?
The Routing Rule and the Escalation Rate
The escalation rate is the number that decides whether on-device saved anything. A routing rule sends each request either to the local model or to a frontier model in the cloud. Every escalated request is billed at the cloud rate, so at a 30% escalation rate the most you can save is 70% of the baseline above.
Router research gives a sense of the range. LMSYS's RouteLLM reported cutting cost by over 85% on MT Bench while keeping 95% of GPT-4's performance, but only 45% on MMLU and 35% on GSM8K. How much you save depends on the task, which is why you measure your own rate in the pilot instead of quoting theirs. Start with a simple rule you can explain to an auditor: input length over the local context budget, a confidence or schema-validation failure, or a task class you have reserved for the cloud. We covered a vendor's auto router that failed 4x as often as Opus on its own test, so do not assume a learned router beats a rule.
What the NPU Badge Does Not Tell You
A TOPS figure and a Copilot+ badge say nothing about which runtimes, models and quantisations work on your exact SKU and driver. The evidence is all in the vendors' own documentation. Phi Silica's GPU path depends on a GPU-maker driver that OEM updates can overwrite. Foundry Local picks a different model variant per device. llama.cpp's Intel NPU support is still in progress, and Ollama has had an open request for AMD NPU support since mid-2024.
So pilot on the real hardware before the refresh order. Test a matrix of each candidate laptop SKU, its shipping driver, your chosen runtime and your chosen quantised model, and record tokens per second, energy on battery and task accuracy. If the NPU path fails on a SKU, you want to know while you can still change the order.
How to Decide
Four questions predict regret better than any benchmark.
- Can you name the model version on every laptop? If not, you cannot audit it. That rules out the OS-provided models for anything regulated.
- Is the reason data or offline use, or is it the token bill? If it is only the bill and your cloud model is Flash-Lite class, the fleet cost probably exceeds the savings. Re-run the math with your own volumes.
- What is the measured escalation rate? Above roughly half, you are running two systems to save less than half the bill.
- Is the fleet mostly Windows on Copilot+ silicon? Then Foundry Local on the NPU is worth the Windows Update dependency for battery life. Mixed or Mac-heavy: llama.cpp everywhere.
What changes the answer: a cloud price rise on your baseline model, a regulator or customer contract that forbids the data flow, or Aion Instruct shipping with a version-pinning story that Phi Silica never had.
What to Do Next
This Week:
- Pull last month's token logs for the candidate task and compute the real annual bill at your current model's price. If it is under the Flash-Lite range above, write down the non-cost reason for going local or stop here.
- Ask your endpoint team which laptop SKUs, drivers and OS builds are in the current image and the next refresh order.
This Month:
- Run Qwen3-4B on llama.cpp and Phi-4-mini on Foundry Local against 500 real requests from the task, with your evaluation, on three representative SKUs.
- Define the telemetry schema and the routing rule, and measure the escalation rate.
Before the Refresh Order:
- Put the model's disk and RAM budget into the standard image spec.
- Make the NPU test matrix a purchase requirement: no SKU goes on the order until your runtime and model pass on its shipping driver.
The Bottom Line
Desktop teams have done this before with antivirus signatures and browser runtimes: a large, versioned artifact pushed to every machine, rolled out in rings, with a rollback plan. A local model is that kind of artifact, and it needs the same release discipline. The OS-provided models skip that discipline by design, which is why they fail a fleet test. Start with Qwen3-4B on llama.cpp, add Foundry Local on the Windows SKUs that pass the NPU pilot, and make the escalation rate the first number you report.
Continue Reading
- Meta's Agent Model Fits on a Laptop. Nothing Logs It.
- Apple's On-Device Model Is 99% Sure. Sample It Five Times.
- FP8 vs NVFP4 vs INT4-AWQ: AWQ Outran NVFP4 on 3 of 4 Models
- Google Gemma 4: Why Apache 2.0 Changes Enterprise AI
- Cloudflare's Auto Router Fails 4x as Often as Opus on Its Own Test
- Best FinOps Tools for AI Spend: No Dashboard Fixes an Untagged Key