EdgeAI for Beginners
by Microsoft
Run, quantize and ship small language models on local hardware, from GGUF conversion to multi-agent apps on Foundry Local.
Overview
EdgeAI for Beginners is an open-source course from Microsoft, first published on GitHub on June 27, 2025 and still maintained in 2026: the repository received dependency and security fixes in August 2026 (including pinning RAGAS below a vulnerable range) and a push on October 10, 2026. It is organized as nine modules, 00 to 08. The first three cover what edge AI is, cloud versus edge trade-offs, and the main small model families (Phi, Qwen, Gemma, BitNET, Phi-Silica). Module 03 covers local and cloud deployment. Module 04, the most technical, is a model optimization toolkit: quantization levels from 1-bit to 8-bit, GGUF and ONNX formats, llama.cpp builds and Q2_K to Q8_0 quantization, Microsoft Olive, Intel OpenVINO with NNCF, Apple MLX with LoRA fine-tuning, Qualcomm QNN for Snapdragon NPUs, and a decision tree for picking a framework. Module 05 moves to SLMOps (distillation, fine-tuning, production deployment), Module 06 to function calling and the Model Context Protocol, Module 07 to the AI Toolkit and Windows, and Module 08 to ten Foundry Local sample apps, including a Chainlit RAG app, a models-as-tools router and a multi-agent system. A separate workshop adds Python sessions and Jupyter notebooks with RAGAS evaluation, and an agentic workshop builds a local multi-agent podcast pipeline on the Microsoft Agent Framework, Ollama and Qwen-3-8B. The repo has about 1.8k stars, 408 forks and translations into more than 50 languages.
At a Glance
- Topic
- Models
- Level
- All Levels
- Format
- Course
- Cost
- Free
- Duration
- ~36-45 hours across 9 modules (20-30 hour quick-start path), self-paced
- Provider
- Microsoft
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Compare cloud and edge inference on latency, cost, privacy and offline operation
- ✓Choose among Phi, Qwen, Gemma and BitNET small model families for a device
- ✓Convert models to GGUF and quantize them with llama.cpp from Q2_K to Q8_0
- ✓Run hardware-aware INT4 and INT8 optimization pipelines with Microsoft Olive and OpenVINO NNCF
- ✓Fine-tune with LoRA and run 4-bit models on Apple Silicon using MLX
- ✓Target Snapdragon NPUs with Qualcomm QNN using INT8, INT16 and mixed precision
- ✓Build local function-calling agents and connect them through the Model Context Protocol
- ✓Ship Foundry Local sample apps including a Chainlit RAG app and a model router
Highlights
- •Covers five optimization toolchains side by side (llama.cpp, Olive, OpenVINO, MLX, QNN) with a framework selection decision tree
- •Ten runnable Foundry Local sample applications plus a notebook workshop that evaluates a local RAG pipeline with RAGAS
- •Maintained in 2026: August 2026 commits patched a RAGAS vulnerability and pinned GitHub Actions to commit SHAs
- •MIT licensed, about 1.8k stars and 408 forks, translated into more than 50 languages
- •GitTrend's review notes gaps: no dependency installation guide, a large repo because of translations, and few real-device demos
Who It’s For
Best For
- ✓Developers who want to run LLMs locally on laptops, Macs or Windows PCs without a cloud API
- ✓Engineers building privacy-sensitive or offline AI features on edge and mobile hardware
- ✓Teams evaluating quantization toolchains before committing to one for production
- ✓Python developers moving from cloud LLM APIs to self-hosted small models
Prerequisites
- •Basic Python and comfort running Jupyter notebooks
- •A machine with at least 8GB RAM for the CPU-mode workshops; a GPU or NPU is optional
- •For Modules 05-08: familiarity with LLM APIs, function calling and command-line tooling
FAQ
What is EdgeAI for Beginners?
EdgeAI for Beginners is Microsoft's free, MIT-licensed GitHub course for developers who want to run small language models on their own laptops, phones and edge devices. After the nine modules you can quantize a model with llama.cpp, Olive, OpenVINO or MLX, serve it locally, and build RAG and multi-agent apps that never call a cloud API.
Is EdgeAI for Beginners free?
EdgeAI for Beginners is free to access.
What level is EdgeAI for Beginners for?
EdgeAI for Beginners is aimed at a all levels audience. Recommended background: Basic Python and comfort running Jupyter notebooks, A machine with at least 8GB RAM for the CPU-mode workshops; a GPU or NPU is optional, For Modules 05-08: familiarity with LLM APIs, function calling and command-line tooling.
How long does EdgeAI for Beginners take?
Expect roughly ~36-45 hours across 9 modules (20-30 hour quick-start path), self-paced. Most learners work through it at their own pace.
What will I learn from EdgeAI for Beginners?
You'll learn: Compare cloud and edge inference on latency, cost, privacy and offline operation; Choose among Phi, Qwen, Gemma and BitNET small model families for a device; Convert models to GGUF and quantize them with llama.cpp from Q2_K to Q8_0; Run hardware-aware INT4 and INT8 optimization pipelines with Microsoft Olive and OpenVINO NNCF; Fine-tune with LoRA and run 4-bit models on Apple Silicon using MLX; Target Snapdragon NPUs with Qualcomm QNN using INT8, INT16 and mixed precision; Build local function-calling agents and connect them through the Model Context Protocol; Ship Foundry Local sample apps including a Chainlit RAG app and a model router.
Topics
Sources
This page was written from 5 sources, 2 on domains other than github.com.