llama.cpp — LLM Inference in C/C++

by ggml.ai

IntermediateDocumentationFree~2-3 hours to build and serve your first quantized model; days to work the full docs tree

Build, quantize and serve LLMs on almost any hardware from a single dependency-free binary.

Start LearningAdded Aug 8, 2026 · Updated Aug 8, 2026

Overview

llama.cpp (123k GitHub stars, 21.4k forks, over 10,300 commits) is the C/C++ inference engine built on the GGML tensor library, with no external dependencies for core functionality. Its reach is the point: hardware acceleration spans NVIDIA CUDA, Apple Metal, AMD HIP, Intel SYCL and OpenVINO, Moore Threads MUSA, Ascend NPU CANN, plus Vulkan and WebGPU for cross-platform GPU use and hand-tuned CPU paths for AVX, ARM NEON and RISC-V. It supports 1.5-, 2-, 3-, 4-, 5-, 6- and 8-bit integer quantization and CPU+GPU hybrid inference, which is what lets a large model run on a machine that cannot hold it in VRAM. The docs/ tree is the real curriculum: build.md and install.md for the toolchain, docker.md, multi-gpu.md, android.md and xcframework.md for deployment targets, build-riscv64-spacemit.md and build-s390x.md for exotic ones, then feature guides — completions.md, function-calling.md, llguidance.md for grammar-constrained output, speculative.md for speculative decoding, multimodal.md, models.md, preset.md, autoparser.md and ops.md — alongside backend/, development/, multimodal/ and ops/ subdirectories. Multimodal support covers image, audio and video through llama-mtmd-cli and llama-server, with pre-quantized Gemma, SmolVLM, Qwen2-VL, InternVL, Ultravox, Voxtral and Qwen2.5 Omni loadable via a single -hf flag or an explicit -m plus --mmproj pair. The shipped tools are llama-cli, llama-server (OpenAI-compatible), and quantization and benchmarking utilities.

At a Glance

Topic: Frameworks
Level: Intermediate
Format: Documentation
Cost: Free
Duration: ~2-3 hours to build and serve your first quantized model; days to work the full docs tree
Provider: ggml.ai
Hands-on: Yes — code/exercises
Certificate: None

What You’ll Learn

✓Build llama.cpp from source with the right backend flags for your accelerator
✓Quantize a model to GGUF and choose a bit-width against a memory budget
✓Serve an OpenAI-compatible HTTP endpoint with llama-server and its presets
✓Split a model across multiple GPUs and offload layers in CPU+GPU hybrid inference
✓Constrain output with grammars and llguidance for reliable structured responses
✓Enable function calling and parse tool-call output from local models
✓Run vision and audio models through llama-mtmd-cli with a multimodal projector
✓Cut latency with speculative decoding using a small draft model
✓Deploy to Android, Docker or an Apple xcframework from the same codebase

Highlights

•Widest hardware coverage of any inference engine — CUDA, Metal, HIP, SYCL, Vulkan, WebGPU, RISC-V, s390x
•Zero-dependency single binary, so it runs where a Python + CUDA stack cannot be installed
•The engine underneath Ollama, LM Studio and most consumer local-LLM tooling — learning it explains those
•Independent benchmarking on an M2 Ultra found it highly efficient for lightweight single-stream inference
•Quantization down to 1.5-bit is documented and supported, not a research branch
•Multimodal image, audio and video inference works from the same server binary

Who It’s For

Best For

✓Engineers deploying LLMs on-device, at the edge, or into air-gapped environments
✓Teams that need inference on non-NVIDIA hardware and cannot use a CUDA-only stack
✓Anyone debugging or tuning Ollama and LM Studio, which sit on top of this engine
✓Developers who want an OpenAI-compatible local endpoint without a Python runtime

Prerequisites

•Comfortable on the command line and building C/C++ projects with CMake
•Basic understanding of quantization, GGUF files and VRAM budgeting
•Knowledge of your own accelerator and its driver/toolkit setup (CUDA, ROCm, Metal)

FAQ

What is llama.cpp — LLM Inference in C/C++?

The repository and documentation for llama.cpp, the C/C++ inference engine on top of the GGML tensor library that most of the local-AI ecosystem is built on. It is for engineers who need LLM inference on hardware a Python stack cannot reach — a laptop CPU, an Android phone, an Intel or AMD GPU, an air-gapped box — and who want to own every layer. Working through it you can build the toolchain, quantize a model to GGUF, and stand up an OpenAI-compatible server.

Is llama.cpp — LLM Inference in C/C++ free?

llama.cpp — LLM Inference in C/C++ is free to access.

What level is llama.cpp — LLM Inference in C/C++ for?

llama.cpp — LLM Inference in C/C++ is aimed at a intermediate audience. Recommended background: Comfortable on the command line and building C/C++ projects with CMake, Basic understanding of quantization, GGUF files and VRAM budgeting, Knowledge of your own accelerator and its driver/toolkit setup (CUDA, ROCm, Metal).

How long does llama.cpp — LLM Inference in C/C++ take?

Expect roughly ~2-3 hours to build and serve your first quantized model; days to work the full docs tree. Most learners work through it at their own pace.

What will I learn from llama.cpp — LLM Inference in C/C++?

You'll learn: Build llama.cpp from source with the right backend flags for your accelerator; Quantize a model to GGUF and choose a bit-width against a memory budget; Serve an OpenAI-compatible HTTP endpoint with llama-server and its presets; Split a model across multiple GPUs and offload layers in CPU+GPU hybrid inference; Constrain output with grammars and llguidance for reliable structured responses; Enable function calling and parse tool-call output from local models; Run vision and audio models through llama-mtmd-cli with a multimodal projector; Cut latency with speculative decoding using a small draft model; Deploy to Android, Docker or an Apple xcframework from the same codebase.

Topics

llama-cpplocal-inferencequantizationggufedge-aiggml

Sources

This page was written from 4 sources, 1 on domains other than github.com.