Vision Language Models: Building VLMs with Hugging Face
by O'Reilly Media
The engineering handbook for training, fine-tuning, and shipping vision-language models with the Hugging Face stack.
Overview
Written by Merve Noyan, Miquel Farré, Andrés Marafioti, and Orr Zohar of Hugging Face, this book is framed by its authors as "the book we wished we had when multimodal work stopped being a research curiosity and became an engineering problem." Its eleven chapters run the full VLM lifecycle: vision-language applications, core architectures, data preprocessing and dataset preparation, model training and fine-tuning, deployment, document AI, video-language models, broader multimodal systems, and agentic vision systems. The stack is the current practitioner one — Hugging Face libraries, PyTorch, and CUDA — and the authors are explicit about method: "We lead with code and concrete examples, and we use theory to explain why things work (or don't) rather than to impress." Coverage spans image captioning and document understanding through zero-shot inference and retrieval-augmented generation over visual data. Published by O'Reilly Media; code and notebooks are hosted on Hugging Face. Available for purchase or through an O'Reilly Learning Platform subscription.
At a Glance
- Topic
- Models
- Level
- Intermediate
- Format
- Book
- Cost
- Paid
- Duration
- 11 chapters
- Provider
- O'Reilly Media
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓How VLM architectures fuse visual encoders with language models
- ✓Prepare and preprocess multimodal datasets for training
- ✓Select, fine-tune, and evaluate a VLM for a specific task
- ✓Deploy VLMs and handle inference trade-offs in production
- ✓Build document AI and video-language applications
- ✓Apply zero-shot inference and multimodal RAG over images and documents
Highlights
- •Written by the Hugging Face researchers behind widely used open VLMs
- •Full lifecycle: data, architecture, training, deployment, applications
- •Covers newer ground: document AI, video-language models, agentic vision systems
- •Companion code and notebooks published on Hugging Face
Who It’s For
Best For
- ✓ML engineers adding multimodal capability to an existing LLM stack
- ✓Builders shipping document understanding or video analysis features
- ✓Researchers who want a concrete engineering view of VLM architectures
Prerequisites
- •Python and PyTorch
- •Familiarity with transformers and LLM fine-tuning
- •Basic computer vision concepts
FAQ
What is Vision Language Models: Building VLMs with Hugging Face?
An O'Reilly book by four Hugging Face practitioners on building real-world vision-language models end to end. It is written for ML engineers, researchers, and builders who need multimodal systems in production rather than a survey of multimodal research.
Is Vision Language Models: Building VLMs with Hugging Face free?
Vision Language Models: Building VLMs with Hugging Face is a paid resource.
What level is Vision Language Models: Building VLMs with Hugging Face for?
Vision Language Models: Building VLMs with Hugging Face is aimed at a intermediate audience. Recommended background: Python and PyTorch, Familiarity with transformers and LLM fine-tuning, Basic computer vision concepts.
How long does Vision Language Models: Building VLMs with Hugging Face take?
Expect roughly 11 chapters. Most learners work through it at their own pace.
What will I learn from Vision Language Models: Building VLMs with Hugging Face?
You'll learn: How VLM architectures fuse visual encoders with language models; Prepare and preprocess multimodal datasets for training; Select, fine-tune, and evaluate a VLM for a specific task; Deploy VLMs and handle inference trade-offs in production; Build document AI and video-language applications; Apply zero-shot inference and multimodal RAG over images and documents.