Vision Language Models: Building VLMs with Hugging Face

Written by four Hugging Face practitioners — Merve Noyan, Andrés Marafioti, Miquel Farré and Orr Zohar — and published by O'Reilly Media in 2026 at roughly 300 pages (ISBN 9798341624047), this is the engineering handbook for training, fine-tuning and shipping vision-language models rather than a survey of the literature. Its eleven chapters run in build order: Introduction to Vision and Language; Vision-Language Model Applications; Vision-Language Model Training; Training Data and Preprocessing for VLMs; Post-Training Vision-Language Models; Core Architectures of Vision-Language Models; Deploying Models for Inference at Scale; Document AI; Video-Language Models; Any-to-Any Models; and Advanced Topics and Cutting-Edge Research. The architectures chapter covers early versus late fusion, encoder-decoder designs and the multimodal attention variants that distinguish current models. Later chapters push past training into the parts most tutorials stop before: dataset curation and preprocessing, post-training and alignment, serving at scale, and the document-understanding and video pipelines that carry most real enterprise VLM work. Coverage spans image captioning and document understanding through to zero-shot inference, retrieval-augmented generation and agentic use. The stack is PyTorch plus the Hugging Face libraries, with NVIDIA CUDA underneath. Noyan is a machine learning engineer on Hugging Face's ML advocacy engineering team, and the author group overlaps with the SmolVLM line, so the material comes from people who trained the models it describes.

Topic

Models

Level

Intermediate

Format

Book

Cost

Paid

Duration

11 chapters, ~300 pages, self-paced

Provider

O'Reilly Media

Hands-on

Yes — code/exercises

Certificate

None

• ✓Compare early-fusion, late-fusion and encoder-decoder VLM architectures and their trade-offs
• ✓Curate and preprocess image-text training data, including chat-template formatting
• ✓Run supervised fine-tuning and post-training alignment on a vision-language model
• ✓Serve VLMs for inference at scale under real memory and throughput constraints
• ✓Build document-understanding pipelines with a VLM instead of a traditional OCR stack
• ✓Extend a VLM to video and to any-to-any multimodal input and output
• ✓Apply zero-shot inference and retrieval-augmented generation to multimodal problems

• •Written by Hugging Face engineers whose own work includes the SmolVLM model line
• •Covers deployment, document AI and video-language models — the chapters most VLM tutorials never reach
• •Published in 2026, so it reflects current architectures rather than 2023-era CLIP-and-BLIP material
• •Organised in whole-lifecycle order: applications, training, data, post-training, architecture, serving
• •Hugging Face's free smol-course carries a SmolVLM2 fine-tuning unit if you want to sample the approach first

This page was written from 3 sources, 2 on domains other than vlmbook.com.

1. 1.vlmbook.com — vlmbook.comvendor
2. 2.books.google.com — Vision Language Models
3. 3.huggingface.co — 1