The Definitive Fine-Tuning Guide for 3D Generation Models
by Hugging Face
LoRA fine-tune Microsoft's TRELLIS.2 image-to-3D model on your own meshes, with the full pipeline, configs and held-out metrics.
Overview
The Definitive Fine-Tuning Guide for 3D Generation Models was published on the Hugging Face blog on November 12, 2026 by Suvaditya Mukherjee, Merve and Pedro Cuenca. It opens by explaining how TRELLIS.2 works stage by stage: DINOv3-L encodes the input image, a dense sparse-structure flow transformer generates a 16-cubed occupancy latent, O-Voxel structured latents feed a sparse shape-flow transformer for geometry, and a texture flow plus VAE decoders produce materials and the final mesh. A planning section then argues which components to train. The authors put rank 8 LoRA adapters on attention and MLP layers of the sparse-structure transformer and rank 16 adapters on attention in the shape transformer (about 9.3M trainable parameters), and they freeze the texture stage because their data has baked lighting and no separate material maps. The walkthrough runs on Hugging Face Jobs with prebuilt Docker images (trellis2-prep and trellis2-train) or locally with Docker, renders 16 Blender views per object headlessly, and trains with PEFT's inject_adapter_in_model, bf16 autocast, gradient checkpointing, AdamW and rectified flow matching with classifier-free guidance dropout, logging to Trackio. The worked example uses a 188-object footwear subset of Google Scanned Objects (179 train, 9 held out). On the held-out shoes, mean F-score at 0.02 rose from 0.276 to 0.803 and Chamfer distance fell from 0.174 to 0.033. The companion MIT-licensed repo holds the scripts.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- ~1 hour read; a full reproduction takes ~12 GPU-hours on one A100 80 GB (about 1h preprocessing, 7.2h structure LoRA, 3.6h shape LoRA)
- Provider
- Hugging Face
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓How TRELLIS.2 splits image-to-3D generation into structure, shape and material stages
- ✓How to choose which transformer stages to fine-tune and which to freeze for your data
- ✓How to preprocess your own meshes into structured latents with headless Blender rendering
- ✓How to inject LoRA adapters with PEFT when a model's custom forward pass breaks PeftModel
- ✓How to train sparse-structure and shape adapters with rectified flow matching and CFG dropout
- ✓How to run preprocessing and training as Hugging Face Jobs or in local Docker containers
- ✓How to evaluate generated geometry with F-score, Chamfer distance and normal consistency
Highlights
- •Reports held-out geometry metrics: F-score at 0.02 from 0.276 to 0.803 on 9 unseen shoes, better on all nine
- •Gives real compute numbers: about 12 hours on one A100 80 GB for preprocessing plus both adapter runs
- •Ships an MIT-licensed companion repo with preprocess.py, finetune.py, Dockerfiles and an eval script, plus released adapters and latents on the Hub
- •States its own limits: geometry only, textures come from the base model, and the eval set is just nine objects
Who It’s For
Best For
- ✓ML engineers adapting an open 3D generator to one product category such as footwear or furniture
- ✓Practitioners who know LoRA on LLMs or image diffusion and want to apply it to a multi-stage 3D pipeline
- ✓Teams evaluating whether fine-tuning beats prompting a base image-to-3D model for their catalog
Prerequisites
- •Working knowledge of PyTorch training loops, LoRA/PEFT and diffusion or flow-matching objectives
- •A Hugging Face account with a write token and access to the gated DINOv3 model
- •An NVIDIA GPU with 24 GB+ and Docker with the NVIDIA runtime for local runs, or Hugging Face Jobs credit for A100 runs
- •Your own 3D meshes in a format Blender can import, uploaded to the Hub
FAQ
What is The Definitive Fine-Tuning Guide for 3D Generation Models?
A Hugging Face guide to LoRA fine-tuning TRELLIS.2, an open image-to-3D generation model, on a custom mesh dataset. It is written for ML engineers who already know PEFT and diffusion or flow-matching training and want to adapt a 3D generator to one product category. You finish with trained adapters, a preprocessing pipeline and a held-out geometry evaluation.
Is The Definitive Fine-Tuning Guide for 3D Generation Models free?
The Definitive Fine-Tuning Guide for 3D Generation Models is free to access.
What level is The Definitive Fine-Tuning Guide for 3D Generation Models for?
The Definitive Fine-Tuning Guide for 3D Generation Models is aimed at a advanced audience. Recommended background: Working knowledge of PyTorch training loops, LoRA/PEFT and diffusion or flow-matching objectives, A Hugging Face account with a write token and access to the gated DINOv3 model, An NVIDIA GPU with 24 GB+ and Docker with the NVIDIA runtime for local runs, or Hugging Face Jobs credit for A100 runs, Your own 3D meshes in a format Blender can import, uploaded to the Hub.
How long does The Definitive Fine-Tuning Guide for 3D Generation Models take?
Expect roughly ~1 hour read; a full reproduction takes ~12 GPU-hours on one A100 80 GB (about 1h preprocessing, 7.2h structure LoRA, 3.6h shape LoRA). Most learners work through it at their own pace.
What will I learn from The Definitive Fine-Tuning Guide for 3D Generation Models?
You'll learn: How TRELLIS.2 splits image-to-3D generation into structure, shape and material stages; How to choose which transformer stages to fine-tune and which to freeze for your data; How to preprocess your own meshes into structured latents with headless Blender rendering; How to inject LoRA adapters with PEFT when a model's custom forward pass breaks PeftModel; How to train sparse-structure and shape adapters with rectified flow matching and CFG dropout; How to run preprocessing and training as Hugging Face Jobs or in local Docker containers; How to evaluate generated geometry with F-score, Chamfer distance and normal consistency.
Topics
Sources
This page was written from 2 sources, 1 on domains other than huggingface.co.