LLaMA Factory Documentation — Unified Efficient Fine-Tuning of 100+ LLMs
by LLaMA Factory (hiyouga)
Fine-tune 100+ open LLMs and VLMs — LoRA, QLoRA, DPO, PPO, KTO — from a config file or a no-code web UI.
Overview
LLaMA Factory is 'an easy-to-use and efficient platform for training and fine-tuning large language models' that lets you fine-tune hundreds of pre-trained models locally without writing code, either through YAML configs and the CLI or through its built-in web UI, LLaMA Board. The documentation walks through setup and installation, data preparation and dataset formats, the LLaMA Board WebUI, the full range of training methods, and then inference and evaluation of the resulting checkpoints. Supported architectures span LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen, Yi, Gemma, Baichuan, ChatGLM, Phi, DeepSeek, GLM and more, including multimodal VLMs. The integrated training methods cover the whole post-training stack: (continuous) pre-training, (multimodal) supervised fine-tuning, reward modeling, PPO, DPO, KTO, and ORPO. On the efficiency side the docs cover 16-bit full-tuning, freeze-tuning, LoRA and 2/3/4/5/6/8-bit QLoRA, advanced optimizers and adapters such as GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LoRA+, LoftQ and PiSSA, and practical accelerations including FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune and rsLoRA — plus distributed training, quantization, multiple inference engines, and experiment monitoring. It is open source (hiyouga/LLaMA-Factory on GitHub), was published as an ACL 2024 demo paper, and runs on CUDA, AMD ROCm, and Ascend NPU via provided Docker images.
At a Glance
- Topic
- Fine-Tuning
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- Self-paced reference; first LoRA fine-tune in ~1-2 hours
- Provider
- LLaMA Factory (hiyouga)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Prepare and format datasets correctly for SFT, preference, and multimodal fine-tuning runs
- ✓Run LoRA and QLoRA (2-8 bit) fine-tunes on open models with limited GPU memory
- ✓Choose between full-parameter, freeze, LoRA, and quantized tuning for a given budget
- ✓Apply preference optimization — DPO, KTO, ORPO — and reward modeling plus PPO for RLHF-style training
- ✓Use the LLaMA Board web UI to configure and launch a fine-tune with no code
- ✓Speed up training with FlashAttention-2, Unsloth, Liger Kernel, and modern optimizers like GaLore and Muon
- ✓Serve and evaluate the fine-tuned checkpoint through the framework's inference engines
Highlights
- •One framework, 100+ supported models and VLMs — no per-model training script rewrite
- •Covers the entire post-training stack: pre-training, SFT, reward modeling, PPO, DPO, KTO, ORPO
- •LLaMA Board WebUI makes a first fine-tune genuinely no-code
- •Peer-reviewed: published as an ACL 2024 demo paper, and widely used in industry and research
Who It’s For
Best For
- ✓ML/AI engineers running their first serious LoRA or QLoRA fine-tune
- ✓Teams post-training open models (Qwen, Llama, Mistral, Gemma, DeepSeek) on domain data
- ✓Researchers who need DPO/KTO/ORPO and reward modeling without building the pipeline
- ✓Practitioners fine-tuning on constrained hardware or on AMD ROCm / Ascend NPU
Prerequisites
- •Python and PyTorch fundamentals
- •Access to a GPU (CUDA, ROCm, or Ascend NPU)
- •Conceptual understanding of supervised fine-tuning, LoRA, and quantization
FAQ
What is LLaMA Factory Documentation — Unified Efficient Fine-Tuning of 100+ LLMs?
The official documentation for LLaMA Factory, the ACL 2024 demo-track framework that unifies efficient fine-tuning across 100+ open language and vision-language models. It is for AI engineers who want to run SFT, LoRA/QLoRA, and preference-optimization runs without writing a training loop from scratch.
Is LLaMA Factory Documentation — Unified Efficient Fine-Tuning of 100+ LLMs free?
LLaMA Factory Documentation — Unified Efficient Fine-Tuning of 100+ LLMs is free to access.
What level is LLaMA Factory Documentation — Unified Efficient Fine-Tuning of 100+ LLMs for?
LLaMA Factory Documentation — Unified Efficient Fine-Tuning of 100+ LLMs is aimed at a intermediate audience. Recommended background: Python and PyTorch fundamentals, Access to a GPU (CUDA, ROCm, or Ascend NPU), Conceptual understanding of supervised fine-tuning, LoRA, and quantization.
How long does LLaMA Factory Documentation — Unified Efficient Fine-Tuning of 100+ LLMs take?
Expect roughly Self-paced reference; first LoRA fine-tune in ~1-2 hours. Most learners work through it at their own pace.
What will I learn from LLaMA Factory Documentation — Unified Efficient Fine-Tuning of 100+ LLMs?
You'll learn: Prepare and format datasets correctly for SFT, preference, and multimodal fine-tuning runs; Run LoRA and QLoRA (2-8 bit) fine-tunes on open models with limited GPU memory; Choose between full-parameter, freeze, LoRA, and quantized tuning for a given budget; Apply preference optimization — DPO, KTO, ORPO — and reward modeling plus PPO for RLHF-style training; Use the LLaMA Board web UI to configure and launch a fine-tune with no code; Speed up training with FlashAttention-2, Unsloth, Liger Kernel, and modern optimizers like GaLore and Muon; Serve and evaluate the fine-tuned checkpoint through the framework's inference engines.