Hugging Face: Fine-tune a Pretrained Model

This is the canonical fine-tuning tutorial in the Hugging Face Transformers documentation, and it was rewritten for the Transformers v5 line: the running example now fine-tunes Qwen/Qwen3-0.6B, a small causal language model, on the karthiksagarn/astro_horoscope dataset from the Hub, rather than the older BERT-plus-Yelp classification walkthrough that dozens of blog posts still reproduce. It moves through four sections — Tokenization, Loading a model, Training configuration, and Training. Tokenization covers loading a dataset with datasets.load_dataset, mapping an AutoTokenizer over the text column with truncation=True and an explicit max_length, using remove_columns to drop raw text the model's forward method will not accept, carving out an eval split with train_test_split, and using DataCollatorForLanguageModeling with mlm=False so each batch is padded dynamically to its own longest sequence instead of a global fixed length. Loading a model explains why dtype="auto" matters: without it PyTorch upcasts bfloat16 checkpoints to float32 and doubles memory. Training configuration is the densest part, covering num_train_epochs, per_device_train_batch_size, learning_rate, bf16 versus fp16 by GPU generation, gradient_accumulation_steps to simulate a larger effective batch, gradient_checkpointing to trade compute for memory, eval_strategy and save_strategy, logging_steps, and load_best_model_at_end. Training then assembles a Trainer with model, args, train and eval datasets, processing_class and data collator, calls train(), and uploads everything with push_to_hub(). It closes by pointing at the Trainer recipes, subclassing, callbacks and data-collator guides. It is maintained in the same repository as the library itself (huggingface/transformers, 163.5k stars, Apache-2.0, PyTorch 2.5+/JAX/TensorFlow, over 1M checkpoints on the Hub), so unlike a blog tutorial it does not silently rot against the current API.

Topic

Fine-Tuning

Level

Intermediate

Format

Documentation

Cost

Free

Duration

~1-2 hours to read and run end to end, self-paced

Provider

Hugging Face

Hands-on

Yes — code/exercises

Certificate

None

• ✓Tokenize a Hub dataset and drop the raw text columns the model's forward method rejects
• ✓Use DataCollatorForLanguageModeling with mlm=False for dynamic per-batch padding instead of fixed padding
• ✓Load a checkpoint with dtype="auto" so bfloat16 weights are not silently upcast and doubled in memory
• ✓Configure TrainingArguments: epochs, per-device batch size, learning rate and logging cadence
• ✓Fit a model into limited VRAM using gradient accumulation and gradient checkpointing
• ✓Choose bf16 on Ampere-or-newer GPUs and fall back to fp16 on older hardware
• ✓Evaluate and checkpoint every epoch, then reload the best checkpoint automatically at the end
• ✓Push fine-tuned weights, tokenizer and generation config to the Hugging Face Hub with push_to_hub()

• •Rewritten for Transformers v5 around a causal LLM (Qwen3-0.6B) — not the stale BERT/Yelp walkthrough still copied across the web
• •Every memory knob a single-GPU practitioner actually needs is on one page: bf16, gradient accumulation, gradient checkpointing
• •Lives in the transformers repo itself (163.5k stars, Apache-2.0), so it is updated with the API rather than after it
• •Short enough to finish in an afternoon, and the example genuinely runs on one consumer GPU at the settings given
• •Ends by routing you into Trainer recipes, callbacks, subclassing and custom collators instead of stopping at 'it trained'

This page was written from 2 sources, 1 on domains other than huggingface.co.

1. 1.huggingface.co — trainingvendor
2. 2.github.com — transformers