Why your fine-tuning job takes longer than the FLOPs say
by Dharma-AI (Hugging Face blog)
Data from 272 production SFT runs showing why FLOP-based fine-tuning time estimates miss, and a config-only model that predicts wall-clock time.
Overview
This Dharma-AI team article on the Hugging Face blog, bylined to Bruno Duarte and published November 18, 2026, previews the paper "Predicting End-to-End Fine-Tuning Time for Generative Language Models", with the full version promised for arXiv. It starts from the standard compute-only estimate, T = 6PD / (N_GPU x FLOPS_peak x MFU), and argues that it ignores setup, model loading, tokenization, evaluation, checkpoint uploads and I/O contention. The data section covers 272 supervised fine-tuning runs (about 7,382 GPU-hours) on a shared pool of NVIDIA H200s. Jobs used 1, 2 or 4 GPUs, models went up to 15B parameters with LoRA and full fine-tuning, the stack was Hugging Face Transformers and Unsloth, and runs lasted from 1.6 to 172.4 hours. The results table compares FLOP baselines at 30%, 45% and 60% MFU, an extended Calculon analytical model, linear and ridge regression, XGBoost and CatBoost under stratified 5-fold cross-validation. The best FLOP setting reaches R² 0.31 with a 12.41-hour MAE, while CatBoost on 24 configuration features reaches R² 0.98 with a 1.71-hour MAE. Later sections cover SHAP feature importance (evaluation set size ranks first, which the authors call correlational rather than causal), error by duration band (accuracy degrades past about 60 hours, where data is sparse) and limitations: SFT only, one GPU type, few long runs. No code or dataset is released yet.
At a Glance
- Topic
- Fine-Tuning
- Level
- Advanced
- Format
- Guide
- Cost
- Free
- Duration
- ~12 min read
- Provider
- Dharma-AI (Hugging Face blog)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Why the 6PD FLOP formula for training time leaves out setup, evaluation, checkpointing and I/O overhead
- ✓How much FLOP-based estimates miss on real jobs, with R² between 0.01 and 0.31 across MFU settings
- ✓Which job configuration features to log so wall-clock time can be predicted before dispatch
- ✓How gradient-boosted regressors like CatBoost and XGBoost compare with linear models and analytical simulators for this task
- ✓Why evaluation set size and checkpoint saving rank among the strongest predictors of fine-tuning duration
- ✓Where config-based prediction breaks down, including runs longer than 60 hours and hardware other than H200
Highlights
- •Built on 272 production runs and about 7,382 GPU-hours rather than synthetic benchmarks
- •Shows full cross-validated tables, including baselines that fail badly, such as Calculon with a negative R²
- •Reports error by duration band and states plainly that runs past 60 hours are poorly covered
- •Turns into a practical checklist: count evaluation and checkpoint costs in any GPU budget instead of quoting FLOPs alone
Who It’s For
Best For
- ✓ML platform engineers who schedule fine-tuning jobs on shared GPU clusters
- ✓Teams that budget or quote GPU-hours for LoRA and full fine-tuning work
- ✓Practitioners who use FLOP or MFU estimates and want to know how far off they run
Prerequisites
- •Working knowledge of supervised fine-tuning, LoRA versus full fine-tuning, and GPU throughput concepts like MFU
- •Basic regression and evaluation metrics (MAE, RMSE, R², cross-validation) and some familiarity with gradient-boosted trees and SHAP
FAQ
What is Why your fine-tuning job takes longer than the FLOPs say?
A Hugging Face team article from Dharma-AI, published November 18, 2026, that previews a paper on predicting end-to-end fine-tuning time. Using 272 supervised fine-tuning runs on shared H200 clusters, it shows why the standard FLOP formula misestimates wall-clock time. It is for ML engineers and platform teams who schedule, budget or quote fine-tuning jobs.
Is Why your fine-tuning job takes longer than the FLOPs say free?
Why your fine-tuning job takes longer than the FLOPs say is free to access.
What level is Why your fine-tuning job takes longer than the FLOPs say for?
Why your fine-tuning job takes longer than the FLOPs say is aimed at a advanced audience. Recommended background: Working knowledge of supervised fine-tuning, LoRA versus full fine-tuning, and GPU throughput concepts like MFU, Basic regression and evaluation metrics (MAE, RMSE, R², cross-validation) and some familiarity with gradient-boosted trees and SHAP.
How long does Why your fine-tuning job takes longer than the FLOPs say take?
Expect roughly ~12 min read. Most learners work through it at their own pace.
What will I learn from Why your fine-tuning job takes longer than the FLOPs say?
You'll learn: Why the 6PD FLOP formula for training time leaves out setup, evaluation, checkpointing and I/O overhead; How much FLOP-based estimates miss on real jobs, with R² between 0.01 and 0.31 across MFU settings; Which job configuration features to log so wall-clock time can be predicted before dispatch; How gradient-boosted regressors like CatBoost and XGBoost compare with linear models and analytical simulators for this task; Why evaluation set size and checkpoint saving rank among the strongest predictors of fine-tuning duration; Where config-based prediction breaks down, including runs longer than 60 hours and hardware other than H200.
Topics
Sources
This page was written from 2 sources, 1 on domains other than huggingface.co.