NVIDIA NeMo Curator Documentation - Scalable Dataset Curation for LLM Training
by NVIDIA
The GPU-accelerated pipeline NVIDIA uses to clean, dedupe and synthesize training data at trillion-token scale.
Overview
NVIDIA NeMo Curator is an Apache-2.0 data curation platform for building the datasets that pretraining and post-training runs consume, and the documentation is organised by modality - text, image, video and audio - each with its own quickstart, workflow guide and beginner tutorial. The text track is the deepest: loader stages for existing JSONL and Parquet files, Common Crawl, arXiv, Wikipedia and custom download pipelines; cleaning, normalization and formatting; language identification, stopword handling and multilingual translation; heuristic and classifier-based quality filtering; and three deduplication modes - exact, fuzzy and GPU-accelerated semantic. A synthetic data generation section drives any OpenAI-compatible endpoint, including NVIDIA NIM or a locally hosted vLLM server, for data augmentation, multilingual question-answer generation, knowledge extraction and paraphrasing existing text into higher-quality prose. Image curation covers embedding generation, aesthetic and NSFW filtering and semantic deduplication; video covers scene detection, clip extraction and motion filtering; audio covers ASR transcription and word-error-rate quality filtering. Pipelines execute on Ray with RAPIDS/cuDF for GPU acceleration through a streaming, stage-based executor with auto-scaled replicas, and the docs include deployment guides for Slurm HPC clusters and Kubernetes. Installation is a single uv or pip command for CPU text pipelines, with a CUDA 12 override file for GPU work. The repository carries roughly 1,700 GitHub stars and 947 commits and ships on a versioned release train - version 26.04 landed in April 2026 with a simplified Resources API.
At a Glance
- Topic
- ML
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~3-5 hours for the per-modality quickstarts; ongoing reference for production pipeline work
- Provider
- NVIDIA
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Build streaming curation pipelines that run on Ray across multi-node GPU clusters
- ✓Download and extract Common Crawl, arXiv and Wikipedia dumps into usable training corpora
- ✓Apply heuristic and classifier-based quality filters to score and drop low-value documents
- ✓Run exact, fuzzy and GPU-accelerated semantic deduplication over very large text datasets
- ✓Generate synthetic training data against any OpenAI-compatible endpoint including local vLLM
- ✓Curate image, video and audio datasets with NSFW, motion and word-error-rate filters
- ✓Deploy curation jobs onto Slurm HPC clusters or Kubernetes using the provided guides
Highlights
- •Apache-2.0 and the same toolkit NVIDIA documents as powering its Nemotron dataset pipelines
- •Four modalities - text, image, video and audio - share one stage-based pipeline API instead of four separate tools
- •Ray plus RAPIDS/cuDF backends with a streaming executor NVIDIA reports running at over 99% GPU utilization
- •Versioned release train rather than a rolling main branch - 26.04 shipped in April 2026 with a simplified Resources API
- •Per-modality quickstarts let you run a small CPU-only text pipeline before committing a GPU cluster
Who It’s For
Best For
- ✓ML engineers assembling pretraining or post-training corpora at scale
- ✓Teams building domain-specific datasets for fine-tuning an open-weight model
- ✓Data engineers who need deduplication and quality filtering at billion-document scale
Prerequisites
- •Python plus working familiarity with distributed data processing concepts
- •For GPU pipelines: NVIDIA GPUs with the CUDA 12 toolkit on Linux x86_64
- •Comfort with JSONL and Parquet dataset formats and cluster job submission
FAQ
What is NVIDIA NeMo Curator Documentation - Scalable Dataset Curation for LLM Training?
Official documentation for NeMo Curator, NVIDIA's open-source toolkit for curating the datasets that pretraining and fine-tuning runs actually consume. It targets ML and data engineers who have outgrown ad-hoc pandas scripts. Working through it you will build Ray-backed streaming pipelines that download, clean, quality-filter, deduplicate and synthetically augment text, image, video and audio corpora across multi-node GPU clusters.
Is NVIDIA NeMo Curator Documentation - Scalable Dataset Curation for LLM Training free?
NVIDIA NeMo Curator Documentation - Scalable Dataset Curation for LLM Training is free to access.
What level is NVIDIA NeMo Curator Documentation - Scalable Dataset Curation for LLM Training for?
NVIDIA NeMo Curator Documentation - Scalable Dataset Curation for LLM Training is aimed at a advanced audience. Recommended background: Python plus working familiarity with distributed data processing concepts, For GPU pipelines: NVIDIA GPUs with the CUDA 12 toolkit on Linux x86_64, Comfort with JSONL and Parquet dataset formats and cluster job submission.
How long does NVIDIA NeMo Curator Documentation - Scalable Dataset Curation for LLM Training take?
Expect roughly ~3-5 hours for the per-modality quickstarts; ongoing reference for production pipeline work. Most learners work through it at their own pace.
What will I learn from NVIDIA NeMo Curator Documentation - Scalable Dataset Curation for LLM Training?
You'll learn: Build streaming curation pipelines that run on Ray across multi-node GPU clusters; Download and extract Common Crawl, arXiv and Wikipedia dumps into usable training corpora; Apply heuristic and classifier-based quality filters to score and drop low-value documents; Run exact, fuzzy and GPU-accelerated semantic deduplication over very large text datasets; Generate synthetic training data against any OpenAI-compatible endpoint including local vLLM; Curate image, video and audio datasets with NSFW, motion and word-error-rate filters; Deploy curation jobs onto Slurm HPC clusters or Kubernetes using the provided guides.
Topics
Sources
This page was written from 3 sources, 1 on domains other than docs.nvidia.com.
- 1.docs.nvidia.com — indexvendor
- 2.github.com — Curator
- 3.docs.nvidia.com — indexvendor