Building an AI Text Detector From Scratch
by Sebastian Raschka
An end-to-end open-source project that builds an AI-written-text classifier, from dataset construction through DistilBERT, LoRA, ModernBERT and Qwen3 fine-tuning to a local API and RLVR experiments.
Overview
Building an AI Text Detector From Scratch is the open-source companion repo to an August 15, 2026 Ahead of AI article by Sebastian Raschka, author of Build a Large Language Model (From Scratch). The article's implementation sections are for paid subscribers. The repository and its code are free under Apache-2.0. The project trains a binary classifier that returns a 0 to 100 score for how likely a text is AI-generated, and the README notes that this score reflects the training distribution, not proof of authorship. Work is organized into 18 numbered stage folders. Stages 1 to 4 build the data: licensed human text, AI-generated text checked with Pangram, light and moderate AI-edited text, and a Hugging Face dataset with grouped splits so that derived texts never leak across train and test. The published dataset, rasbt/human-vs-ai-50k, has 50,710 rows (40.6k train, 5.09k validation, 5.07k test) with per-row source licenses. Stages 5 to 11 compare a TF-IDF logistic regression baseline with calibration, a ChatGPT API baseline, full DistilBERT fine-tuning, DistilBERT with LoRA and MiCA, ModernBERT, GPT-2 and Qwen3. Stages 12 to 14 analyze length bias, learning curves and throughput, latency and memory. Stages 15 to 17 package a CLI, a local classifier server and a React plus FastAPI browser UI. Stage 16 is a case study on labeled Substack notes, and stage 18 uses the detector as a verifier in RLVR experiments. The stack is Python 3.12 with uv, optional dependency groups, Ollama or external APIs for data generation, and npm for the UI. Trained checkpoints such as rasbt/ai-text-detector-modernbert are on the Hugging Face Hub.
At a Glance
- Topic
- Fine-Tuning
- Level
- Intermediate
- Format
- Tutorial
- Cost
- Free
- Duration
- Self-paced project in 18 numbered stages; the training stages need a CUDA GPU
- Provider
- Sebastian Raschka
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Construct a labeled human-versus-AI text dataset with grouped splits that prevent leakage
- ✓Build and calibrate a TF-IDF logistic regression baseline before reaching for transformers
- ✓Benchmark an API-based ChatGPT classifier against locally fine-tuned models on the same data
- ✓Fine-tune DistilBERT fully and with LoRA and MiCA parameter-efficient methods
- ✓Fine-tune ModernBERT, GPT-2 and Qwen3 as sequence classifiers and compare their results
- ✓Analyze length bias, learning curves, throughput, latency and memory for a deployed classifier
- ✓Package a model as a CLI, a local inference server and a React plus FastAPI interface
- ✓Use a trained classifier as a reward verifier in reinforcement learning with verifiable rewards
Highlights
- •Covers the whole lifecycle in one repo: data construction, baselines, five model families, error analysis, deployment and RL
- •Careful data hygiene, with grouped splits and per-row source licenses in the published 50k-row Hugging Face dataset
- •Compares full fine-tuning, LoRA and MiCA side by side on the same task
- •Trained checkpoints (ModernBERT, Qwen3 0.6B, DistilBERT, GPT-2) are published on the Hugging Face Hub, so you can test inference before training
- •Code is free under Apache-2.0 even though the companion article's walkthrough is paywalled
Who It’s For
Best For
- ✓ML engineers who want a reference project for building a production text classifier
- ✓Practitioners comparing full fine-tuning against LoRA-style parameter-efficient methods
- ✓Engineers exploring classifier-as-verifier setups for RLVR training
- ✓Developers who learn best by reading and running a complete, staged codebase
Prerequisites
- •Comfortable with Python and PyTorch, including running training notebooks
- •Basic understanding of transformer classifiers and fine-tuning
- •Access to a CUDA GPU for the training stages; Python 3.12 and uv installed
FAQ
What is Building an AI Text Detector From Scratch?
Building an AI Text Detector From Scratch is Sebastian Raschka's free, Apache-2.0 GitHub project for engineers who want to see a full text-classification workflow done properly. Over 18 numbered stages it builds a 50k-sample human-versus-AI dataset, compares baselines and fine-tuned models, serves the detector locally and uses it as an RLVR verifier.
Is Building an AI Text Detector From Scratch free?
Building an AI Text Detector From Scratch is free to access.
What level is Building an AI Text Detector From Scratch for?
Building an AI Text Detector From Scratch is aimed at a intermediate audience. Recommended background: Comfortable with Python and PyTorch, including running training notebooks, Basic understanding of transformer classifiers and fine-tuning, Access to a CUDA GPU for the training stages; Python 3.12 and uv installed.
How long does Building an AI Text Detector From Scratch take?
Expect roughly Self-paced project in 18 numbered stages; the training stages need a CUDA GPU. Most learners work through it at their own pace.
What will I learn from Building an AI Text Detector From Scratch?
You'll learn: Construct a labeled human-versus-AI text dataset with grouped splits that prevent leakage; Build and calibrate a TF-IDF logistic regression baseline before reaching for transformers; Benchmark an API-based ChatGPT classifier against locally fine-tuned models on the same data; Fine-tune DistilBERT fully and with LoRA and MiCA parameter-efficient methods; Fine-tune ModernBERT, GPT-2 and Qwen3 as sequence classifiers and compare their results; Analyze length bias, learning curves, throughput, latency and memory for a deployed classifier; Package a model as a CLI, a local inference server and a React plus FastAPI interface; Use a trained classifier as a reward verifier in reinforcement learning with verifiable rewards.
Topics
Sources
This page was written from 3 sources, 2 on domains other than github.com.