Language Models for Text Classification: From Bag-of-Words to Jev
by Sebastian Raschka
A visual, benchmarked history of text classification, from bag-of-words and LSTMs to ModernBERT, GPT-style classifiers and calibrated decision APIs like Jev.
Overview
Language Models for Text Classification: From Bag-of-Words to Jev is a long-form visual guide that Sebastian Raschka, author of Build a Large Language Model (From Scratch), published in his Ahead of AI magazine on September 29, 2026. It has six sections. Section 1 covers the pre-transformer era: naive Bayes, logistic regression and XGBoost on bag-of-words features, then word embeddings, RNNs and CNNs. Section 2 compares encoder-style models (BERT, ModernBERT), decoder-style LLMs (GPT-2, Qwen3 0.6B) and encoder-decoder models (T5). Sections 3 to 6 examine Jev, a proprietary classification model from TypeSafe AI with Choice, Noul and Score APIs: how its API differs from text-to-text classification, its results with BERT- and GPT-style setups, what is known about its architecture, and a discussion of calibration covering temperature scaling, Brier loss and reinforcement learning with calibration rewards (RLCR). Most methods are benchmarked on the IMDb sentiment set. Reported accuracies include 89.9% for bag-of-words logistic regression, about 92% for a GPT-2 124M classifier, about 95% for ModernBERT and 96.47% for the Jev Choice API, which cost $0.65 for all 25,000 reviews. The article links code in Raschka's machine-learning-book, deeplearning-models and LLMs-from-scratch repositories. Its practical advice: keep bag-of-words plus logistic regression as a baseline, fine-tune a classifier such as ModernBERT for repeated high-volume tasks, and use a cheap LLM or Jev-like API for one-off decisions. Jev's internals are not public, so the architecture section is informed speculation. An independent pilot benchmark (AbdelStark/jev-benchmarks) found Jev well ahead of GLiNER2.5 on AG News and Banking77 but poorly calibrated on emotion labels.
At a Glance
- Topic
- Models
- Level
- Intermediate
- Format
- Guide
- Cost
- Free
- Duration
- Long-form article (~1 hour read); linked notebooks optional
- Provider
- Sebastian Raschka
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Build a bag-of-words baseline with naive Bayes, logistic regression and XGBoost for text
- ✓Explain how word embeddings, RNNs and CNNs were used for pre-transformer text classification
- ✓Compare encoder, decoder and encoder-decoder transformers as classifiers using IMDb benchmark numbers
- ✓Decide between a classic baseline, a fine-tuned ModernBERT and an LLM API for a task
- ✓Use typed decision APIs like Jev's Choice, Noul and Score endpoints for classification calls
- ✓Explain calibration with temperature scaling, Brier loss and RL with calibration rewards (RLCR)
- ✓Weigh accuracy against latency and dollar cost using measured runs on 25,000 IMDb reviews
Highlights
- •Puts classic ML, fine-tuned encoders, GPT-style classifiers and a 2026 decision API on one benchmark (IMDb) with accuracy, time and cost
- •Gives a practical rule of thumb for when to fine-tune versus call an API, from an author who has written two books on the subject
- •Links runnable notebooks for logistic regression, LSTM, CNN and GPT-2 classification in the author's GitHub repos
- •Treats calibration seriously (ECE, Brier score, RLCR), a topic most classification tutorials skip
- •Free to read, with no paywall on the article body
Who It’s For
Best For
- ✓ML engineers building production text classifiers who need to choose a model family
- ✓LLM application developers deciding whether a classification step needs an LLM call at all
- ✓Practitioners who want a single refresher spanning classic NLP to modern decision APIs
Prerequisites
- •Basic supervised learning: train/validation/test splits, cross-entropy, accuracy
- •Familiarity with neural networks, embeddings and the idea of transformer pretraining and fine-tuning
- •Some exposure to calling LLM APIs helps for the Jev sections
FAQ
What is Language Models for Text Classification: From Bag-of-Words to Jev?
This free September 2026 guide by Sebastian Raschka traces text classification from bag-of-words models through RNNs, CNNs, encoder and decoder transformers to Jev-style decision APIs. It is for ML engineers choosing between a logistic regression baseline, a fine-tuned ModernBERT and an LLM API for a real classification task.
Is Language Models for Text Classification: From Bag-of-Words to Jev free?
Language Models for Text Classification: From Bag-of-Words to Jev is free to access.
What level is Language Models for Text Classification: From Bag-of-Words to Jev for?
Language Models for Text Classification: From Bag-of-Words to Jev is aimed at a intermediate audience. Recommended background: Basic supervised learning: train/validation/test splits, cross-entropy, accuracy, Familiarity with neural networks, embeddings and the idea of transformer pretraining and fine-tuning, Some exposure to calling LLM APIs helps for the Jev sections.
How long does Language Models for Text Classification: From Bag-of-Words to Jev take?
Expect roughly Long-form article (~1 hour read); linked notebooks optional. Most learners work through it at their own pace.
What will I learn from Language Models for Text Classification: From Bag-of-Words to Jev?
You'll learn: Build a bag-of-words baseline with naive Bayes, logistic regression and XGBoost for text; Explain how word embeddings, RNNs and CNNs were used for pre-transformer text classification; Compare encoder, decoder and encoder-decoder transformers as classifiers using IMDb benchmark numbers; Decide between a classic baseline, a fine-tuned ModernBERT and an LLM API for a task; Use typed decision APIs like Jev's Choice, Noul and Score endpoints for classification calls; Explain calibration with temperature scaling, Brier loss and RL with calibration rewards (RLCR); Weigh accuracy against latency and dollar cost using measured runs on 25,000 IMDb reviews.
Topics
Sources
This page was written from 3 sources, 1 on domains other than magazine.sebastianraschka.com.