GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by Sebastian Raschka (Ahead of AI)
An illustrated explainer of looped (recurrent-depth) transformers, what they cost, and whether they explain GPT-6 Astra's shorter reasoning traces.
Overview
Sebastian Raschka, author of Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch), published this free Ahead of AI article on September 9, 2026. It runs about 7,000 words with 23 figures and six numbered sections. Section 1 reviews GPT-6 Astra's benchmark results, including Artificial Analysis indexes and Terminal-Bench, and notes that harness choice moves scores. It also explains reported computer-use training as a loop of screenshots, actions and verifier rewards, and states that Astra is still an RL-trained reasoning model. Section 2, the core of the piece, explains looped transformers: applying a 22-block stack twice gives 44 block applications with no new parameters, but compute matches a deeper model and each pass needs its own KV cache entries (Nanbeige4.2-3B ablations found that sharing the cache hurt quality). It then covers Universal Transformers with adaptive halting, Ouro, and Mixture-of-Recursions, where a learned router picks a loop count per token. Section 3 contrasts recurrence over depth with RNN recurrence over time. Section 4 weighs the evidence that Astra loops at all and labels it rumor-based. Section 5 looks at hidden chains of thought and token usage. Section 6 surveys papers on recurrent-depth latent reasoning (Geiping et al., 2025), virtual logic depth (Zhu et al., 2025), the compute-matched SMELT scaling laws (September 2026) and the full-bandwidth transformer (August 2026). The article has no code of its own; it links a Hugging Face implementation.
At a Glance
- Topic
- Models
- Level
- Intermediate
- Format
- Guide
- Cost
- Free
- Duration
- ~30 min read (about 7,000 words, 23 figures)
- Provider
- Sebastian Raschka (Ahead of AI)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Explain how a looped transformer reuses the same block stack to increase effective depth
- ✓Reason about why looping saves parameters but not compute or KV cache memory
- ✓Compare Universal Transformers' adaptive halting with Mixture-of-Recursions' per-token routing
- ✓Distinguish recurrence over depth in looped transformers from recurrence over time in RNNs
- ✓Interpret why shorter reasoning traces can reflect capability rather than hidden reasoning
- ✓Summarize recent looped-transformer papers on latent reasoning, memorization and compute-matched scaling
- ✓Judge frontier benchmark claims by checking which harness produced the score
Highlights
- •Free to read in full, with 23 original diagrams in the style of Raschka's LLM architecture sketches
- •Separates reported facts from rumor, with a whole section on whether Astra uses looping at all
- •Covers papers from August and September 2026 (SMELT, full-bandwidth transformer) next to the classic Universal Transformers work
- •Quantifies the trade-offs, such as SMELT's reported 6.8 to 18% training-compute savings at the same loss
- •Drew 74 comments on the post and a related video was submitted to Hacker News
Who It’s For
Best For
- ✓ML engineers tracking new architecture ideas in frontier and open-weight models
- ✓Practitioners deciding whether recurrent-depth models matter for their serving costs
- ✓Readers of Raschka's from-scratch books who want the next architecture topic
Prerequisites
- •Working knowledge of the transformer block, attention and the KV cache
- •Basic familiarity with reasoning models and chain-of-thought
FAQ
What is GPT-6 Astra, Looped Transformers, and Hidden Reasoning?
A free, illustrated long-form explainer by Sebastian Raschka, published September 9, 2026, for engineers who know the transformer basics and want to understand looped transformers. It covers how reusing blocks adds depth without parameters, why the KV cache does not shrink, adaptive and routed loop counts, and the recent research on latent reasoning.
Is GPT-6 Astra, Looped Transformers, and Hidden Reasoning free?
GPT-6 Astra, Looped Transformers, and Hidden Reasoning is free to access.
What level is GPT-6 Astra, Looped Transformers, and Hidden Reasoning for?
GPT-6 Astra, Looped Transformers, and Hidden Reasoning is aimed at a intermediate audience. Recommended background: Working knowledge of the transformer block, attention and the KV cache, Basic familiarity with reasoning models and chain-of-thought.
How long does GPT-6 Astra, Looped Transformers, and Hidden Reasoning take?
Expect roughly ~30 min read (about 7,000 words, 23 figures). Most learners work through it at their own pace.
What will I learn from GPT-6 Astra, Looped Transformers, and Hidden Reasoning?
You'll learn: Explain how a looped transformer reuses the same block stack to increase effective depth; Reason about why looping saves parameters but not compute or KV cache memory; Compare Universal Transformers' adaptive halting with Mixture-of-Recursions' per-token routing; Distinguish recurrence over depth in looped transformers from recurrence over time in RNNs; Interpret why shorter reasoning traces can reflect capability rather than hidden reasoning; Summarize recent looped-transformer papers on latent reasoning, memorization and compute-matched scaling; Judge frontier benchmark claims by checking which harness produced the score.
Topics
Sources
This page was written from 3 sources, 2 on domains other than magazine.sebastianraschka.com.