Understanding Positional Embeddings in Modern LLMs
by Sebastian Raschka (Ahead of AI)
How LLMs encode token position, from GPT-2's learned table to RoPE, partial RoPE and NoPE layers, with PyTorch code.
Overview
Understanding Positional Embeddings in Modern LLMs is a November 15, 2026 article from Sebastian Raschka's Ahead of AI newsletter, running roughly 9,000 to 10,000 words with about 38 figures. Raschka is the author of Build a Large Language Model (From Scratch) and maintains the rasbt/LLMs-from-scratch repository, which the article links for a full Qwen3 implementation. It is organized in seven numbered sections. Section 1 traces where position information enters a model and why causal masking alone gives only an implicit order signal. Section 2 covers learned absolute embeddings in GPT-2, with its 1,024-position limit, and the fixed sinusoidal encodings of the original transformer. Sections 3 and 4 explain RoPE step by step: rotating queries and keys but not values, split-half feature pairing, a worked numeric example, shift invariance and the frequency schedule with base 10,000. Section 5 implements RoPE in PyTorch, inside causal attention, and shows the interleaved pairing used in the original LLaMA code. Section 6 surveys 2026 designs: partial RoPE in MiniMax-M3, NoPE layers in SmolLM3, and a learned relative position bias in Inkling, with counts from Raschka's LLM Architecture Gallery. ALiBi and YaRN are not covered.
At a Glance
- Topic
- Models
- Level
- Intermediate
- Format
- Tutorial
- Cost
- Free
- Duration
- ~40 min read, plus time to run the PyTorch snippets
- Provider
- Sebastian Raschka (Ahead of AI)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Why causal masking gives a transformer only an implicit signal about token order
- ✓How GPT-2's learned absolute position table works and why it caps context at 1,024 tokens
- ✓How RoPE rotates query and key feature pairs so attention depends on relative offset
- ✓How the RoPE frequency schedule is computed from a base of 10,000 and head dimension
- ✓How to implement RoPE in PyTorch inside causal attention and verify shift invariance
- ✓How split-half and interleaved RoPE pairing conventions differ and convert between each other
- ✓How partial RoPE, NoPE layers and learned relative position bias appear in 2026 open-weight models
Highlights
- •Reached 226 points and 27 comments on Hacker News the day it was published
- •Builds RoPE up from a single rotated feature pair with worked numbers before showing the full rotation matrix
- •Ties the theory to real 2026 configs, such as MiniMax-M3 rotating 64 of 128 head dimensions
- •Includes runnable PyTorch functions and links to a from-scratch Qwen3 notebook in a widely used GitHub repo
- •HN readers note it leans toward implementation over theory and skips ALiBi and YaRN, so pair it with a context-extension source if you need those
Who It’s For
Best For
- ✓Engineers implementing or porting transformer architectures in PyTorch
- ✓Practitioners reading open-weight model configs who want to understand rope_theta and partial rotary settings
- ✓Readers of Build a Large Language Model (From Scratch) moving on to current architectures
- ✓Researchers comparing positional schemes across 2025 and 2026 model releases
Prerequisites
- •Working knowledge of self-attention: queries, keys, values and causal masking
- •Comfort reading PyTorch tensor code and basic linear algebra such as dot products and 2D rotations
FAQ
What is Understanding Positional Embeddings in Modern LLMs?
Understanding Positional Embeddings in Modern LLMs is a long illustrated tutorial by Sebastian Raschka, published November 15, 2026, explaining how transformer language models represent token order. It is for engineers who read model configs or implement architectures and want to know what RoPE does, why partial RoPE and NoPE layers appeared in 2026 open-weight models, and how to code RoPE themselves.
Is Understanding Positional Embeddings in Modern LLMs free?
Understanding Positional Embeddings in Modern LLMs is free to access.
What level is Understanding Positional Embeddings in Modern LLMs for?
Understanding Positional Embeddings in Modern LLMs is aimed at a intermediate audience. Recommended background: Working knowledge of self-attention: queries, keys, values and causal masking, Comfort reading PyTorch tensor code and basic linear algebra such as dot products and 2D rotations.
How long does Understanding Positional Embeddings in Modern LLMs take?
Expect roughly ~40 min read, plus time to run the PyTorch snippets. Most learners work through it at their own pace.
What will I learn from Understanding Positional Embeddings in Modern LLMs?
You'll learn: Why causal masking gives a transformer only an implicit signal about token order; How GPT-2's learned absolute position table works and why it caps context at 1,024 tokens; How RoPE rotates query and key feature pairs so attention depends on relative offset; How the RoPE frequency schedule is computed from a base of 10,000 and head dimension; How to implement RoPE in PyTorch inside causal attention and verify shift invariance; How split-half and interleaved RoPE pairing conventions differ and convert between each other; How partial RoPE, NoPE layers and learned relative position bias appear in 2026 open-weight models.
Topics
Sources
This page was written from 3 sources, 1 on domains other than magazine.sebastianraschka.com.