Interaction Models: A Scalable Approach to Human-AI Collaboration
by Thinking Machines Lab
Thinking Machines' design for a full-duplex audio, video and text model that talks, listens and delegates to a background agent at once.
Overview
Interaction Models: A Scalable Approach to Human-AI Collaboration is a research post Thinking Machines Lab published on May 11, 2026 on its Connectionism blog (DOI 10.64434/tml.20260511), announcing a research preview of TML-Interaction-Small, a 276B-parameter mixture-of-experts model with 12B active parameters. The authors argue that turn-based models, which freeze input while they generate and depend on voice-activity-detection harnesses, limit how much a human can steer an AI, and that interactivity belongs inside the model. The post runs from The collaboration bottleneck to Capabilities, Our approach, Benchmarks and Limitations and future work. The system pairs a real-time interaction model with an asynchronous background model that does deeper reasoning and tool use on the shared conversation context. Inputs and outputs are interleaved 200ms chunks; there are no standalone encoders (dMel audio features, 40x40 image patches through an hMLP, a flow head for audio output), all trained from scratch with the transformer. On inference, the team upstreamed a streaming-session feature to SGLang, tuned MoE kernels for latency and used batch-invariant kernels for bitwise trainer-sampler alignment at under 5% overhead. Reported results include 0.40s turn-taking latency on FD-bench V1 against 1.18s for GPT-realtime-2.0 (minimal), plus new internal benchmarks such as TimeSpeak and CueSpeak. Weights, code and benchmark datasets are not released.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~25 min read
- Provider
- Thinking Machines Lab
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Why turn-based voice models with voice-activity-detection harnesses limit real-time human-AI collaboration
- ✓How time-aligned 200ms micro-turns let a model listen and speak at the same time
- ✓How encoder-free early fusion handles audio with dMel features and images with 40x40 patches
- ✓How a real-time interaction model hands harder reasoning and tool calls to a background model
- ✓Which inference optimizations matter for streaming MoE serving, including persistent streaming sessions in SGLang
- ✓How batch-invariant kernels give bitwise trainer-sampler alignment for reinforcement learning at under 5 percent overhead
- ✓How full-duplex models are evaluated on FD-bench, Audio MultiChallenge and new time-awareness benchmarks
Highlights
- •A detailed public architecture write-up for a large full-duplex multimodal model, including serving and determinism engineering most labs leave out
- •States its limits plainly: context grows fast in long sessions, quality drops on poor connections and larger models are still too slow to serve
- •334 points on Hacker News (May 2026); Sean Goedecke's review argues full-duplex voice already exists in Moshi and similar models and that the scale and video input are the real advance
- •Benchmarks are mostly the lab's own and some favour 'instant' comparisons (GPT-realtime-2.0 xhigh beats it on Audio MultiChallenge), so read the tables critically
- •Nothing to run: weights, code and the TimeSpeak and CueSpeak datasets are not released
Who It’s For
Best For
- ✓Engineers building voice or real-time multimodal agents who want to know where model-native interactivity is heading
- ✓ML researchers working on streaming inference, full-duplex speech or multimodal early fusion
- ✓Inference engineers interested in low-latency MoE serving and deterministic RL training
Prerequisites
- •Solid understanding of transformer architectures and mixture-of-experts models
- •Familiarity with speech or voice-agent pipelines (ASR, TTS, voice activity detection) and LLM serving
FAQ
What is Interaction Models: A Scalable Approach to Human-AI Collaboration?
A Thinking Machines Lab research post for ML and voice-agent engineers on interaction models: a model that processes audio, video and text in 200ms micro-turns, so it can respond, interject and run tools while the user is still talking. It covers the architecture, inference tricks, benchmarks and the limits the authors admit.
Is Interaction Models: A Scalable Approach to Human-AI Collaboration free?
Interaction Models: A Scalable Approach to Human-AI Collaboration is free to access.
What level is Interaction Models: A Scalable Approach to Human-AI Collaboration for?
Interaction Models: A Scalable Approach to Human-AI Collaboration is aimed at a advanced audience. Recommended background: Solid understanding of transformer architectures and mixture-of-experts models, Familiarity with speech or voice-agent pipelines (ASR, TTS, voice activity detection) and LLM serving.
How long does Interaction Models: A Scalable Approach to Human-AI Collaboration take?
Expect roughly ~25 min read. Most learners work through it at their own pace.
What will I learn from Interaction Models: A Scalable Approach to Human-AI Collaboration?
You'll learn: Why turn-based voice models with voice-activity-detection harnesses limit real-time human-AI collaboration; How time-aligned 200ms micro-turns let a model listen and speak at the same time; How encoder-free early fusion handles audio with dMel features and images with 40x40 patches; How a real-time interaction model hands harder reasoning and tool calls to a background model; Which inference optimizations matter for streaming MoE serving, including persistent streaming sessions in SGLang; How batch-invariant kernels give bitwise trainer-sampler alignment for reinforcement learning at under 5 percent overhead; How full-duplex models are evaluated on FD-bench, Audio MultiChallenge and new time-awareness benchmarks.
Topics
Sources
This page was written from 4 sources, 2 on domains other than thinkingmachines.ai.