Groq
by Groq, Inc.
The fastest way to run open models — inference on purpose-built LPU silicon, not GPUs.
Groq is an AI inference cloud that runs open-weight language, speech and agentic models on its own LPU silicon instead of GPUs. It targets teams whose product breaks when the model is slow — voice agents, real-time assistants and long reasoning chains — behind an OpenAI-compatible API with a no-credit-card free tier and per-token pricing from $0.05 per million.
Groq runs a public inference cloud on its own silicon: the LPU (Language Processing Unit), a deterministic architecture designed for sequential token generation rather than the parallel matrix throughput GPUs optimise for. GroqCloud exposes that hardware as an OpenAI-compatible API serving open-weight models — Llama 3.1 8B at roughly 560 tokens per second, Llama 3.3 70B at about 280 t/s, GPT-OSS 20B at around 1,000 t/s and GPT-OSS 120B at about 500 t/s, all with 131K context — alongside Whisper speech-to-text and Groq Compound, a hosted agentic system with built-in tool use running near 450 t/s. Founded in 2016, the company pivoted from selling chips to selling tokens in 2024 (its CEO's public line was 'we no longer sell hardware') and now states it operates 13 data centres across North America, Europe, the Middle East and Asia-Pacific, serves more than five million developers, processes trillions of tokens weekly, and is scaling toward 200 MW of capacity by the end of 2027. In December 2025 NVIDIA agreed to a non-exclusive licence of Groq's inference patent portfolio and software — reported by CNBC at roughly $20 billion, which NVIDIA has not confirmed — and hired founder Jonathan Ross, president Sunny Madra and other staff, while Groq itself continued as a separate company; it then raised $650 million in June 2026 led by Disruptive and Infinitum. The proposition is narrow and concrete: the lowest time-to-first-token and highest sustained tokens-per-second available for open models, at $0.05 to $0.90 per million tokens, with a Batch API and prompt caching that each halve the rate.
The engineering lead on a latency-bound AI product — voice agents, live copilots, multi-step agent loops — where token generation speed is the user experience, not a benchmark line.
Interactive-speed generation on open models (280–1,000 tokens/second) at $0.05–$0.90 per million tokens, reachable with an OpenAI SDK base-URL change.
At a Glance
- Category
- Infrastructure & Cloud
- Pricing
- Usage-based, Freemium, Contact for pricing
- Target Market
- CTOs, Enterprise Developers, ML Engineers, AI Product Leads
- Deployment
- Cloud-first, API-based, Self-hosted
- Founded
- 2016
- Headquarters
- Mountain View, California, United States
- Team Size
- 201-500
- Customers
- 5M+ developers; thousands of AI-native companies and Fortune 500 enterprises (vendor-stated, unnamed)
Key Features
- ✓LPU inference architecture
Deterministic silicon built for sequential token generation, delivering 280 to 1,000 tokens per second on production models.
- ✓OpenAI-compatible API
Acts as a drop-in replacement for the OpenAI SDK, so migrating an existing application is usually a base-URL and key change.
- ✓Batch API at 50% off
Asynchronous jobs returned within 24 hours to 7 days are billed at half the on-demand token rate.
- ✓Prompt caching
Repeated input prefixes bill at 50% off and cached tokens do not count against organisation rate limits.
- ✓Groq Compound agentic system
A hosted agentic model with built-in tool use running at roughly 450 tokens per second for agent workloads.
- ✓Whisper speech-to-text
Whisper Large v3 Turbo transcribes audio at $0.04 per hour, well under typical hosted transcription pricing.
- ✓Free tier without a credit card
30 requests and 6,000 to 30,000 tokens per minute at no cost, enough to prototype before committing spend.
Capabilities
Use Cases
- •Real-time voice agents
Sub-second time-to-first-token keeps spoken conversations natural, where slower GPU-hosted models introduce audible pauses that break the illusion.
- •Multi-step agent loops
Agents multiply token counts across steps, so faster per-call generation is what makes long chains viable inside an interactive product.
- •Overnight batch processing
Large backlogs of classification, extraction or summarisation run through the Batch API at half the on-demand rate.
- •Closed-model cost replacement
Teams swapping a frontier closed model for open weights on GroqCloud cut per-token spend by roughly an order of magnitude.
- •Bulk audio transcription
Whisper Large v3 Turbo at $0.04 per audio hour makes meeting, call-centre and podcast transcription economical at volume.
Ideal For
Best For
- ✓Real-time voice and conversational agents where time-to-first-token decides whether the interaction feels human
- ✓Agent loops and reasoning chains whose per-step latency compounds across many model calls
- ✓High-volume batch workloads (classification, extraction, summarisation) that can absorb a 24-hour turnaround for half price
- ✓Teams migrating off closed frontier APIs to open weights primarily to cut per-token spend
- ✓Bulk audio transcription pipelines priced against Whisper Large v3 Turbo at $0.04 per audio hour
Not Ideal For
- ✗Teams that need GPT, Claude or Gemini — Groq's catalogue is entirely open-weight, so it cannot be a single-vendor replacement
- ✗Workloads needing generous free-tier headroom: the free plan caps at 30 requests and 6,000–30,000 tokens per minute, enforced per organisation rather than per API key
- ✗Buyers who need published pricing for on-premises GroqRack, fine-tuning or SLAs — all three are sales-negotiated with no rate card
- ✗Applications pinned to a specific preview model, which Groq labels evaluation-only and may deprecate
Integrations
Deployment
Market & Ratings
5M+ developers; thousands of AI-native companies and Fortune 500 enterprises (vendor-stated, unnamed)
Market Analysis
Pros
- ✓Fastest published token generation for open models — 280 to 1,000 t/s depending on model size
- ✓Aggressive pricing with stacking Batch API (50%) and prompt-cache (50%) discounts
- ✓Free tier needs no credit card, so technical evaluation costs nothing
- ✓One production review measured 99.94% uptime with p99 latency within 15% of median, better tail consistency than GPU alternatives
Cons
- ✗Catalogue is entirely open-weight — no GPT, Claude or Gemini — so it cannot replace a closed-model vendor outright
- ✗Rate limits apply per organisation, not per API key, and the free tier's 30 requests/minute is restrictive for anything beyond prototyping
- ✗Preview models are explicitly evaluation-only and subject to deprecation, complicating production version pinning
- ✗Fine-tuning, on-premises GroqRack and enterprise SLAs carry no published pricing and require sales contact
- ✗Persistent practitioner scepticism that per-token unit economics are subsidised rather than structurally cheap
- ✗Key leadership including founder Jonathan Ross and president Sunny Madra left for NVIDIA under the December 2025 licensing deal
Pricing
Free
$0
- ✓All models, no credit card required
- ✓30 requests per minute
- ✓6,000–30,000 tokens per minute by model
- ✓14,400 requests per day at organisation level
Developer
From $0.05 per 1M input tokens
- ✓1,000 requests per minute
- ✓250,000–300,000 tokens per minute
- ✓Pay-as-you-go, no monthly minimum
- ✓Batch API at 50% off
- ✓Prompt caching at 50% off cached input
Enterprise
Contact for pricing
- ✓Custom rate limits and SLAs
- ✓Dedicated support
- ✓Volume pricing
- ✓On-premises GroqRack
- ✓Fine-tuning access
Metered purely per token — $0.05 to $0.90 per million depending on model, with no idle infrastructure charge. The Batch API halves every rate and prompt caching takes another 50% off repeated prefixes, so cached batch work lands near a quarter of list. Rate limits are enforced per organisation rather than per API key, so extra keys do not multiply quota. Speech is priced by audio hour ($0.04 for Whisper Large v3 Turbo, $0.111 for Large v3). On-premises GroqRack, fine-tuning and enterprise SLAs have no published pricing and require a sales conversation.
Security & Compliance
Connect
Sources
This page was written from 6 sources, 5 on domains other than groq.com.
Stay Ahead of the Curve
Weekly enterprise AI insights for technology leaders. No spam, no vendor pitches—unsubscribe anytime.
SubscribeRelated Products
WEKA NeuralMesh
Microservices storage and memory fabric built for AI training and inference at exabyte scale
Tsuga
Bring-your-own-cloud observability that keeps telemetry, and its cost, inside your own AWS account
OpenObserve
Open-source, Rust-based observability on object storage — logs, metrics, traces and LLM telemetry in one binary
Coralogix
AI-native observability that queries logs, metrics and traces in place — index-free, in your own S3 bucket