Groq runs LLM inference on custom LPU silicon and the result is the fastest hosted inference on the market, 800+ TPS on Llama 3.1 8B, 1000 TPS on GPT OSS 20B, 500 TPS on GPT OSS 120B. For real-time chat, agent workflows, and voice apps where latency is felt, Groq's speed advantage is meaningful and the per-token pricing is competitive with other serverless providers. The trade is model coverage: Groq supports a curated set of models, not the full OSS catalog. When the model you need is supported, Groq usually wins.
- Custom LPU hardware, 800-1000 TPS on supported models
- Llama 3.1 8B at $0.05 input / $0.08 output, 840 TPS
- GPT OSS 120B at $0.15 input / $0.60 output, 500 TPS
- 50% off cached prompts; Batch API at 50% lower cost
- Whisper transcription at $0.04-$0.11/hour
- Smaller model catalog than Together / Fireworks
- LPU architecture is the moat, but limits which models can be served
- Less flexibility for custom model deployment
- Pricing per-token is competitive, not dramatically cheaper
- Tied to Groq's hardware roadmap for new model support
Groq runs LLM inference on custom-designed LPU (Language Processing Unit) silicon, purpose-built for transformer inference and dramatically faster than GPU-based serving on the models it supports. The speed shows up in user-facing latency: real-time chat feels different on Groq than on GPU-based providers, especially for streaming token output.
Where it fits
Groq is the right pick when latency is a felt experience in your product, real-time chat, voice agents, interactive code generation, anything where the user perceives the time-to-first- token and the streaming speed of subsequent tokens. The LPU architecture delivers 800-1000 TPS on smaller models and 400-500 TPS on larger ones, often 5-10x what GPU-based providers deliver.
For voice apps, Groq’s Whisper inference is fast enough for real-time transcription at $0.04-$0.11/hour of audio. For batch processing of large token volumes, the Batch API offers 50% off synchronous pricing.
Avoid Groq when the model you need isn’t in the catalog (the selection is curated, not exhaustive), when you need custom model deployment, or when raw cost per token matters more than latency and your traffic isn’t latency-sensitive.
Pricing in practice
Llama 3.1 8B Instant at $0.05/M input + $0.08/M output (840 TPS). Llama 3.3 70B Versatile at $0.59 input / $0.79 output (394 TPS). GPT OSS 20B at $0.075 / $0.30 (1000 TPS). GPT OSS 120B at $0.15 / $0.60 (500 TPS). Qwen3 32B at $0.29 / $0.59 (662 TPS). Llama 4 Scout at $0.11 / $0.34 (594 TPS).
Prompt caching delivers 50% off cached input tokens on hits. The Batch API offers 50% lower cost for asynchronous large-scale workloads. Whisper transcription is $0.04-$0.111/hour. Code execution and web search are built-in tools available in the API.
How it compares
Together AI, Broader catalog (200+ models) at lower TPS. Pick when model variety matters more than raw speed.
Fireworks AI, High-performance serverless on GPU. Pick when Together’s catalog isn’t quite right and you want a Together peer.
OpenRouter, Router across 400+ models from 60+ providers. Pick when you want one API and the flexibility to swap providers based on speed/cost.
What changed recently
Groq has continued to add model support through 2025-2026 with Llama 4 Scout, GPT OSS variants, and Qwen3 32B landing on the LPU platform. The Batch API gained additional throughput capacity. Prompt caching shipped with 50% discount on cached input, meaningful for agent workflows with long system prompts. Whisper turbo support expanded the voice surface. The team continues to invest in adding frontier OSS models to the catalog as they release, with the LPU architecture’s model-support gap closing over time.
Sources
- Groq Pricing, groq.com, 2026
- Groq Docs, groq.com
- Groq LPU Architecture, groq.com
- Groq Models, groq.com