Modal launches Auto Endpoints to put transparent LLM inference behind a single CLI command

Modal is launching Auto Endpoints, a product that deploys production-ready LLM inference with a single CLI command while keeping the full serving stack visible and modifiable. The pitch targets the gap between managed inference APIs that hide their configuration and self-managed deployments that require tuning a web of engine parameters before the first request lands cleanly.

What shipped

Auto Endpoints are OpenAI API-compatible inference services backed by Modal applications that users can inspect, fork, and change. Each endpoint ships with configurations pre-optimized by Modal’s team for specific models, with GPU selection, engine settings, and scaling behavior already tuned. Deploying one looks like:

modal endpoint create --name agent --model zai-org/GLM-5.2-FP8

Under the hood, each endpoint runs as a Modal application with auto-scaling across multiple replicas. The differentiator from a vanilla managed provider is observability: dashboards expose engine-level metrics including speculative decoding acceptance rates and per-replica latency rather than just aggregate request throughput. The underlying application code is also accessible, so teams that need to deviate from Modal’s defaults can modify the serving configuration directly.

Why inference ownership is hard

Running LLM inference sounds straightforward until you hit the configuration surface. Engines like vLLM and SGLang expose dozens of parameters: tensor parallelism settings, KV cache sizes, speculative decoding configurations, and quantization choices that interact in non-obvious ways. Managed inference providers abstract all of that, but the abstraction makes debugging latency spikes harder and prevents tuning for a specific model mix or traffic pattern.

Modal’s bet appears to be that a meaningful segment of teams wants to own their inference stack without building it from scratch. Auto Endpoints try to occupy that middle ground: the hard configuration work is done upfront by Modal’s team, but the underlying application remains accessible if you need to deviate from the defaults. The “inference you actually own” framing in the announcement is a direct counter-positioning to the black-box model that most managed providers use.

Where it fits

Auto Endpoints are priced on Modal’s standard compute model, billed per GPU-second. Total cost depends heavily on which model you’re serving and how steady your traffic is. For low-traffic workloads, per-second billing may run higher than a dedicated managed endpoint from a provider offering reserved capacity. The product appears most compelling for teams with moderate, variable traffic who want to avoid both the cold-start costs of fully serverless inference and the management overhead of always-on GPU instances.

The product also fits naturally alongside Modal’s existing sandboxes and serverless GPU workloads. A team running evals or fine-tuning on Modal can now serve the resulting model from the same platform without switching inference providers. That horizontal expansion is likely part of the play: Auto Endpoints are a higher-leverage entry point than raw serverless GPUs, and teams that adopt them for a specific model tend to find reasons to run other workloads on the same platform.

What’s worth watching

Modal raised a $355M Series C at a $4.65B valuation in May 2026. Auto Endpoints look like a product milestone funded by that round: an attempt to grow past infrastructure-for-AI-developers into a broader inference platform. The question is whether the “transparent stack” positioning holds up as the product scales. Pre-optimized configurations are only as good as the team maintaining them. If model updates or new hardware outpace Modal’s tuning cadence, the “optimized” claim weakens.

The independent inference market is crowded. Fireworks, Together AI, Groq, and OpenRouter all offer managed endpoints for open models. Modal’s differentiation is the editable stack, but most teams adopting managed inference want less operational surface, not more. Whether the transparency argument resonates beyond ML-infrastructure teams is the open question over the next few quarters.

Sources cited

1. Introducing Modal Auto Endpoints modal.com
2. Modal Series C announcement modal.com