On 1 October 2026, Modal said Modal Clusters are generally available. The new unit of work is a multi-node cluster behind one decorator: @modal.clustered. The pitch is blunt. Training and serving “owned intelligence” at scale needs petaFLOP/s of compute and terabit/s of networking across many nodes — and owning that intelligence does not have to mean owning the hardware or the fabric ops team.
Cowry Tech News is covering this as a US AI/cloud infrastructure piece next to CoreWeave’s Vera Rubin NVL72 production note. Different buyers. CoreWeave’s story is rack-scale generation handoff on an AI-specialist cloud. Modal’s story is serverless-priced multi-node: request a cluster, get InfiniBand/RDMA configured for PyTorch and NCCL, bill by the second, tear it down. Neither piece is a raise tip sheet.
What GA actually ships
From Modal’s primary post:
- Clusters have been in battle-testing for about 1.5 years; GA is announced 1 October 2026.
- API surface:
@modal.clustered(size=..., rdma=True)on top of existing@app.function(gpu=...). - Helpers:
modal.Cluster.from_context(),container_ips(),container_rank()for world size and rendezvous. - Integrates with existing Modal primitives: Volumes for checkpoints, Cloud Bucket Mounts for data, Queues for orchestration.
- Inter-node communication via InfiniBand verbs at up to 6.4 Tbps, automatically configured for PyTorch and NCCL when
rdma=True. - Time-to-cluster: Modal claims code runs on a cluster within seconds of requesting it, billed by the second.
- GPU health assurance and remediation extended to RDMA health.
- Contrast Modal draws: elsewhere, on-demand cluster access often means hourly billing or reservations; Modal says run when you want and pay for what you use.
- Cluster size is bounded by the workspace plan’s GPU limits; large jobs need outreach.
Example shape from the post (illustrative of the API, not a CTN-run workload):
@app.function(gpu="B300:8")
@modal.clustered(size=4, rdma=True)
def train_model():
cluster = modal.Cluster.from_context()
container_ips = cluster.container_ips()
container_rank = cluster.container_rank()
world_size = len(container_ips)
main_addr = container_ips[0]
...That is the contract in code. Specs without a decorator are marketing. A decorator without plan headroom is still blocked.
Why Modal rebuilt scheduling for N nodes
Modal’s traditional scheduler is described as greedy: nodes poll for work; the scheduler assigns per node with heuristics around placement, fragmentation, and cost. Multi-node jobs need an atomic unit of N nodes, not N lucky single-node picks.
Modal says it built a gang scheduler with an observe → plan → act loop:
- Collect metadata for all multi-node workloads.
- Assign workloads to nodes, grouping by availability zone and network ID.
- Spin up new capacity when pending clusters do not fit.
- Notify live nodes of the schedule.
The gang scheduler pulls from the same capacity pool as the rest of Modal. Modal claims that yields cluster acquisition in seconds and “the only place where you pay truly serverless pricing” for this pattern. Treat “fastest on the market” as company positioning. Treat “same pool + gang place” as the engineering claim that matters for burst training and multi-node inference.
For operators: if your jobs fail when two of four nodes land on the wrong network island, you already know why gang scheduling exists. Ask Modal how AZ and network ID constraints show up in errors when capacity is tight.
RDMA without becoming a fabric team
Modal’s post walks the bandwidth math that makes RDMA non-optional for serious multi-node work:
- Example: syncing ~717 GB of BF16 weights from trainer to rollout engine each training step of “GLM 4.7” — nearly two minutes over 50 Gbps TCP vs under two seconds over RDMA, repeated for thousands of steps.
- Example: prefill-decode disaggregation on Llama 3.1 70B moving ~10 GB of KV cache per 32k-token prompt inside a few-hundred-millisecond time-to-first-token budget.
RDMA moves GPU or CPU memory between hosts without bouncing through kernel-space on the data path, which is how Modal states a node can push up to 6.4 Tbps. The hard part Modal claims to hide: every cloud’s RDMA stack wants different drivers, environment variables, and userspace libraries. The product answer is one flag — rdma=True — with automated setup so PyTorch + NCCL workloads work out of the box across the clouds Modal runs on.
Security detail that is easy to skip in a launch post: Modal prefers gVisor over runc in a multi-tenant environment. gVisor lacked RDMA support, so Modal says it implemented RDMA in gVisor and upstreamed the changes. That is a concrete OSS claim. Buyers who care about tenant isolation should still ask how uverbs proxying is audited and what failure looks like when the NIC or fabric flaps.
Customer anecdotes (Modal-hosted)
Modal names three customer stories on the GA post. Attribute them as Modal-published customer quotes, not CTN interviews.
Decagon (customer-experience agents) fine-tuned open models with up to a trillion parameters using Miles on Modal Clusters, with Modal contributing LoRA support to Miles. Cyrus Asgari, Research Lead, is quoted on model performance being central and on Clusters letting the team train at that scale while focusing on agent quality.
1x (NEO home robot) pre-trains a world model on Modal Clusters — multi-node B300 with RDMA for big runs, fleets of single-node jobs for evals — same scheduler for both. Sam Sinha, Head of World Model Lab, is quoted on bursting to hundreds of B300s with RDMA and spinning down between runs, paying for what they use.
Runway (Gen-4.5 and Aleph 2.0 video models) uses Modal Clusters to spread inference across multiple nodes, with Modal’s autoscaler scaling clusters as a unit. Kamil Sindi, CTO, is quoted on multi-node inference making elasticity hard and on clustering plus configurable autoscaling helping scale production inference.
Useful existence proofs. Not your workload. Trillion-parameter fine-tunes, robot world models, and frontier video inference do not share one NCCL graph.
Where Clusters fit next to “just buy NVL72”
Modal Clusters are not a Vera Rubin availability story. They are a multi-node programming and billing story on Modal’s pool — including B300 examples in customer quotes and docs-shaped GPU strings in the sample.
Choose Clusters when:
- You need multi-node training or multi-node inference sporadically, not a permanent reserved island.
- Your team already lives in Modal functions, Volumes, and Queues.
- You want RDMA/NCCL without staffing a fabric specialist for every cloud account.
Prefer a specialist AI cloud or hyperscaler reservation when:
- You need a named rack generation (for example Vera Rubin NVL72) with a multi-year capacity plan.
- Your compliance boundary demands pinned hardware and a private fabric you control.
- Your serving SLOs assume dedicated peers that never compete with a shared serverless pool.
Many serious teams will use both patterns. Burst experiments and evals on Clusters. Park steady-state production on reserved NVL72-class capacity. The failure mode is pretending one decorator replaces a capacity contract.
Operator checklist
- Plan limits. Before promising a 16-node job in standup, read your workspace GPU limits. Modal says large clusters need a conversation.
- RDMA on/off. Confirm whether your job truly needs
rdma=True. Wrong answer wastes money or silently crawls over TCP-class paths if you misconfigure elsewhere — on Modal, trust but verify with a tiny all-reduce canary. - PyTorch/NCCL assumptions. GA automation targets that stack. Other runtimes: ask support before the deadline.
- Placement. Gang scheduling groups by AZ and network ID. Design jobs so a partial place is a hard fail you can retry, not a half-alive cluster.
- Data path. Checkpoints to Volumes, training data via Cloud Bucket Mounts — map IAM and egress costs before the first overnight run.
- Health. RDMA health remediation is part of the pitch. Ask what signals page a human when a NIC is bad versus when your job is just slow.
- Autoscaling multi-node inference. Runway’s quote is about clusters scaled as a unit. Validate cold-start and scale-down behavior for your latency budget.
- Attribution. Customer quotes and 6.4 Tbps ceiling come from Modal’s blog. Prefer your own NCCL tests for any public performance claim you make about your product.
Failure modes
Serverless multi-node fails when people treat it like a pet cluster. Leaving a size=8 job hanging because the training script deadlocks. Assuming InfiniBand verbs are available in every region Modal can place a single GPU. Mixing single-node greedy jobs and gang jobs until the pool fragments and “seconds to cluster” becomes “minutes of pending.” Shipping a decorator sample to production without pinning image, CUDA, and NCCL versions.
Another failure mode is blog-math inflation. Modal’s GLM weight-sync and Llama KV-cache examples explain why RDMA exists. They are not benchmarks of your model. Re-run the arithmetic on your parameter count, precision, and step time.
gVisor + RDMA is a sophistication signal and a debugging surface. When something breaks at the sandbox boundary, your on-call needs Modal’s runbooks, not only nvidia-smi folklore.
How eng leads should trial this week
Do not start with a 64-GPU prestige run. Start with a four-node canary that proves three things: gang place succeeds in one AZ/network domain, rdma=True yields NCCL all-reduce latency in the ballpark you expect versus a known TCP baseline, and teardown actually stops the bill when the function exits. Log container ranks, peer IPs, and NCCL version into your Volume so the next person is not reverse-engineering Slack screenshots.
Second, map your existing Modal habits onto Clusters. If you already checkpoint to Volumes and pull data from Cloud Bucket Mounts, keep that path. The new risk is treating the cluster as an always-on lab notebook. Serverless economics punish forgotten jobs harder than a reserved island you already paid for monthly.
Third, write the “when we escalate to reserved NVL72” rule before the first success. The moment a job becomes latency-sensitive production inference with a contractual SLO, Clusters’ shared pool and plan caps may be the wrong control plane — even if the decorator felt magical in a research sprint. Put that decision in the runbook so product managers do not hear “we scaled on Modal” and assume the production architecture is finished.
Fourth, inventory compliance. Multi-tenant gVisor with upstreamed RDMA is a strong story for startups and AI-native teams. Regulated buyers will still ask where weights live, which subprocessors touch gradients, and whether RDMA health events are visible in their SIEM. Have answers before legal finds the GA blog.
What this piece is not
- Not a Modal pricing teardown with invented unit economics.
- Not an AppSec audit of gVisor RDMA (Arjun if we go deep).
- Not a claim that Clusters replace CoreWeave/hyperscaler reserved NVL72 fleets.
- Not permission to treat customer quotes as CTN-reported independent benchmarks.
Related: CoreWeave puts NVIDIA Vera Rubin NVL72 into production — Cognition first (AI/cloud full). Optional thin wire: Modal Clusters GA Brief.
Sources
- Modal Blog: Peyton Walters, “Modal Clusters are generally available” (2026-10-01). https://modal.com/blog/modal-clusters-generally-available