Podcast: Distributed LLM Inference on Kubernetes with llm-d, Wasm

Red Hat engineer Edoardo Vacchi and his team developed llm-d, a distributed inference orchestration system within the vLLM family designed to schedule, route, and balance multi-node large language model workloads on Kubernetes clusters, as detailed in InfoQ's report. The initiative tackles the fundamental mismatch between traditional cloud-native traffic routing and memory-bound neural network execution.

Vacchi outlined how standard Kubernetes ingress controllers fail during real-world AI inference because conventional load balancers disregard GPU memory states, key-value caches, and prefix structures across compute pools. Alongside llm-d, Vacchi highlighted developments spanning backend WebAssembly runtime sandboxing and hardware acceleration systems like TornadoVM that execute compiled Java workloads directly on GPU backends.

This breakdown covers how llm-d expands single-node engine efficiencies across multi-node Kubernetes infrastructure, coordinates cache-aware scheduling, handles idle-period batch execution, and classifies queries for cost-effective model routing. It is written for platform architects, Kubernetes infrastructure engineers, and machine learning operations teams tasked with deploying production language models without exhausting scarce enterprise hardware. As clusters face higher concurrent user demand and lengthening conversational contexts, managing multi-node memory states dictates whether expensive GPU investments operate at peak throughput or stall on duplicate prompt computation.

01What actually changed

Deploying language models across standard container orchestration historically relied on traditional load balancers like round-robin or least-connections routers to distribute incoming HTTP requests. These standard network routing mechanisms break model efficiency because they treat model instances as interchangeable, stateless application servers.

While single-instance runtimes like vLLM introduced PagedAttention to optimize memory consumption and high-throughput execution on individual machines, deploying multiple nodes inside Kubernetes reintroduced severe bottlenecks. Standard ingress layers distribute follow-up turns of a multi-turn conversation arbitrarily across available nodes, forcing separate instances to repeatedly parse identical prompt prefixes from scratch.

To solve this distribution breakdown, the llm-d project establishes a distributed architecture across Kubernetes nodes specifically engineered for model memory mechanics. Instead of handling inference requests as disconnected network calls, the system synchronizes scheduling across instances while preserving node-level memory advantages.

It monitors key-value cache locality and shared prefix allocations across all participating GPUs in the cluster. This infrastructure shift converts isolated single-node inference engines into a coordinated multi-node compute collective capable of serving sustained user traffic without continually clearing and reloading attention state data across pod replicas.

02How it works

At runtime, llm-d coordinates traffic routing by inspecting incoming prompt structures and mapping them against existing node cache states. The scheduler evaluates KV caches and prefix caches to identify which worker pod already retains the memory tensors for an ongoing dialogue or shared system prompt.

Rather than randomly dispatching requests, llm-d attempts to route subsequent conversation turns to the exact GPU instance that retains the cached context, avoiding costly attention recalculation phases. At the same time, the scheduling algorithm tracks per-instance load limits to prevent individual worker nodes from degrading into resource starvation while maintaining cache stickiness.

Beyond interactive conversational routing, llm-d incorporates opportunistic batch scheduling and query classification mechanisms to extract maximum work from hardware allocations. During cluster idle periods when real-time conversational traffic drops, the system triggers scheduled batch inference runs to process background analytical tasks and asynchronous jobs.

It also applies semantic classification algorithms to analyze prompt complexity as queries arrive at the ingress layer. The router classifies the incoming prompt and sends simpler requests to compact, cost-effective models, reserving massive model checkpoints for complex queries while preserving data privacy by containing routing decisions entirely within the private cluster boundary.

03Why it matters

Accelerated computing hardware represents one of the largest capital expenses in modern infrastructure, making low GPU utilization an unsustainable operating cost for enterprise teams. When Kubernetes routes a prompt to an arbitrary node that lacks the precomputed attention state, the GPU wastes compute cycles reprocessing identical input tokens before producing the first output token. Vacchi pointed out that while single-node runtimes resolved memory fragmentation inside individual cards, production platforms require orchestrating hundreds of gigabytes of distributed cache memory across multiple physical servers to keep tensor cores fully saturated.

The system addresses this operational boundary by combining memory-aware ingress with multi-tier execution technologies like WebAssembly sandboxes and TornadoVM hardware translation layers. TornadoVM enables developers to compile Java code directly into GPU kernels, demonstrating systems such as a GPU-accelerated Llama 3 implementation written entirely in Java.

Meanwhile, Wasm backend runtimes offer lightweight process isolation, cross-language compilation from languages like Rust, and frictionless workload migration between browser clients and server nodes. Integrating these modular runtime techniques with distributed orchestration ensures enterprise clusters run compute tasks on the most efficient available hardware without leaving processor capacity idle.

04Who is affected

Platform engineers and site reliability teams managing machine learning deployments on Kubernetes directly encounter the architectural shift introduced by llm-d. Teams migrating proof-of-concept AI features into high-volume production face steep performance drop-offs if they rely on standard ingress controllers, ingress-nginx, or service mesh proxies designed for stateless web microservices. Adopting distributed inference coordination forces infrastructure operators to re-evaluate their networking, autoscaling thresholds, and pod distribution budgets around GPU memory footprints rather than raw CPU or HTTP connection metrics alone.

Software developers designing agentic applications, multi-agent frameworks, and local-first systems are similarly impacted by these distributed execution patterns. Agentic workflows generate expansive, multi-turn contexts and execute small, repetitive automation tasks that require strict memory isolation. The maturation of backend WebAssembly runtimes allows developers to sandbox autonomous agent processes, move application code dynamically between client devices and cloud servers, and execute small language model inference directly on host CPUs or browsers, freeing dedicated GPU clusters to handle distributed heavy-weight inference pipelines orchestrated through llm-d.

05What to watch

Infrastructure teams must monitor the ongoing alignment between the upstream vLLM ecosystem and llm-d development as multi-node serving standards mature across the open-source community. Key technical milestones include tracking how llm-d refines prefix cache tracking across expanding node topologies, measures cross-instance cache eviction latency, and balances sticky cache routing against dynamic pod autoscaling events. Operators should also observe how semantic classification rules adapt to evolving workloads without creating latency spikes at the ingress proxy layer during high-volume traffic bursts.

Engineers should simultaneously follow developments in heterogeneous runtime environments bridging CPU and accelerator execution boundaries. Research into translating WebAssembly instructions directly into GPU kernels, along with continued TornadoVM work bringing native hardware acceleration to JVM-based microservices, points toward hybrid serving topologies. As enterprise Kubernetes platforms incorporate both browser-edge inference and central multi-GPU clusters, the balance between client-side execution, CPU-based sandboxing, and coordinated multi-node GPU inference will define the cost profile of enterprise artificial intelligence deployments.