The open AI-native stack on Kubernetes: From open models to distributed

Google Cloud is introducing a unified open-source AI-native stack for Kubernetes and Google Kubernetes Engine at KubeCon + CloudNativeCon North America 2026, for AI workloads moving from isolated research prototypes into large-scale, multi-tenant production fleets. Google Cloud Blog's report by Abdelfettah Sghiouar, Cloud Developer Advocate at Google Cloud, dated November 10, 2026, includes sample manifests teams can try on their own Kubernetes clusters, plus live demos and technical sessions, a decade after Kubernetes became the standard for orchestrating microservices and containerized applications.

This piece covers four production hurdles, then OpenRL, llm-d 1.0, Ray on GKE with KubeRay, Agent Substrate, gVisor, and TorchTPU. It is for platform engineers, ML researchers, and AI builders who want high-performing open-weight models such as the Gemma family, adapted to proprietary enterprise datasets with reinforcement learning, and served with predictable latency, high throughput, and maximum hardware efficiency.

01DolphinGemma and four production hurdles

Production AI is moving beyond generic calls to closed, black-box APIs, and organizations place long-term competitive advantage in domain specialization. The cases are DolphinGemma, an LLM that helps scientists study how dolphins communicate, trained on acoustic data; legal platforms evaluating complex contracts; and engineering teams building autonomous coding agents. The objective is frontier-level task performance at a fraction of the serving cost and latency.

Scaling reinforcement-learning post-training, stateful distributed inference, and autonomous agent execution on traditional containers still hits four hurdles. Accelerator underutilization appears when training and rollout generation alternate, so expensive GPUs and TPUs stay idle for most of the training run. Cache-blind load balancing follows: standard Kubernetes round-robin and least-connections routing scatters multi-turn agent calls across arbitrary pods and wipes out key-value cache locality.

Bespoke Ray or Slurm clusters running beside Kubernetes split capacity and multiply maintenance toil. That split is the framework-fragmentation hurdle. Agent security in multi-tenant clusters is the fourth: untrusted, model-generated shell commands and scripts in a shared environment expose the cluster to sandbox escapes and privilege escalation.

02Sandbox projects plus TorchTPU

Google Cloud is introducing and expanding the stack on Kubernetes and Google Kubernetes Engine. OpenRL, a CNCF Sandbox project, is a Kubernetes-native post-training service that simplifies reinforcement-learning post-training and maximizes accelerator utilization. llm-d 1.0, also CNCF Sandbox, is an open distributed inference control plane for prefix-cache-aware routing, disaggregated serving, and multi-tier key-value cache management.

Ray on GKE with KubeRay, a CNCF Sandbox project, brings Kueue-driven scheduling to distributed Ray compute. Agent Substrate, CNCF Sandbox, is an open orchestration runtime for lifecycle management and execution targets on multi-turn AI agent workflows. gVisor, CNCF Sandbox, is a kernel-level container sandbox that delivers strong system call isolation when untrusted agent code runs in multi-tenant clusters.

TorchTPU gives upstream vLLM and SGLang first-class, native PyTorch support on Google Cloud TPUs and GPUs, with day-zero parity across TPU v5e, v6e, and v7x alongside NVIDIA GPUs. Teams stay on those upstream engines, without private forks or rewritten pipelines.

03Four primitives and delta weights

Supervised fine-tuning is now standard practice, but domain excellence in multi-step reasoning, mathematical proofing, and autonomous tool use requires reinforcement learning. ML researchers want to iterate rapidly in Python and PyTorch with standard reward functions, while platform teams need multi-tenant governance, quota enforcement, and predictable scheduling. Google created OpenRL and contributed it to the CNCF Sandbox to bridge that experimentation with production Kubernetes fleets.

Researchers write standard Python around four primitives: forward_backward, optim_step, sample, and save_state. OpenRL translates those primitives into Kubernetes scheduling workflows without requiring developers to learn Kubernetes internals or manage infrastructure code. Cooperative accelerator time-slicing interleaves training steps and rollout inferences on the same physical accelerators, covering compute-bound training and latency-sensitive rollout generation.

Shipping full multi-gigabyte model weights over cluster networking after every optimization step creates massive network saturation. OpenRL transmits only parameter deltas to rollout workers, which eliminates rollout cold starts. Zero-config dynamic resource allocation automatically profiles context lengths, batch dimensions, and LoRA versus full fine-tuning, then dynamically bin-packs workloads across a heterogeneous accelerator fleet with native Kubernetes DRA claims.

04PrefixCacheAware pool and KV offload

Under interactive, bursty agent traffic, standard ingress controllers treat LLM requests as stateless HTTP, scatter sequential turns across replicas, and constantly rebuild key-value cache. Google Cloud, Red Hat, IBM Research, CoreWeave, and NVIDIA co-founded llm-d, with contributions from over 500 engineers across 25+ organizations, and llm-d 1.0 is the CNCF Sandbox control plane that coordinates engines such as vLLM and SGLang into one cluster-wide serving fabric.

The sample InferencePool uses apiVersion inference.networking.x-k8s.io/v1alpha1, name gemma-serving-pool, namespace ai-serving, and a targetRef to the apps Deployment vllm-gemma-27b. Scheduling strategy is PrefixCacheAware with fallback LeastQueueDepth, and kvCacheSyncInterval is 50ms, so multi-turn requests reach replicas that already hold matching key-value context. The Endpoint Picker integrates into the Kubernetes Gateway API Inference Extension, GKE manages it natively as GKE Inference Gateway, and it reads real-time key-value cache residency across worker pods, doubling key-value cache hit rates from 35% to 70% and cutting median time-to-first-token by 70-85% on multi-turn agent workloads.

When high-bandwidth memory nears capacity, llm-d cascades key-value cache to host CPU RAM, then local NVMe SSD, Google Cloud Managed Lustre, or Cloud Storage, avoiding a context recalculation on long-context calls. DisaggregatedSet, built on Kubernetes LeaderWorkerSet, decouples Encoding, Prefill, and Decode onto specialized hardware pools over high-speed interconnects, with independent horizontal scaling and coordinated zero-downtime rolling upgrades. llm-d-async-processor pulls latency-tolerant tasks from Pub/Sub or Redis into slack capacity at strict priority so interactive sessions never degrade, and Prism, the benchmarking dashboard, tracks throughput, inter-token latency, time-to-first-token, and cache hit ratios.

05RayCluster, RayJob, and RayService

Inference and reinforcement-learning loops have specialized runtimes, while distributed data preparation, embedding generation, and distributed PyTorch training depend heavily on Ray. KubeRay is an open-source, Kubernetes-native operator for the complete lifecycle of Ray clusters, jobs, and services. Those objects are the RayCluster, RayJob, and RayService custom resources.

On Google Kubernetes Engine, Ray on GKE with KubeRay, a CNCF Sandbox framework, adds enterprise lifecycle management, elastic autoscaling, and Kueue-driven scheduling for distributed Ray compute and data processing. That scheduling is how RayCluster, RayJob, and RayService run on GKE capacity.