Using NVIDIA Run:ai GPU Fractioning on Amazon EKS
Amazon EKS validated open-source NVIDIA Run:ai GPU Fractioning on an EC2 g5.xlarge node, running two 8 GiB memory-bounded pods on a single A10G accelerator.
By Dillip Chowdary • Nov 20, 2026 • Source: AWS Machine Learning Blog

At a glance
- Product
- NVIDIA Run:ai GPU Fractioning
- Release date
- October 13, 2026
- Validated hardware
- NVIDIA A10G Tensor Core GPU (23,028 MiB)
- Kubernetes version
- Amazon EKS 1.36
- Driver requirement
- NVIDIA R615 or newer (tested 615.71.10)
- Memory limit
- Enforced 8 GiB contract per Pod
In this post
Amazon Web Services published a technical walkthrough detailing how engineering teams can partition an NVIDIA A10G Tensor Core accelerator into memory-bounded allocations on Amazon Elastic Kubernetes Service. According to AWS Machine Learning Blog's report, the setup uses open-source NVIDIA Run:ai components to run two concurrent pods on an Amazon EC2 g5.xlarge instance. Pairing the KAI Scheduler with the GPU Fractioning Operator bypasses standard device-plugin limits that previously restricted single GPUs to one container.
This analysis details the architecture, dependencies, and configuration required to establish enforceable memory boundaries without physical partitioning hardware. The guide is designed for Kubernetes administrators and platform engineers looking to improve cluster density for lightweight artificial intelligence workloads. It examines how custom operating system images and container runtimes enforce memory contracts across shared accelerators.
01GPU sharing mechanisms and memory isolation
Kubernetes clusters traditionally share graphics processing units through Multi-Instance GPU partitioning, device-plugin time-slicing, or Multi-Process Service architectures. Multi-Instance GPU provides hardware-level isolation but requires specific accelerator models. Time-slicing multiplexes workloads across compute cycles without providing memory isolation or preventing out-of-memory crashes across neighboring processes.
NVIDIA Run:ai GPU Fractioning builds upon Multi-Process Service by establishing Kubernetes-integrated scheduling alongside enforceable per-pod memory contracts. The fractioning operator supports either default time-slicing or streaming multiprocessor sharing compute modes. While default mode maintains distinct error domains for separate workloads, streaming multiprocessor sharing uses a combined domain lacking strict fault containment.
| Sharing Approach | Memory Isolation | Compute Mode | Hardware Scope |
|---|---|---|---|
| Multi-Instance GPU | Hardware partition | Hardware sliced | Specific supported GPUs |
| Device Time-Slicing | None | Interleaved cycles | Broad GPU support |
| Multi-Process Service | Process-managed | Concurrent CUDA | Compute capability 3.5+ |
| NVIDIA Run:ai Fractioning | Enforced per-pod limit | Time-slicing or SM sharing | Supported GPUs without MIG |
Organizations choose fractioning when workloads require strict memory boundaries on hardware lacking physical partitioning support. Alternative options such as Dynamic Resource Allocation supply emerging Kubernetes APIs for device assignment. The tested implementation uses NvFractions annotations rather than Dynamic Resource Allocation to bind pods to designated memory allocations.
02Workload suitability and capacity planning constraints
Fractioning primarily benefits interactive development environments, exploratory notebooks with extended idle gaps, and small inference microservices. Short evaluation runs and compatibility verification tests also maximize node efficiency when grouped together. Sizing calculations require teams to benchmark peak memory consumption alongside runtime overhead rather than relying on baseline weights.
Allocating memory boundaries does not guarantee proportional compute performance or prevent contention during simultaneous execution spikes. Higher container density improves hardware utilization metrics without establishing universal throughput gains or cost reductions. Platform teams must validate application latency under full concurrent load before migrating production endpoints to shared accelerators.
Note: Memory reservation contracts strictly enforce address space limits but do not assign dedicated compute slices to individual containers. Increased workload density does not automatically translate into improved request throughput or lower operational expenses, requiring engineers to measure latency under simultaneous peak traffic before running production tasks.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
03Single-node topology and control plane architecture
The verified deployment architecture operates within the US East Ohio Region on Kubernetes version 1.36. The Amazon Elastic Kubernetes Service control plane provisions inside an AWS-managed virtual private cloud, connecting to worker instances in private subnets. Cluster management tools including eksctl, Helm, Packer, and the AWS CLI direct the provisioning pipeline.
A dedicated m7i.large system instance hosts cluster infrastructure, including the KAI Scheduler and GPU Fractioning Operator controllers. The KAI Scheduler places memory-bounded workloads onto a tainted g5.xlarge worker instance housing a single NVIDIA A10G accelerator with 23,028 megabytes of memory. Taints prevent non-accelerated system pods from scheduling onto the GPU node.
The worker node executes containerd version 2.3.5 integrated with the Node Resource Interface to manage container lifecycles. A localized fractiond daemon works alongside Multi-Process Service to establish runtime isolation boundaries between client processes. Simultaneously, a metricsd background daemon captures per-pod allocation telemetry, enabling administrators to verify exact consumption figures against assigned memory contracts.
04Custom Linux image and driver dependencies
Standard Amazon Linux 2023 optimized images for Amazon Elastic Kubernetes Service supply the R580 and R595 driver streams. However, NVIDIA GPU Fractioning Operator version 0.1.7 mandates NVIDIA driver branch R615 or newer. Because R615 represents a short-lived branch not installed by default, the walkthrough builds a custom machine image incorporating required drivers.
The validated software environment combines NVIDIA driver version 615.71.10 with Container Toolkit version 1.20.1-1 on standard Amazon Linux 2023. Cluster orchestration uses GPU Operator version 26.7.1, NVIDIA device plugin version 0.20.1, and KAI Scheduler version 0.18.5. Initial validation concluded on October 22, 2026, and follow-up testing on November 18, 2026, confirmed identical execution stability.
Building the machine image locally ensures host drivers integrate directly with the root operating system rather than running within operator-managed driver containers. Engineers must re-evaluate driver support matrices and CUDA dependencies whenever targeting alternative hardware profiles. A10G accelerators supply sufficient memory headroom for runtime overhead, whereas smaller devices require adjusted memory contracts.
05Environment configuration and disposable cluster prerequisites
Deploying the sample environment requires administrative access across Amazon EKS, CloudFormation, Virtual Private Cloud, EC2, Elastic Block Store, and Systems Manager. Administrators must verify regional service quotas for g5.xlarge instances, m7i.large controllers, and temporary m5.xlarge image builders. Workstation prerequisites include Git, curl, jq, Helm, Packer, kubectl, and AWS CLI version two.
Authentication relies on standard AWS credentials, prioritizing temporary session tokens through AWS IAM Identity Center or assumed roles. Engineers clone the official sample repository, copy the walkthrough environment template to a local configuration file, and restrict permissions to mode 600. Running the initial check script verifies active identity credentials and cluster variables before provisioning infrastructure.
The testing process validates that two pods, each assigned an 8 gibibyte memory contract, successfully allocate 6 gibibytes of GPU memory simultaneously. Subsequent verification confirms that the operator rejects an oversized 9 gibibyte request exceeding the contract boundary. Because the environment provisions billable control planes, gateways, and compute instances, teams should complete cleanup immediately after testing.
Frequently asked questions
What is NVIDIA Run:ai GPU Fractioning?
It is Kubernetes-integrated orchestration software that uses Multi-Process Service and the KAI Scheduler to enforce per-Pod GPU-memory limits. It allows multiple containers to share a single GPU concurrently without requiring Multi-Instance GPU hardware partitioning.
Which NVIDIA driver version is required for GPU fractioning?
The GPU Fractioning Operator version 0.1.7 requires NVIDIA driver branch R615 or newer, which is not the default in standard Amazon Linux 2023 images. The walkthrough validated driver version 615.71.10 paired with NVIDIA Container Toolkit 1.20.1-1.
Does GPU fractioning provide hardware fault isolation or reserved compute?
No, fractioning enforces memory boundaries but does not reserve proportional compute or provide strict hardware fault containment. The default time-slicing mode maintains separate error domains, while streaming multiprocessor sharing uses a shared error domain.
What instance type and GPU were used in the Amazon EKS validation?
The testing environment used an Amazon EC2 g5.xlarge instance equipped with a single NVIDIA A10G Tensor Core GPU containing 23,028 MiB of memory. It successfully ran two Pods with 8 GiB memory contracts allocating 6 GiB each.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement