Kubernetes for AI Workloads: Why K8s Became the AI Operating System
Something significant happened in the 2024 CNCF Annual Survey results that did not get enough attention. The survey found that 54% of organizations are running AI and machine learning workloads on Kubernetes. This number, combined with the finding that 80% of organizations using containers now run Kubernetes in production (up from 66% in 2023), tells a clear story: Kubernetes is no longer just the platform for microservices and web applications. It has become the operating system for AI.
The same survey found that 98% of organizations run data-intensive workloads on cloud-native platforms. AI inference, model training, data pipelines — the workloads that define the current era of enterprise computing — are running on the same infrastructure platform that runs everything else. This convergence is not accidental. It is the result of Kubernetes solving a set of problems that AI workloads desperately need solved.
Why AI Workloads Landed on Kubernetes
AI workloads have characteristics that make them particularly difficult to manage without a robust orchestration platform. They are resource-intensive, requiring GPUs and large memory allocations. They are heterogeneous, often combining different hardware types in a single pipeline. They are bursty, with inference demand that can spike unpredictably. And they are increasingly composed of multiple communicating services — model servers, embedding services, vector databases, orchestration layers — that need to be deployed, scaled, and managed as a system.
Kubernetes addresses all of these requirements through capabilities that already existed for general-purpose workloads but translate directly to AI use cases.
GPU Scheduling: Treating Accelerators as First-Class Resources
The most obvious requirement for AI workloads is GPU access. Kubernetes supports GPU scheduling through the device plugin framework, which allows hardware vendors to expose their devices as schedulable resources. The NVIDIA GPU Operator, for example, automates the deployment of GPU drivers, container runtimes, and device plugins across a Kubernetes cluster.
In practice, this means you can request GPUs in your pod specification the same way you request CPU and memory. The scheduler handles placement, ensuring your model serving pod lands on a node with the right GPU type and available capacity. For clusters with mixed GPU types — A100s for training, T4s for inference, L4s for cost-optimized serving — node labels and affinity rules direct workloads to the appropriate hardware.
Multi-instance GPU (MIG) support allows a single physical GPU to be partitioned into isolated instances, each with dedicated compute and memory. This is particularly valuable for inference workloads where a full GPU is overkill — you can serve multiple models on a single GPU with hardware-level isolation between them.
Model Serving: vLLM, SGLang, and the New Inference Stack
The model serving landscape has evolved rapidly. Two years ago, serving a large language model required significant custom engineering. Today, projects like vLLM and SGLang have made high-performance LLM inference accessible as deployable services that run natively on Kubernetes.
vLLM implements PagedAttention, a memory management technique that dramatically improves throughput by efficiently managing the KV cache during inference. It supports continuous batching, allowing new requests to be added to an ongoing batch without waiting for all current requests to complete. The result is 2-4x higher throughput compared to naive serving approaches.
SGLang takes a different approach, using a radix tree-based prefix caching system that is particularly effective for workloads with shared prefixes — common in agentic systems where multiple requests share the same system prompt and tool definitions. For agentic AI workloads where the same agent framework is handling multiple concurrent requests, prefix caching can reduce time-to-first-token significantly.
Both projects deploy as standard container images on Kubernetes. Horizontal pod autoscaling can scale inference replicas based on GPU utilization, queue depth, or request latency. The same Kubernetes primitives that scale web applications scale model servers.
Resource Management: The Cluster as a Shared AI Platform
One of the most practical advantages of running AI on Kubernetes is resource management across teams and workloads. In many organizations, GPU resources are scarce and expensive. Without a shared platform, teams end up with dedicated GPU servers that are idle 80% of the time and overloaded the other 20%.
Kubernetes resource quotas and limit ranges allow platform teams to allocate GPU capacity across namespaces — the data science team gets a quota for experimentation, the production inference workloads get guaranteed capacity, and burst capacity is shared across teams based on priority classes. When a training job finishes and releases GPUs, those resources are immediately available for other workloads. No tickets, no manual reallocation, no idle hardware.
Priority-based preemption adds another layer. Production inference workloads can be assigned higher priority than development and experimentation workloads. If the cluster is at capacity and a production workload needs to scale, Kubernetes can preempt lower-priority workloads to free resources. This is not a theoretical capability — it is the mechanism that makes shared GPU clusters practical.
Autoscaling for AI
Kubernetes autoscaling was designed for web traffic patterns, but the same mechanisms work for AI workloads with appropriate metrics. The Horizontal Pod Autoscaler (HPA) can scale based on custom metrics exposed by model servers — pending request queue depth, average inference latency, GPU memory utilization, or tokens-per-second throughput. KEDA (Kubernetes Event-Driven Autoscaler) adds support for scaling based on external signals like message queue depth, allowing inference services to scale to zero when idle and spin up on demand.
For AI workloads, scale-to-zero is transformative. GPU resources are expensive, and not every model needs to be hot at all times. A rarely used specialized model can be loaded on demand, serve a burst of requests, and then release its GPU allocation. The cold-start penalty for loading a model is measured in seconds — acceptable for many use cases and dramatically better than paying for idle GPU capacity around the clock.
The Agentic AI Layer
Agentic AI adds a layer of complexity beyond model serving. An agentic system is not a single model responding to single requests. It is an orchestration layer that manages multi-step workflows, coordinates multiple model calls, integrates with external tools and APIs, and maintains state across interactions. This is, fundamentally, a distributed systems problem — exactly the kind of problem Kubernetes was built to solve.
An agentic AI deployment on Kubernetes typically includes:
- Orchestration services: Stateful or stateless services that manage agent workflows, routing, and policy enforcement.
- Model serving endpoints: One or more model servers handling inference requests from the orchestration layer.
- Tool execution sandboxes: Isolated containers that execute tool calls with scoped credentials and network policies.
- Supporting services: Vector databases for RAG, message queues for async workflows, observability infrastructure for monitoring and tracing.
Kubernetes manages the lifecycle of all these components as a unified system. Service discovery connects them. Network policies isolate them. Resource quotas bound them. Rolling updates deploy new versions without downtime. This is not a novel architecture pattern — it is the same microservices architecture that has been proven at scale, applied to AI workloads.
Why Our Platform Is K8s-Native
Our platform is built on Kubernetes because the alternative — building a custom orchestration platform — would mean reinventing scheduling, service discovery, secret management, autoscaling, network policy, and a hundred other capabilities that Kubernetes already provides. That is not a productive use of engineering effort. Every platform capability — from the OATS tool synthesis engine to the SmartModelRouter — runs as Kubernetes-native workloads.
Code Mode, our platform's browser-based development environment, exemplifies this approach. It runs VS Code in the browser with AI pair programming, where each user session executes inside a dedicated Kubernetes pod with resource limits, network policies, and scoped credentials. The same K8s isolation primitives that protect production workloads protect the development environment. There is no separate security model for development versus production.
Multi-Cloud Management is native because Kubernetes is the abstraction layer. Our platform provisions and manages AI infrastructure across AWS, Azure, and GCP through a unified interface. The same Helm charts work on EKS, GKE, AKS, bare-metal clusters, and k3s installations. There are no cloud-specific dependencies baked into the platform. If you can run kubectl apply, you can run our platform. For air-gapped and on-premises deployments, the entire platform runs without any external connectivity requirements.
Our platform integrates with the Kubernetes ecosystem rather than replacing it. Secrets are managed through Kubernetes secrets or external secret operators. The MCP Workshop deploys MCP servers as standard K8s deployments with hot-reload and zero-downtime rollouts. Monitoring uses OpenTelemetry and standard Kubernetes metrics. Network policies use standard Kubernetes NetworkPolicy resources. Teams do not need to learn a new operational model — they use the same tools and practices they already use for every other workload on their cluster.
The Convergence
The CNCF survey data paints a picture of convergence. AI workloads are not a separate category of infrastructure anymore. They are workloads, running on the same platform as everything else, managed by the same teams, using the same tools.
This convergence is good for organizations. It means AI infrastructure does not require a separate team with separate tooling and separate expertise. It means the investments you have already made in Kubernetes — the clusters, the monitoring, the CI/CD pipelines, the team expertise — apply directly to your AI workloads. It means portability, because a Kubernetes workload runs the same way everywhere.
The organizations that recognized this early are the ones running production AI on their existing Kubernetes clusters today. The ones that treated AI as a special snowflake — separate infrastructure, separate tools, separate operational model — are the ones struggling with fragmented infrastructure, duplicated effort, and the operational burden of managing two distinct platforms.
Kubernetes did not become the AI operating system because anyone planned it that way. It became the AI operating system because it was already the operating system for everything else, and AI workloads turned out to need the same things every other workload needs: scheduling, scaling, isolation, observability, and portability.