Buoyant has released Kinference, an open-source control plane designed to manage AI inference workloads running on the vLLM engine within Kubernetes clusters. The project aims to help IT teams curb GPU waste, enforce cost and compliance boundaries, and selectively extend self-hosted AI infrastructure into the cloud when needed.
What Kinference Actually Manages
vLLM, originally built by researchers at the University of California, Berkeley's Sky Computing Lab, combines an inference server for handling network traffic with an inference engine that uses a technique called PagedAttention to maximize memory throughput. Kinference sits on top of that engine, adding a control layer that coordinates distributed vLLM workloads across a Kubernetes cluster. According to Buoyant CEO William Morgan, this makes it easier for IT teams to scale AI workloads cost-effectively rather than relying solely on manual tuning or default scheduling behavior.
Why Self-Hosting Is Gaining Ground
Morgan points to a broader shift among organizations moving AI workloads into infrastructure they directly control, largely driven by security and compliance concerns. Because AI models often process sensitive data, many enterprises are wary of routing that information through third-party model providers. Self-hosting also changes the cost structure: instead of paying per-token fees to a cloud API, organizations allocate fixed infrastructure resources to specific workloads, which can improve predictability and allow finer control over quality of service for different teams or applications.
Targeting the GPU Utilization Problem
A central motivation behind Kinference is addressing a persistent inefficiency in AI infrastructure: despite ongoing GPU shortages, actual utilization rates are often in the single digits, Morgan said. Kinference attempts to close that gap by using GPU cache data, request queuing information, and other signals to distribute inference requests more efficiently across available hardware. The tool can also attribute GPU consumption to specific tenants, groups, or teams, giving organizations visibility into where infrastructure spending is going and where it might be optimized.
A Hybrid Option for Overflow Capacity
Kinference is not strictly limited to on-premises operation. IT teams can configure it to route selected inference requests to cloud providers under defined conditions, such as when local GPU pools become saturated, when specific tenants require additional capacity, or based on characteristics of the incoming request itself. This gives organizations a measured way to keep sensitive workloads local while still tapping outside capacity during demand spikes.
It remains unclear exactly how many AI workloads currently run on Kubernetes clusters industry-wide, but as that number grows, tools like Kinference reflect an emerging need to manage AI alongside traditional applications on infrastructure that, as Morgan notes, remains fundamentally limited.