Self-hosted inference is moving from experiment to strategy. Open weight models are close enough to frontier models that many teams no longer need to send every request out. Running them on your own GPUs brings new infrastructure problems.
Buoyant Co-founder and CEO William Morgan examines that shift with Alan Shimel at KubeCon + CloudNativeCon in Salt Lake City.
From Service Mesh to Inference
Buoyant created Linkerd, the first service mesh, in 2016. It was one of the earliest projects accepted into CNCF and the first service mesh to graduate. Buoyant sells an enterprise distribution for companies that need extra guarantees.
Its new open source project, Kinference, is a control plane on top of vLLM. Once a team runs more than one vLLM worker, it needs load balancing and routing. Those are the same problems Linkerd has solved for a decade, now applied to generative AI.
Morgan notes that inference requests are HTTP at the base. Yet latencies and routing decisions are completely different from traditional traffic.
Why Run Your Own GPUs
Buoyant runs inference internally on its own GPUs. Morgan says it is economically efficient and gives the company full control. It also lets the team try the latest open weight models without worrying about token costs.
He expects AI adoption to resemble multi-cloud. Organizations will keep models and data on premises and use frontier models for perhaps 10% of their work.
Balancing Two Open Source Projects
Linkerd depends on Buoyant to fund many of its maintainers. Adding a second, unrelated project means double the resources. Morgan's confidence comes from serving the same audience of Kubernetes platform owners.
He says discipline and a clear framework for investment are essential. Even quarterly planning now feels like a stretch.
There is no commercial version of Kinference yet. Buoyant's focus is helping companies move their inference spend onto open weight models and GPUs they control. Morgan sees that transition to self-hosted inference as the real opportunity.
Up Next in KubeCon 2026 Salt Lake City
-
AI Agents Prove System Reliability Be...
System reliability has always depended on finding weaknesses before customers do. AI is now changing how that work gets done. Teams can test, fix and verify faster than ever.
Gremlin Co-founder and CEO Kolton Andrus examines that shift with Alan Shimel at KubeCon + CloudNativeCon in Salt Lake Ci...
-
AI Agents Deserve the Same Accountabi...
Agent accountability is quickly becoming a business problem, not just an IT one. AI agents now work across finance, operations, sales and product teams. Leaders need a way to govern them that everyone can understand.
Gravitee's Tim Tyler examines that challenge with Alan Shimel at KubeCon + Clou...
-
Cloud Native Maturity Signals Who Is ...
Cloud native developers are now more than 21 million strong. New research suggests they are also leading the way on AI. The gap with everyone else is hard to ignore.
SlashData's Liam Bollmann-Dodd and CNCF's Bob Killen examine that link with Alan Shimel at KubeCon + CloudNativeCon in Salt Lake C...