Self-Hosted Inference Brings Service Mesh Lessons to AI
15m
Self-hosted inference is moving from experiment to strategy. Open weight models are close enough to frontier models that many teams no longer need to send every request out. Running them on your own GPUs brings new infrastructure problems.
Buoyant Co-founder and CEO William Morgan examines that shift with Alan Shimel at KubeCon + CloudNativeCon in Salt Lake City.
From Service Mesh to Inference
Buoyant created Linkerd, the first service mesh, in 2016. It was one of the earliest projects accepted into CNCF and the first service mesh to graduate. Buoyant sells an enterprise distribution for companies that need extra guarantees.
Its new open source project, Kinference, is a control plane on top of vLLM. Once a team runs more than one vLLM worker, it needs load balancing and routing. Those are the same problems Linkerd has solved for a decade, now applied to generative AI.
Morgan notes that inference requests are HTTP at the base. Yet latencies and routing decisions are completely different from traditional traffic.
Why Run Your Own GPUs
Buoyant runs inference internally on its own GPUs. Morgan says it is economically efficient and gives the company full control. It also lets the team try the latest open weight models without worrying about token costs.
He expects AI adoption to resemble multi-cloud. Organizations will keep models and data on premises and use frontier models for perhaps 10% of their work.
Balancing Two Open Source Projects
Linkerd depends on Buoyant to fund many of its maintainers. Adding a second, unrelated project means double the resources. Morgan's confidence comes from serving the same audience of Kubernetes platform owners.
He says discipline and a clear framework for investment are essential. Even quarterly planning now feels like a stretch.
There is no commercial version of Kinference yet. Buoyant's focus is helping companies move their inference spend onto open weight models and GPUs they control. Morgan sees that transition to self-hosted inference as the real opportunity.