Beyond Tokens per Second: Measuring Real-World Agentic AI Work with Signal 65’s Pinnacle Benchmark | Utilizing AI Episode 45
Utilizing AI: The Podcast about the Practical Applications of AI
•
34m
Standard AI benchmarks measure speed and raw token throughput, but how do you measure whether artificial intelligence is actually completing enterprise work correctly? In this episode of Utilizing AI, co-hosts Stephen Foskett and Brad Shimmin welcome Ryan Shrout, President and GM of Signal 65, to introduce "Pinnacle"—a new benchmark designed to evaluate real-world, agentic workflows. Ryan breaks down why traditional hardware and model metrics miss the mark on business value, how Pinnacle evaluates performance across clean "governed" versus messy "as-found" data environments, and what recent test runs reveal about token efficiency, hallucination resistance, and hardware scaling. From comparing frontier models to exposing the hidden costs of max-reasoning modes, this discussion offers a grounded look at quantifying true enterprise AI performance. This and more on Utilizing AI, part of Futurum Media and the Futurum Podcast Network.
Up Next in Utilizing AI: The Podcast about the Practical Applications of AI
-
Outpacing Hardware: GPU Memory Bottle...
The hardware needed for artificial intelligence applications today may not be a match for tomorrow's needs.
On this episode of Utilizing AI, host Stephen Foskett, Frederic Van Haren of HighFens and Brad Shimmin of Futurum discuss the reality of AI hardware as models grow and needs evolve. Hardwa...
-
Open-Weights, Data Sovereignty, and L...
When cutting-edge AI capabilities shift overnight from exclusive enterprise clouds to permissive open-weights models accessible to anyone, IT leaders are left navigating a chaotic landscape of unprecedented power and heightened security risks.
In this episode of Utilizing AI, hosts Stephen Fosk...
-
Demystifying AI: How Open-Weights and...
What if the key to stripping away apocalyptic hyperbole and demystifying artificial intelligence lies in putting open-weights models directly into the hands of everyday creators worldwide? In this episode of Utilizing AI, host Stephen Foskett and guest Guy Currier explore how the shift toward ope...
