Beyond Tokens per Second: Measuring Real-World Agentic AI Work with Signal 65’s Pinnacle Benchmark | Utilizing AI Episode 45
Utilizing AI Podcast
•
34m
Standard AI benchmarks measure speed and raw token throughput, but how do you measure whether artificial intelligence is actually completing enterprise work correctly? In this episode of Utilizing AI, co-hosts Stephen Foskett and Brad Shimmin welcome Ryan Shrout, President and GM of Signal 65, to introduce "Pinnacle"—a new benchmark designed to evaluate real-world, agentic workflows. Ryan breaks down why traditional hardware and model metrics miss the mark on business value, how Pinnacle evaluates performance across clean "governed" versus messy "as-found" data environments, and what recent test runs reveal about token efficiency, hallucination resistance, and hardware scaling. From comparing frontier models to exposing the hidden costs of max-reasoning modes, this discussion offers a grounded look at quantifying true enterprise AI performance. This and more on Utilizing AI, part of Futurum Media and the Futurum Podcast Network.
Up Next in Utilizing AI Podcast
-
Outpacing Hardware: GPU Memory Bottle...
The hardware needed for artificial intelligence applications today may not be a match for tomorrow's needs.
On this episode of Utilizing AI, host Stephen Foskett, Frederic Van Haren of HighFens and Brad Shimmin of Futurum discuss the reality of AI hardware as models grow and needs evolve. Hardwa...
-
Open-Weights, Data Sovereignty, and L...
When cutting-edge AI capabilities shift overnight from exclusive enterprise clouds to permissive open-weights models accessible to anyone, IT leaders are left navigating a chaotic landscape of unprecedented power and heightened security risks.
In this episode of Utilizing AI, hosts Stephen Fosk...
-
Navigating the Model Deluge: How Ente...
Enterprise IT is currently facing a dizzying explosion of new AI models, frameworks, and platforms—making integration, rather than security or governance, the primary roadblock to actual business success. In this episode of Utilizing AI, host Stephen Foskett is joined by Futurum Group analysts Br...
