This is second in the series of articles I plan to write about my learnings of Building an AI product, defining evals, iterating through solutions and finally demonstrating impact on the customer & business. If you have not read the first article, start here.

<aside> đź’ˇ

First, why evals?

AI is non-deterministic but your product and customers who are using it need “reliability” in doing their tasks! Evals are a critical part of AI product development and Product Managers should be the internal champions of rigorous evals!

It is a systematic way to measure and communicate AI product quality to your customers and teams.

How to do Evals

CleanShot 2025-08-09 at 16.31.53@2x.png

Example : How I did Evals for Developer Copilot, a coding agent