Introduction

In recent years, the landscape of AI has undergone dramatic transformation. While ML or AI-powered products were once the domain of a select few companies and teams, the advent of Large Language Models (LLMs) has democratized intelligence, enabling everyone to build AI powered products.

However, AI responses or products are not deterministic and measuring efficacy of these products on various tasks could be a new & challenging territory to navigate.

As someone building LLM-powered products, I am developing mental models and frameworks to help in development and evaluation of AI-powered products

Framework for effective evaluation of AI powered products

Step 1 : Define your task

• Clearly articulate the specific task your LLM-powered product is designed to perform.
• Task : “Generate code snippets as per context, Generate text, classification, sentiment analysis, QnA, conversations bot, etc”

Step 2 : Define quantitative metrics to evaluate response

These aspects of LLM responses can be measured :

• Task Fidelity : How well the model is performing the task :
  • F1 score, Precision, Recall, (Typically classification tasks, etc)
  • BLEU Score : To evaluate quality of translated text against human reference
  • Perplexity : How well a model predicts a sample
  • Custom metrics like “responses contain so and so keywords”