MB Logo
Home Work Projects Notes Practice About
← All Notes
Essay ·

The Ultimate Guide to AI Evals (in Plain English)

A comprehensive, practical playbook for testing and improving AI systems. Covers observability, error analysis, LLM judges, RAG evaluation, and production guardrails.

Source Material: This playbook distills the core methodology from the industry-leading Maven course “AI Evals For Engineers & PMs” taught by Hamel Husain (25+ years ML engineering, previously Airbnb & GitHub) and Shreya Shankar (PhD UC Berkeley, incoming Carnegie Mellon professor). Discovered via Nikita Fedulov on LinkedIn. If you build AI products, their course is the gold standard.

A quick audio-visual overview generated via NotebookLM to summarize the core concepts before diving into the details:

What Are AI Evals, and Why Should You Care?

Traditional software is deterministic: same input always gives the same output. You type 2 + 2 into a calculator and get 4. Every time. You write a unit test that says “2 + 2 must equal 4” and you’re done.

AI is fundamentally different. AI is non-deterministic: the same question can produce a thousand different valid answers. Ask an AI to write a polite email and it might write something completely different each time — all technically correct. Traditional unit tests simply don’t work here.

Because testing AI is hard, most teams fall back on “vibe checks.” They type a few prompts, read the responses, and if it feels okay, they ship it. Then real users start doing things nobody anticipated: the AI hallucinates facts, leaks private data, or ignores dietary restrictions. When someone fixes the prompt for one bug, they accidentally break ten other things without realizing it.

AI Evals are systematic, automated tests for your AI. They let you confidently say: “My AI follows dietary restrictions 95% of the time, and here’s the data.”

The AI Eval Lifecycle

Everything follows the scientific method. There are no shortcuts:

  1. Observe (Tracing) — Record everything your AI does.
  2. Hypothesize (Error Analysis) — Read the recordings, find what’s breaking.
  3. Experiment (Build Evals) — Build automated tests for those specific failures.
  4. Measure (Metrics) — Track performance with real statistics, not gut feelings.
  5. Iterate (Close the Loop) — Fix the root causes and prevent regressions.

The rest of this guide walks through each phase in detail with practical actions you can implement today.

Phase 1: Observability (Install the Security Cameras)

If you can’t see what the AI did, you can’t evaluate it.

Think of observability like security cameras for your AI. You install a tool that sits in the background and quietly records everything that happens during every user interaction.

Key Concepts

  • Trace: A complete recording of one full user interaction, from the moment the user types something to the moment the AI responds. It captures everything that happened behind the scenes.
  • Span: A single step within a trace. For example, one database lookup, one LLM call, or one tool execution. A trace is made up of multiple spans.

What Gets Captured

Every trace should record:

  • The exact message the user typed
  • The hidden “System Prompt” you gave the AI behind the scenes
  • Every tool the AI tried to use (searching a database, calling an API)
  • The data those tools returned
  • The final response shown to the user
  • How long each step took and how much it cost

Practical Action

Use open-source tools like Arize Phoenix (free, runs locally) or Langfuse (cloud or self-hosted). Both offer “auto-instrumentation” — you add a few lines of code and all your OpenAI/Anthropic calls are automatically logged with a visual UI for browsing traces.

Phoenix Setup (5 minutes):

from phoenix.otel import register
import openai

# Register Phoenix as your trace collector
tracer_provider = register(
  project_name="my-ai-app",
  endpoint="http://localhost:6006/v1/traces",
  auto_instrument=True  # Automatically traces all OpenAI calls
)

client = openai.OpenAI()
# Every call from here is now automatically traced
response = client.chat.completions.create(...)

Langfuse Setup (5 minutes):

# Just swap your import — everything else stays the same
from langfuse.openai import OpenAI
client = OpenAI()

# Automatically traced by Langfuse
response = client.chat.completions.create(...)

Phase 2: Error Analysis (The Secret Sauce)

Do not skip this to build fancy dashboards. This is where you learn what is actually broken.

Most teams jump straight to building automated LLM judges. This is a massive mistake. You cannot automate a fix if you don’t know what the failure modes are. Generic metrics like “helpfulness” won’t catch that your bot is sending Markdown formatting in SMS text messages, or suggesting honey-based recipes to vegans.

Why PMs and QAs Must Lead This

Engineers know if the code runs. Product Managers know if the experience is good. PMs have the domain expertise to spot the difference between a technically correct answer and a genuinely useful one. They know that recommending a “connected bathroom” vs. a “disconnected bathroom” actually matters to the user, even if both answers are technically valid.

The Error Analysis Workflow

Step 1 — Dimensional Sampling (Generate Diverse Scenarios)

Don’t just test your AI with obvious questions. You need to stress-test the edges. Create a list of important user dimensions and combine them:

DIMENSIONS = {
  "dietary_restriction": ["vegan", "keto", "no restrictions"],
  "cuisine_type": ["Italian", "Asian", "Mexican"],
  "skill_level": ["beginner", "advanced"]
}

# This produces diverse test queries like:
# "I'm a total beginner and vegan. Can you suggest an easy Italian dinner—
# "I'm on keto and advanced — give me a complex Mexican brunch."

Step 2 — Open Coding (Read 100 Traces)

Sit down and manually read 100 recorded interactions. Spend no more than 30 seconds per trace. Don’t overthink it. Just scribble quick notes on a scratchpad when you spot something wrong:

  • “User asked for vegan, AI suggested honey.”
  • “Response had weird bold Markdown in an SMS.”
  • “AI forgot the user’s allergy from 2 messages ago.”

Total time: roughly 45 minutes for 100 traces.

Step 3 — Axial Coding (Group the Errors)

Look at your messy notes and group them into 4 to 6 clear failure categories:

  • Dietary Ignored — The AI suggested food that violates the user’s diet.
  • Format Error — The AI used Markdown, emojis, or weird characters in the wrong channel.
  • Context Loss — The AI forgot something the user said earlier.
  • Tone Issue — The AI was rude, too casual, or off-brand.

Tip: Feed your messy notes into an LLM and ask it to group them: “Based on these failure notes, create 4-6 systematic failure labels. Keep them short (2 words max) and distinct.”

Step 4 — Prioritize

Count how many times each category appeared. Fix the most frequent and dangerous errors first. A formatting error might happen 20% of the time (annoying but low severity), while a dietary violation might happen only 5% of the time (rare but potentially dangerous). Fix the dangerous one first.

Phase 3: Building Evaluators (Automating the Checks)

Now that you know exactly how your AI fails, you can build automated tests to catch those failures at scale — so you never have to manually read 10,000 traces again.

There are two types of evaluators. Always try Code-Based first. Only use LLM-as-a-Judge when code can’t handle it.

Type A: Code-Based Evals (Free, Instant, 100% Reliable)

If you can express the check as a simple true/false rule, always use code. It’s free, runs in milliseconds, and never makes mistakes.

Use code-based evals for:

  • Format validation — Is there Markdown in an SMS? Does the response exceed the character limit?
  • Required field checks — Did the AI include the date and time?
  • Tool call validation — Did the AI call the right function with the right parameters?
  • Pattern matching — Does the response contain a phone number, credit card number, or email address it shouldn’t?
import re

def eval_no_markdown_in_sms(trace) -> dict:
    """Check that the AI didn't use bold/italic markdown in an SMS response."""
    response = trace['assistant_message']
    if re.search(r'\*\*.*?\*\*', response):
        return {'passed': False, 'reason': 'Found bold markdown in SMS'}
    if re.search(r'\*.*?\*', response):
        return {'passed': False, 'reason': 'Found italic markdown in SMS'}
    return {'passed': True, 'reason': 'Clean text, no markdown'}

Rule of thumb: Never use an expensive LLM to check if a string contains an email address. That’s what regex is for.

Type B: LLM-as-a-Judge (For Subjective Judgment)

When you need to check something like tone, factual accuracy, policy compliance, or dietary adherence — things that require genuine understanding — you use a powerful AI model (like GPT-4o) to read the trace and act as a “Judge.”

The 7-Step Workflow for Building a Reliable Judge

  1. Generate Traces: Run your AI on your diverse test queries from Phase 2.
  2. Label Ground Truth: A human must manually grade 150-200 of these traces as PASS or FAIL. This is your “answer key.”
  3. Split the Data (Crucial): Divide your graded traces into three piles:
    • Train (15%) — Used to create few-shot examples for your judge prompt.
    • Dev (40%) — Used to iterate and improve your judge prompt.
    • Test (45%) — Touched only once for the final, honest evaluation.
    • If you skip this split, you will accidentally “teach to the test” and your metrics will be fake.
  4. Write the Judge Prompt: Define crystal-clear PASS/FAIL criteria. Use binary scoring (Pass/Fail), never 1-5 scales. Include 2-3 examples from your Train set. (See the template below.)
  5. Validate on Dev Set: Run the judge on the Dev set. Compare its grades to the human grades. If it’s wrong too often, tweak the prompt and repeat.
  6. Final Test: Run the judge exactly once on the Test set. This gives your final, unbiased accuracy numbers.
  7. Scale: Once the judge passes validation, deploy it to automatically grade thousands of production traces every day.

The Judge Prompt Template

The secret trick: force the AI to explain its reasoning BEFORE giving the final grade. If it blurts out “PASS” first, it hasn’t thought about it. If it writes an explanation first, it actually deliberates.

System Prompt for Your Judge:

You are an expert evaluator assessing [Insert Your Domain].

DEFINITIONS:

  • [Define specific terms so the AI doesn’t guess. e.g., “Vegan means absolutely no animal products, including honey, gelatin, and dairy.”]

EVALUATION CRITERIA:

  • PASS: [State exactly what a passing response looks like]
  • FAIL: [State exactly what a failing response looks like]

WHAT DOES NOT COUNT AS A FAILURE:

  • [List acceptable edge cases. Otherwise the judge will be too strict and flag everything.]

EXAMPLES: [Insert 1 clear PASS, 1 clear FAIL, and 1 tricky borderline case from your Train set. Show the full reasoning.]

TASK: Evaluate the following interaction:

  • User Query: {query}
  • AI Response: {response}

Return your evaluation in JSON:

{
  "explanation": "Your step-by-step reasoning here...",
  "label": "PASS" or "FAIL"
}

Phase 4: Metrics and Statistics (The Accuracy Trap)

How to know if your LLM Judge is actually catching failures — or just pretending to.

The Trap Most Teams Fall Into

Most people measure “Agreement” — how often the judge agrees with the human grader. Sounds reasonable, right?

Here’s the problem: If your AI only fails 10% of the time, a completely broken judge that just shouts “PASS!” at everything will have 90% agreement. The dashboard looks great. The metrics look solid. But the judge is completely useless because it never catches a single failure.

The Solution: Two Metrics You Must Track

Instead of overall agreement, measure these separately:

  • TPR (True Positive Rate / Recall): When the AI genuinely did a good job, how often does the judge correctly say PASS? (Target: above 80%)
  • TNR (True Negative Rate / Specificity): When the AI genuinely messed up, how often does the judge correctly catch it and say FAIL? (Target: above 80%)

Both must be above 80%. A judge with 95% TPR but only 22% TNR is dangerous — it looks accurate on paper but misses the vast majority of real failures that slip through to your users.

Statistical Correction with judgy

Even a great judge makes small mistakes. If your judge has a TPR of 95% and TNR of 100%, the raw pass rate it reports is slightly skewed. The open-source judgy Python library takes your judge’s known error rates and mathematically corrects the dashboard number.

Instead of seeing “Pass rate: 92%” you’ll see “We are 95% confident the true success rate is between 84% and 99%.” — which is far more honest and useful for decision-making.

Phase 5: Advanced Evaluation Scenarios

Evaluating RAG (Retrieval-Augmented Generation)

If your AI searches company documents before answering (this is called RAG), it can fail in two completely different ways:

  1. It went to the wrong file cabinet. The search engine pulled up the wrong document. (Retrieval Failure)
  2. It got the right file, but read it wrong. The right document was found, but the AI hallucinated or misinterpreted it. (Generation Failure)

You must evaluate these separately. If retrieval is broken, no amount of prompt engineering will fix the AI’s answers.

Retrieval Metrics:

  • Recall@K: Did the correct document appear somewhere in the top K search results?
  • MRR (Mean Reciprocal Rank): How high did it rank? (1st result is much better than 5th.)

Fixing Retrieval: Try different chunking strategies, add metadata filters, implement hybrid search (BM25 + semantic), or try query expansion. (Note: Standard tokenizers strip numbers, which ruins recipes and code. Consider domain-specific tokenizers.)

Fixing Generation: Improve the system prompt, add few-shot examples, require chain-of-thought reasoning, or implement citation requirements so the AI always points to its source.

Multi-Step Agents and Pipelines

If your AI performs 7 different steps behind the scenes before replying to the user (parse request → plan tools → search database → validate results → compose answer), and the final answer is wrong, you need to know which step dropped the ball.

Write a separate LLM judge for each step. Run your test dataset through the pipeline and chart the failure distribution. You’ll quickly see that maybe Step 3 (search) is fine, but Step 5 (compose answer) is where 80% of failures happen.

Multi-Turn Conversations

Don’t just evaluate single messages in isolation. Test entire conversation histories for:

  • Context Loss: Does the AI remember the user’s dietary restriction from 3 turns ago?
  • Contradiction: Does the AI say one thing in turn 2 and the opposite in turn 5?
  • Instruction Drift: Does the AI slowly forget its system prompt persona over a long conversation?

Production Safety (Online Evals / Guardrails)

Everything discussed above is Offline Evaluation — you’re reviewing recordings of past interactions. For critical safety issues, you need Online Evals (Guardrails) that run in real-time before the response reaches the user.

  • Code-Based Guardrails (under 1ms): Regex scanners that instantly block responses containing credit card numbers, Social Security numbers, or prompt injection patterns like “ignore previous instructions.”
  • Embedding-Based Guardrails (10-50ms): Similarity checks that flag responses drifting too far from expected topics.
  • LLM Guardrails: Use fast, cheap models (like Claude Haiku or GPT-4o-mini) to check for off-topic or harmful content before the response ships.

Phase 6: Closing the Loop

Evals are completely useless if they don’t drive action.

Root-Causing

When an eval flags a failure, ask where it failed. Was it the system prompt? Bad retrieval results? Wrong tool parameters? A model hallucination? Fix the root cause, not the symptom.

Regression Testing

Every time you fix a bug, add that exact failing case to a Regression Suite. Run this suite before every prompt change or model upgrade. This ensures you never accidentally re-introduce old bugs when fixing new ones.

The Golden Rule

The cycle never stops: Run evals → Identify failure → Fix the root cause → Run evals again → Verify improvement → Add to regression suite → Repeat. Teams that close this loop ship dramatically better AI products than teams that just build dashboards and hope for the best.

Your 4-Week Rollout Plan

Here is a realistic plan for going from zero to a working eval system:

Week 1 — Foundation: Install an observability tool (Phoenix or Langfuse). Manually review 100 traces. Categorize the errors into 4-6 failure buckets.

Week 2 — Code Evals: Build 2-3 code-based evaluators for your most frequent objective failures (formatting issues, missing required fields, wrong tool calls).

Week 3 — LLM Judge: Label 150 traces as Pass/Fail. Split them into Train/Dev/Test. Build, validate, and test an LLM-as-a-Judge for your most important subjective failure (e.g., dietary compliance, tone).

Week 4 — Production: Implement real-time safety guardrails (PII detection, prompt injection blocking). Set up automated daily eval runs on a sample of production data. Start fixing root causes identified by your evals. Build your first regression suite.

← All Notes