The topic the rest of this course rests on. An evaluation is to an AI product what a test suite is to ordinary software: the thing that lets you change a prompt, a model or a retrieval setting and know whether you made it better or worse. Teams without one ship on intuition, and intuition does not notice a four point regression.
| # | Lesson | The question it answers |
|---|---|---|
| 01 | Why You Cannot Unit Test a Model | What replaces an assertion, and what does that cost? |
| 02 | Building the Dataset | Why thirty real cases beat five thousand synthetic ones |
| 03 | Graders: Code, Judge and Human | Which grader for which claim, and what is each one blind to? |
| 04 | A Judge That Agrees With Humans | How do you validate the judge before trusting its numbers? |
| 05 | The Statistics You Cannot Skip | Confidence intervals, repeated runs, and reading flipped cases |
| 06 | The Release Gate | What has to pass before a prompt, a model or a setting ships? |
| 07 | Online Evals and Reading Traces | What do you measure on live traffic, and how often do you read it? |
| 08 | Evals as the Product Artifact | Why the eval set, not the requirements document, is the real specification |
| 09 | Evaluating Retrieval Separately | Why a single end to end score hides the actual fault |
| 10 | Evaluating Agents on End State | Why grading the path punishes better solutions |
| 11 | Keeping an Eval Set Alive | How does it avoid becoming a museum of last year’s bugs? |
| 12 | Worked: An Eval Suite From Nothing | One real feature, from zero cases to a release gate |