Full course coming soon

Building and Running AI System Evaluations

7 chapters · 31 lessons

1. 1. Defining Evaluation Objectives and Success Criteria

   4 lessons

   Learn to translate business requirements and user needs into testable evaluation objectives.

2. 2. Building Representative Test Datasets

   4 lessons

   Create test data that reflects real usage patterns, distribution shifts, and adversarial inputs.

3. 3. Designing and Implementing Evaluation Harnesses

   5 lessons

   Build the infrastructure to run controlled, repeatable tests across different system configurations.

4. 4. Selecting and Applying Scoring Criteria

   5 lessons

   Choose appropriate metrics and judgment methods for different evaluation objectives.

5. 5. Evaluating Agentic and Multi-Step Systems

   4 lessons

   Extend evaluation methods to systems that take actions, use tools, and operate over time.

6. 6. Producing Reproducible and Inspectable Reports

   4 lessons

   Document evaluation methodology, results, and limitations in ways that support decision-making.

7. 7. Integrating Evaluation into Product and Operations

   5 lessons

   Use evaluation to support release gates, vendor selection, regression testing, and incident response.