Evaluations and Observability for Claude Apps
View badge details
About This Course
Build eval harnesses that catch regressions before they ship: test-bank design, LLM-as-judge patterns, tracing + structured logging, cost + latency monitoring, and CI integration for prompt changes.
Course Curriculum
10 Lessons
Why evals for Claude apps
Motivation for systematic evals: catching regressions on prompt changes, model version updates, corpus updates. Test-bank + scoring layer + CI. Zero handwaving.
Build an eval scorecard - Lab Exercises
4-case bank (happy/edge/adversarial) with must_contain + must_not_contain deterministic checks. Reports pass rate.
LLM-as-judge patterns
Design LLM-judge prompts: structured verdicts via tool_use, per-criterion rubrics, cross-validation against human labels, avoiding self-affirmation bias.
LLM-judge harness - Lab Exercises
Score one answer against 3 semantic criteria (relevance, tone, correctness) via LLM-judge with tool_use schema.
Regression testing on prompt changes
Run the eval bank before + after any prompt/model change; block merge on regression via CI.
Regression harness comparing 2 prompt versions - Lab Exercises
Runs 3-case bank against 2 system prompt versions, flags regressions (case passed in A but failed in B).
Tracing and structured logging
Ship trace_id per request, structured JSON logs, and integrate with OpenTelemetry or Datadog for production Claude apps.
Trace logger for Claude calls - Lab Exercises
Wrap Claude calls with structured JSON trace emission (trace_id, tokens, latency, cost, stop_reason).
Cost + latency dashboards
What metrics to plot: P50/P95/P99 latency per agent, cost per request over time, tool-call success rates, token efficiency (output tokens per user request).
Full eval + monitoring stack capstone - Lab Exercises
CLD-AI-109 deliverable: bank + per-case JSON trace + aggregate scorecard (pass rate, P50/P95 latency, cost).
This course includes:
- 24/7 AI Instructor Support
- Live Lab Environments
- 5 Hands-on Lessons
- Completion Badge
Earn Your Badge
Complete all lessons to unlock the Evaluations and Observability for Claude Apps achievement badge.
Evaluations and Observability for Claude Apps
Awarded on completion of CLD-AI-109. The holder can design eval banks, implement LLM-as-judge patterns, add structured tracing + cost/latency metrics, and gate deploys on eval regressions.
Skills You'll Earn
Complete all lessons in this course to earn this badge