Agentic AI Private

Agent Evaluation and Observability with LangSmith and Langfuse (AGNTDV104)

2 days

Instrument, trace, and evaluate AI agents with LangSmith and Langfuse, using datasets, LLM-as-judge evaluators, CI regression gates, and online feedback.

Talk to a Learning Consultant Request a Private Class

Register or Request Training

Course Overview

Agents are hard to improve because they are hard to see. A failure might come from a prompt, a retrieval miss, a tool error, a routing decision, or a model change made by a vendor last night. Without tracing and evaluation, teams ship on vibes and regress silently.

This course gives students the instrumentation and measurement discipline that makes agent development an engineering activity. Students instrument an existing agent with OpenTelemetry-based tracing, work hands-on in both LangSmith and Langfuse, and build evaluation suites that run offline in CI and online against production traffic. The course covers dataset construction, LLM-as-judge design and its calibration traps, human review workflows, regression gating, and the cost and latency analysis that drives most real optimization decisions.

Students leave with a dashboard, an evaluation suite that blocks bad releases, and an online feedback loop that turns production failures into test cases.

Who should attend: ML and software engineers responsible for LLM application quality, QA and platform engineers, and technical leads who need defensible quality metrics.

Course Benefits

Instrument agents, chains, and retrieval with complete, low-overhead traces
Operate LangSmith and Langfuse, and articulate where each fits
Build evaluation datasets from production traffic, synthesis, and human curation
Write deterministic, statistical, and LLM-as-judge evaluators, and validate judges against human labels
Run offline evaluations in CI with regression gates and release criteria
Run online evaluations, A/B tests, and canary releases on live traffic
Analyze cost, latency, and token usage per step and act on it
Build human review and annotation workflows that feed back into datasets
Monitor drift, alert on quality regressions, and debug incidents from traces

Delivery Methods

Private Class
Delivered for your team at your site or online.

Course Outline

Day 1 — Observability and Offline Evaluation
- Module 1: What Makes Agents Hard to Evaluate
  - Non-determinism, multi-step attribution, and the absence of a single correct answer
  - Quality dimensions: correctness, groundedness, task completion, safety, tone, cost, latency
  - Offline vs. online evaluation; what each can and cannot tell you
  - Building the measurement loop before the optimization work
- Module 2: Tracing and Instrumentation
  - Traces, spans, runs, and sessions; the agent trace model
  - OpenTelemetry and GenAI semantic conventions
  - Instrumenting LangChain/LangGraph, LlamaIndex, and plain SDK calls
  - What to capture: inputs, outputs, tokens, tool calls, retrieved context, metadata, and user IDs
  - Sampling, redaction of sensitive data, and overhead
- Module 3: Datasets
  - Dataset design: coverage, difficulty distribution, and known-hard cases
  - Sources: production traces, synthetic generation, SME curation, adversarial cases
  - Golden answers, reference-free cases, and expected-tool-call traces
  - Versioning datasets and avoiding leakage
- Module 4: Writing Evaluators
  - Deterministic checks: exact match, schema validity, tool-call correctness, latency and cost budgets
  - Retrieval metrics for RAG steps
  - LLM-as-judge: rubric design, pairwise vs. pointwise, position and verbosity bias
  - Calibrating judges against human labels; measuring the judge itself
  - Trajectory evaluation for multi-step agents
Day 2 — Production Quality Loops
- Module 5: Evaluation in CI
  - Running suites on every change; prompt, model, and retrieval changes as versioned artifacts
  - Regression gates, thresholds, and handling noisy metrics
  - Statistical significance with small datasets; repeat runs and variance
  - Comparison experiments and release criteria
- Module 6: Online Evaluation and Feedback
  - Production scorers and sampled online judging
  - User feedback: explicit signals, implicit signals, and their biases
  - Human annotation queues and review workflows
  - Closing the loop: promoting production failures into the dataset
- Module 7: Experiments and Rollout
  - Prompt and model A/B testing on live traffic
  - Canary releases and automatic rollback on quality metrics
  - Model migration: evaluating a new model version before it is forced on you
  - Prompt and configuration versioning and management
- Module 8: Cost, Latency, and Operations
  - Per-step cost and token attribution; finding the expensive node
  - Latency breakdown and the retrieval/rerank/generation budget
  - Caching impact and measuring it honestly
  - Dashboards, alerting on quality and cost drift, and on-call debugging from traces
  - Self-hosting Langfuse; data residency and retention

Class Materials

Each student receives a comprehensive set of materials, including course notes and all class examples.

Class Prerequisites

Experience in the following is required for this Agentic AI class:

Python proficiency
A working LLM application or agent, such as one built in Building AI Agents with LangGraph and LangChain (AGNTDV101), RAG Pipelines with LlamaIndex and Vector Databases (AGNTDV102), or Model Context Protocol (MCP) for Connecting Agents to Enterprise Systems (AGNTDV103), or equivalent experience

Prerequisite Courses

Courses that can help you meet these prerequisites:

Follow-on Courses

Deploying Agents on Kubernetes, Databricks, and Snowflake

Have questions about this course?

We can help with curriculum details, delivery options, pricing, or anything else. Reach out and we’ll point you in the right direction.

Talk to a Learning Consultant