Technical Talk: Testing for Failure: Evaluating AI Agents

#security #ai #testing #failure-mode #gain #defense
Share

Tech Talk : Testing for Failure: Evaluating AI Agents with Nate Summers


An AI agent can respond differently to the same input across runs, and changing its model or its instructions can change which tests it passes. That makes it a testing problem, and this talk treats it as one. It introduces evaluations ("evals") for AI systems using Inspect, an open-source framework for evaluating AI models and agents developed by the UK AI Security Institute and Meridian Labs. Inspect is also used by NIST's Center for AI Standards and Innovation, the UK institute's US counterpart, including in its published evaluations of agent hijacking, the class of attack this talk's demo tests.

We will cover how to define test cases, run an agent against them, and score its behavior, then run a prepared eval live. The demo tests whether instructions planted in a proposed code change (a pull request) can redirect an AI code review agent, an attack known as prompt injection. We will test the agent with and without a prompt-injection defense using the same cases, examine variation across repeated runs, and discuss what the results establish and what remains untested. No prior experience with AI evaluation is required. Attendees will learn how to turn a suspected failure mode into a test case and interpret the resulting evidence.



  Date and Time

  Location

  Hosts

  Registration



  • Add_To_Calendar_icon Add Event to Calendar

Loading virtual attendance info...

  • Contact Event Hosts
  • Starts 03 November 2026 08:00 AM UTC
  • Ends 20 November 2026 08:00 AM UTC
  • No Admission Charge


  Speakers

Nate of University Of Idaho

Topic:

Testing for Failure: Evaluating AI Agents

This talk introduces AI evaluations (evals) for testing the reliability, security, and behavior of AI agents using Inspect, an open-source evaluation framework. Through a live demonstration, attendees will explore how prompt injection attacks hidden in proposed code changes can redirect an AI code review agent. The session will compare agent behavior with and without prompt-injection defenses, examine variations across repeated runs, and discuss how to interpret evaluation results and their limitations.

Biography:

Nate Summers is an Enterprise Applications Developer at the University of Idaho's Office of Information Technology, where he leads the AI & Applications development team and works with campus researchers to transform prototypes into production-ready systems. He has ten years of IT experience, including five years in application development, and has been working with AI-assisted development tools since 2019.

Email:





Agenda

  • Introduction to AI Evaluations (Evals) and the Inspect Framework

  • Designing Test Cases for AI Agents

  • Running Evaluations and Scoring Agent Behavior

  • Live Demo: Testing Prompt Injection in AI Code Review Agents

  • Comparing Agent Performance With and Without Security Defenses

  • Analyzing Results, Repeated Runs, and Limitations

  • Q&A and Discussion