BEGIN:VCALENDAR
VERSION:2.0
PRODID:IEEE vTools.Events//EN
CALSCALE:GREGORIAN
BEGIN:VTIMEZONE
TZID:America/Los_Angeles
BEGIN:DAYLIGHT
DTSTART:20270314T030000
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;BYDAY=2SU;BYMONTH=3
TZNAME:PDT
END:DAYLIGHT
BEGIN:STANDARD
DTSTART:20261101T010000
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=11
TZNAME:PST
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20261103T190405Z
UID:8C052EBD-8263-4FE5-8D30-E86D005153FD
DTSTART;TZID=America/Los_Angeles:20261120T150000
DTEND;TZID=America/Los_Angeles:20261120T160000
DESCRIPTION:An AI agent can respond differently to the same input across ru
 ns\, and changing its model or its instructions can change which tests it 
 passes. That makes it a testing problem\, and this talk treats it as one. 
 It introduces evaluations (&quot;evals&quot;) for AI systems using Inspect\, an open
 -source framework for evaluating AI models and agents developed by the UK 
 AI Security Institute and Meridian Labs. Inspect is also used by NIST&#39;s Ce
 nter for AI Standards and Innovation\, the UK institute&#39;s US counterpart\,
  including in its published evaluations of agent hijacking\, the class of 
 attack this talk&#39;s demo tests.\n\nWe will cover how to define test cases\,
  run an agent against them\, and score its behavior\, then run a prepared 
 eval live. The demo tests whether instructions planted in a proposed code 
 change (a pull request) can redirect an AI code review agent\, an attack k
 nown as prompt injection. We will test the agent with and without a prompt
 -injection defense using the same cases\, examine variation across repeate
 d runs\, and discuss what the results establish and what remains untested.
  No prior experience with AI evaluation is required. Attendees will learn 
 how to turn a suspected failure mode into a test case and interpret the re
 sulting evidence.\n\nSpeaker(s): Nate\, \n\nAgenda: \n-\n\nIntroduction to
  AI Evaluations (Evals) and the Inspect Framework\n\n-\n\nDesigning Test C
 ases for AI Agents\n\n-\n\nRunning Evaluations and Scoring Agent Behavior\
 n\n-\n\nLive Demo: Testing Prompt Injection in AI Code Review Agents\n\n-\
 n\nComparing Agent Performance With and Without Security Defenses\n\n-\n\n
 Analyzing Results\, Repeated Runs\, and Limitations\n\n-\n\nQ&amp;A and Discus
 sion\n\nVirtual: https://events.vtools.ieee.org/m/591145
LOCATION:Virtual: https://events.vtools.ieee.org/m/591145
ORGANIZER:sameermankotia2000@gmail.com
SEQUENCE:2
SUMMARY:Technical Talk: Testing for Failure: Evaluating AI Agents
URL;VALUE=URI:https://events.vtools.ieee.org/m/591145
X-ALT-DESC:Description: &lt;br /&gt;&lt;p&gt;An AI agent can respond differently to the
  same input across runs\, and changing its model or its instructions can c
 hange which tests it passes. That makes it a testing problem\, and this ta
 lk treats it as one. It introduces evaluations (&quot;evals&quot;) for AI systems us
 ing Inspect\, an open-source framework for evaluating AI models and agents
  developed by the UK AI Security Institute and Meridian Labs. Inspect is a
 lso used by NIST&#39;s Center for AI Standards and Innovation\, the UK institu
 te&#39;s US counterpart\, including in its published evaluations of agent hija
 cking\, the class of attack this talk&#39;s demo tests.&lt;/p&gt;\n&lt;p&gt;We will cover 
 how to define test cases\, run an agent against them\, and score its behav
 ior\, then run a prepared eval live. The demo tests whether instructions p
 lanted in a proposed code change (a pull request) can redirect an AI code 
 review agent\, an attack known as prompt injection. We will test the agent
  with and without a prompt-injection defense using the same cases\, examin
 e variation across repeated runs\, and discuss what the results establish 
 and what remains untested. No prior experience with AI evaluation is requi
 red. Attendees will learn how to turn a suspected failure mode into a test
  case and interpret the resulting evidence.&lt;/p&gt;&lt;br /&gt;&lt;br /&gt;Agenda: &lt;br /&gt;&lt;
 ul&gt;\n&lt;li&gt;\n&lt;div&gt;\n&lt;div&gt;\n&lt;p&gt;Introduction to AI Evaluations (Evals) and the
  Inspect Framework&lt;/p&gt;\n&lt;/div&gt;\n&lt;/div&gt;\n&lt;/li&gt;\n&lt;li&gt;\n&lt;div&gt;\n&lt;div&gt;\n&lt;p&gt;Desi
 gning Test Cases for AI Agents&lt;/p&gt;\n&lt;/div&gt;\n&lt;/div&gt;\n&lt;/li&gt;\n&lt;li&gt;\n&lt;div&gt;\n&lt;d
 iv&gt;\n&lt;p&gt;Running Evaluations and Scoring Agent Behavior&lt;/p&gt;\n&lt;/div&gt;\n&lt;/div&gt;
 \n&lt;/li&gt;\n&lt;li&gt;\n&lt;div&gt;\n&lt;div&gt;\n&lt;p&gt;Live Demo: Testing Prompt Injection in AI 
 Code Review Agents&lt;/p&gt;\n&lt;/div&gt;\n&lt;/div&gt;\n&lt;/li&gt;\n&lt;li&gt;\n&lt;div&gt;\n&lt;div&gt;\n&lt;p&gt;Comp
 aring Agent Performance With and Without Security Defenses&lt;/p&gt;\n&lt;/div&gt;\n&lt;/
 div&gt;\n&lt;/li&gt;\n&lt;li&gt;\n&lt;div&gt;\n&lt;div&gt;\n&lt;p&gt;Analyzing Results\, Repeated Runs\, an
 d Limitations&lt;/p&gt;\n&lt;/div&gt;\n&lt;/div&gt;\n&lt;/li&gt;\n&lt;li&gt;\n&lt;div&gt;\n&lt;div&gt;\n&lt;p&gt;Q&amp;amp\;A 
 and Discussion&lt;/p&gt;\n&lt;/div&gt;\n&lt;/div&gt;\n&lt;/li&gt;\n&lt;/ul&gt;
END:VEVENT
END:VCALENDAR

