AI Under the Hood: How to Build Reliable Automated Tests for Non-Deterministic Systems
-
40 min
Functionality powered by AI behaves differently on every run — and classic automation approaches break quickly in this environment. In this talk, I’ll walk through how to build reliable automated tests for non-deterministic AI-driven features, using real challenges from my own experience.
We’ll discuss:
-
how to automate flows that behave differently each time, and which techniques help stabilise these scenarios;
-
how to validate results when exact expectations no longer work: where and how you can still rely on regular assertions, and where an evaluation-based approach becomes essential;
-
the nuances of selecting test data for AI-driven functionality, and why an incorrect dataset can completely break your checks.
This talk will be useful for anyone testing AI-driven features, unstable flows, and anyone looking for ways to maintain quality control in a highly variable environment.