Monday 15th June – Full Day Tutorial
I’ll be at the EuroSTAR conference for my 20th year! I hope you’ll join me!
Now that serious software and huge systems are being built from AI-based components, and using AI-based tools, what is left for humans to do? Responsibility demands at least one thing : that we ask the question did it do a good job? This question must be answered with testing.
It might help to think of AI, and especially generative AI, as a sort of magic box. People say it can do anything and everything. How do we test such a claim? How might we use a magic box in our testing? How can we responsibly release AI products and work with AI-powered testing tools? To answer these questions, testers must raise their skills to a new level — and even developers who’ve never enjoyed it will now find testing to be a large part of their work.
Testing AI isn’t like testing normal software. Yes, all the historic problems of testing are still with us, but there are new challenges. One is that GenAI doesn’t behave deterministically. A lot of the testing we need to do is therefore probabilistic and statistical.
A single round of a test is no better than a one-off sales demo. To test effectively, you have to run many trials, study the results, and report on the problems you find. But another challenge is dealing with the developers and managers: we have to figure out what kinds of problems are worth reporting, and what problems will be marked “can’t fix.”
In this workshop, you’ll learn why AI is a new and different testing problem, how you can cope with that problem, and how using AI to help perform testing is intriguing and powerful but also risky.
I will also be sharing content from my book Taking Testing Seriously, that addresses these issues.