AI-Powered Testing Practices | Why Didn't You Test That?

06:53 - Testing AI accuracy rather than the tests.

13:50 - Testing community is playing catch up.

20:43 - SME knowledge and AI.

27:37 - Auditability, repeatability and explainability.

34:33 - Testing and safeguards.

41:28 - Manual and exploratory testing.

48:26 - Vendor and organization collaborations.

Welcome to episode 14 of the Why Didn’t You Test That? Podcast! In this episode, the Curiosity team, Rich Jordan and Ben Johnson-Ward, are joined by Alex Martins, VP of Strategy at Katalon, to discuss the implications and challenges of AI-Powered Testing.

This episode goes beyond the hype and marketing euphoria of AI, to weigh up productivity gains coming from GPT-4 and large language models (LLM) in the software quality space. Guest, Alex Martins leads the conversation around the need to put the tester at the centre of AI-powered testing, and only then, start building out AI use cases and safeguards.

Where the development community has seen tangible gains in AI deployment, the uplift in AI-powered testing practices is just beginning. So, how will this impact software testing professionals? Also, how will SME knowledge evolve as organizations develop bespoke LLMs? From the tester's perspective, who pilots the technology for them?

Ben Johnson-Ward argues, if artificial intelligence is used to create test outputs, then testers will have to evaluate the output of these tests to determine if they are correct. This approach may lead to a decrease in productivity as testers spend time testing the output of AI generated tests.

Testers will be able to fine-tune their AI models and build out a broader toolkit. But what does this look like? While organizations are adopting AI in testing, there will also be impact on the metrics of repeatability, explainability, and auditability. With this in mind, internal AI committees can establish rules to abate uncertainty.

Rich Jordan follows up on Ben's point, explaining how from the human perspective, AI may be limited in determining if an application meets the needs of the users. In this use case, AI becomes the co-pilot, a new tool for experts to enhance collaboration, while testers remain primary-pilots. Repeatability is discussed as a characteristic that humans are comfortable with in testing, but can AI offer better alternatives to traditional methods of monitoring code changes and integration flows?

AI-powered practices in software testing and test coverage are still in their early stages. There is a need to find more real use cases that provide tangible value to the testing community, to match those seen in the development community.

This requires ongoing collaboration, learning, and sharing of experiences among organizations and industry professionals. Finally, the possibilities and potential benefits of AI are too significant to ignore, despite the discomfort and challenges it brings in delivering quality software, faster.

Watch the complete episode to learn more!