top of page
Writer's pictureBrittany Stewart

Transforming Software Testing with AI: Early Insights from Our Journey

Updated: Nov 3


Transforming Software Testing with AI: Early Insights from Our Journey - Cover image
Transforming Software Testing with AI: Early Insights from Our Journey

As a Senior QA Specialist, I recognize a challenge we're all facing in our industry: delivering high-quality software faster while managing costs effectively. I've spent the past year exploring AI solutions, particularly using generative AI tools, ChatGPT and Claude, and want to share my early insights with you.

Key Observations:

  • Significantly reduced time spent on test case generation and documentation

  • Faster identification of edge cases and potential issues

  • More comprehensive test coverage, especially for complex features

  • Improved efficiency in debugging and automation tasks


Strategic Potential:

  • Faster market response through streamlined testing processes

  • Better risk management through enhanced test coverage

  • More efficient resource utilization

  • A scalable approach that can grow with business needs

Read on for my implementation journey, practical examples, and specific ways to start incorporating AI into your testing processes.

My AI Testing Journey Using Generative AI in Testing


The 'Fast and Cheap' Challenge 


Recently, I faced a familiar scenario that probably resonates with many of you: I was feeling the pressure of tight deadlines and budget constraints while trying to maintain high-quality standards to deliver test results “fast and cheap”. In situations like this, there's always the temptation to cut corners. However, I've discovered that by embracing AI thoughtfully, we can meet these challenges without compromising quality.

Starting Small

When I first started exploring AI tools, I was skeptical. Would they actually save time? Would the quality be good enough? I decided to start small by using ChatGPT for test case generation on one project. The results were promising enough that I later added Claude to my toolbox.

Here's what I've learned works best:


Real-World Success Stories

1. Test Case Generation

I use AI to help brainstorm test scenarios, especially edge cases that are easy to miss. Here's a real example: For a homepage video component, I asked Claude to generate test scenarios. Not only did it cover the obvious cases, but it also suggested testing:

  • Functional test scenarios

  • Non-functional test scenarios

  • User-focused test scenarios (happy paths and edge cases)

  • Checklist-style test scenarios

  • Behavior-Driven Development (BDD) scenarios

Screenshot of Requirements to Test Scenario conversion using Generative AI
Screenshot of Requirements to Test Scenario conversion using Generative AI

This took about 5 minutes to generate and review. It saved me at least an hour of brainstorming!

I eventually settled on the checklist format for more interactive and functional use for the team. Again, fast and cheap. 

Tip: When using AI for test generation, always start with a clear description of your feature and any specific business rules.


2. Code Debugging Support

As someone who transitioned from design to QA, I sometimes struggle with automation code. Here's how I use AI to help:

  • Get plain-English explanations of error messages

  • Find better selectors for flaky tests

  • Understand complex code patterns

Real Story: When our Playwright tests were failing and reports were broken last month, Claude helped identify outdated selectors and configuration files – saving hours of developer time.

Screenshot of results for using generative AI to support Automated Testing
Screenshot of results for using generative AI to support Automated Testing

3. Documentation and Time Savings

One of my favorite time-savers is using AI for:

  • Converting manual test steps into automated test scripts

  • Writing clear bug reports

  • Generating test summary reports

Quick Tip: I keep a collection of effective prompts in a note for quick reuse. This saves time and gives more consistent results.

Practical Implementation Tips

Based on my experiences with using both ChatGPT and Claude.ai, I’ve refined our approach to AI integration:

  • Specific Prompt Engineering: 

    • Tailoring prompts to leverage Claude.ai's advanced contextual understanding. These were my top-used prompts on my most recent project during the test design and planning phase:

      Helpful Generative AI Prompts
      Helpful Generative AI Prompts

Please rephrase the following requirement in simpler terms, highlighting any ambiguities: 


What potential questions or clarifications might a developer or tester need to fully understand this requirement?


Please generate the Test Scenarios in the following format:

"Verify that user can..." or similar. 


Please create a simple list of items to verify or steps to perform for the requirement.

Please analyze the following requirements:

  1. Rephrase it in simpler terms, highlighting any ambiguities.

  2. List potential questions or clarifications a developer or tester might need.

  3. Generate test scenarios as a bulleted list, starting each with "Verify that user can..." or similar phrasing.

  4. Create a simple list of items to verify or steps to perform.

  5. Estimate the testing time needed based on the following guidelines:

    • Simple user stories: 1-2 hours

    • Moderate complexity: 2-4 hours

    • Complex user stories: 4-8 hours or more

Consider factors such as complexity, feature size, number of test scenarios, potential system impact, tester experience, and development quality when estimating time.

Please provide your analysis and recommendations based on this combined approach.

  • Continuous Human Oversight: Regular review and validation of AI-generated content is needed. During the presentation of this topic, an interesting question was raised by my team member raised an interesting question: "How much time savings does one see when you still have to review the outputs?"In my opinion, the time savings happen mainly in the generation stage more than in the review stage. Especially once you have the prompts that work for you. I would spend about 15 minutes reviewing the outputs and tweaking them.


  • Quick Wins:

    Using tools like Playwright Code Gen made it super easy to get started with my automated tests. Of course, more advanced patterns and techniques came into play but for a first go creation, I am loving this tool more and more.


  • Visual testing with AI-enhanced image comparison: I had the opportunity to try Applitools autonomous. I was very impressed with the tool. However, this is NOT the fast and cheap solution our client was looking for. So if you are looking for that, this is not the best option for you. The suggestions of use above may be more beneficial in the short term. 

    Screenshot of Visual Testing using AI Powered Tools
    Screenshot of Visual Testing using AI Powered Tools - Comparison
What About Quality? 

I know what you're thinking - this sounds great, but does it actually work? From my experience, I've observed several positive trends:

  • Improved test coverage across our features (especially finding edge cases we might have missed)

  • More comprehensive test scenarios generated in less time

  • Significant reduction in time spent writing initial test cases = more time for exploratory testing

Important: I still review everything the AI generates. It usually takes about 15 minutes to review and adjust the output, but it's much faster than starting from scratch.


Getting Started Guide 

If you want to try this approach:

  1. Start with one small feature or area

  2. Use AI to generate test ideas first - it's low risk and high reward

  3. Document effective prompts for reuse

  4. Keep track of what works and what doesn't

  5. Gradually expand to more complex testing scenarios


Looking Forward


I'm still learning new ways to leverage these tools every day. The key is starting small, focusing on practical applications, and gradually expanding your AI testing toolkit as you gain confidence.


Have you tried integrating AI into your testing workflow? I'd love to hear about your experiences!


Happy Testing! 🚀



Comments


bottom of page