Part III: A Proper Mix of Manual and Automated Testing is the Answer.
Final Installment of “The Myth of 100% QA Automation” Series
In the previous two installments of our series, we looked at the reasons why treating automation as a “silver bullet” is a bad idea, and why using 100% automated testing is simply NOT possible. In this final edition, we’ll examine applications where automation is the right solution and how to continue to add value as a Manual QA Engineer.
We’ve established that for real-time, first pass, functional testing and business validation you need a human involved in the process. But, does that mean all automation is bad? No! There are some very good reasons to employ automation including:
- Performance Testing: Can’t measure page load times, transactions per second, and other metrics accurately by hand.
- Load Testing: Can’t simulate hundreds or thousands of virtual users manually, nor measure system performance.
- Regression Testing: Written AFTER the main software workflow, once the application has been stabilized, and all known bugs have been filed and dealt with, then automation is useful for protecting sets of functionality.
- Data-Driven Testing: Using a spreadsheet of inputs and expected results, iterate through a list.
When it comes to automation, consider everything in moderation. Use automation for those areas that it must be used or areas where the payoff is large. But, for basic functional testing – you still need manual quality assurance.
The Multiplier Effect
There are a number of situations where due to the pure number of times a test needs re-executed that the return on investment of covering the test with automation makes good sense. Some of the factors include:
- Browsers: Does it work on Chrome, Firefox, IE11, and Edge?
- Platforms: Windows 10, iOS, Android, OSX?
- Versions: Do you have multiple versions to support?
- Release Branches: Do you have multiple releases? Or per-customer branches?
- Roles: Do you use Role-based Access Control? How many role types?
- High permutation of settings: Do you use a rules engine with many varied inputs?
- Data Driven testing: Do you have a spreadsheet full of inputs and expected outputs?
There’s a linear relationship between the number of variables (previously stated) and the justification for automation.
- Bugs are found where you least expect them.
- Humans are creative and analytical. Automated tests are not.
- Good testing is repeatable, but also variable.
Don’t try to be something you’re not. It’s fair to say that your company is not Google, LinkedIn, Facebook, or [fill in any other ad-driven, free for the user, don’t care if it breaks platforms]. Determine your own organizations tolerance for risk and model your testing strategies with that in mind!
Stop trying to emulate rich, IT-employee heavy, tech companies. We can solve many problems by throwing more money at them. Automation often does not solve resource constraints. You likely have a limited budget, so let’s use that budget wisely.
A Proper Mix of Manual and Automated Testing
Here’s a solution we’ve found that works for many companies:
In this ideal pipeline, we have two groups of QA engineers. The first manual testing group is responsible for developing test strategy, writing manual test cases, preparing the test environment, and finding the initial bugs in the application. Once a feature has been tested, and primarily functions, the manual team sets to protect that functionality by developing an Automation Test plan which gets forwarded to the automation team. Regression testing in subsequent sprints is done manually until the automation tests have been created.
The automation team receives the automation test plan, and generates a series of automated tests which protect the base functionality associated with the features. Once those automated tests are written and debugged, then they are used in two different ways:
- To speed up repeated regression test runs for the manual testers.
- To act as a smoke testing gate to protect the main line branches from inadvertent regressions which break base functionality.
By using both manual testing and automated testing in the right mix, it’s possible to leverage the benefits of each. Use manual testing where needed, and allow time-shifted automation tests to follow. This necessary delay between feature release and automation test completion is also a useful method, allowing the feature to mature and stabilize prior to installing regular automation test runs.
- DISPEL the myth that Automation is a “silver bullet”.
- CONVINCE decision-makers to use Automation at the right time and in the right place.
- CONTINUE to show the value Manual QA brings to the table.
- BE CREATIVE when testing!
- TRUST your Manual QA staff to create the Automation Pipeline.
In each edition of this article series, we covered a wealth of information about the issues that exist with trying to implement a fully automated quality assurance strategy. Do you still have questions about the ideal software pipeline for automation? We’d be happy to share examples that we’ve found that work for many companies, just like yours. Contact Us today!