Deciding how to incorporate stability testing into your overall test strategy can be a challenging task. In this article, we discuss how to incorporate stability testing with your feature validation, performance, and regression testing, and how to leverage CDRouter's parallel testing capabilities to streamline the process.
The four types of connected device testing
As we cover in our definitive guide, “How to Build an Automated Test Strategy,” there are four primary types of testing that are each applicable to different phases of product development.
- Feature validation involves testing individual features and functions as they are added or updated, as well as security testing. This is generally tested on a per-commit or per-build basis and can vary in test duration.
- Performance benchmarking means measuring ideal throughput, latency, etc., in a controlled manner to get baseline results for product evaluation and regression testing. This is generally done purposefully when a device is ready to undergo performance testing, i.e., per build, and can vary in test duration.
- Stability testing is testing over a longer period that combines performance and feature testing to reveal bugs that may arise from regular product use. This is generally done on long test cycles, such as over a weekend, over an entire week, or longer.
- Regression testing compares changes from a new firmware version to the old firmware version to ensure no new bugs were created. It is especially important for security testing. This is generally done per build or per commit.
The importance of stability testing
As we’ve discussed in this article and video, stability testing helps identify any potential issues or limitations that may arise during the regular use of connected devices. So much of QA Testing uses a fresh reboot and does a test on the scale of an hour. Customers leave them running for weeks. The issues found by QA and customers in these two scenarios are completely different. The reality is that customers will perceive these issues as a poor device even though it is, in fact, just a bug that reveals itself after prolonged usage.
Stability testing enables developers to address problems early on and prevent them from affecting the end user's experience. In the long run, stability testing contributes to customer satisfaction and reduces the risk of device failure. You can find an example and sample test package for stability testing in CDRouter here.
Because stability testing can take up a lot of testing time, it can be difficult to make time for it in your overall test strategy. Instead of treating stability testing as an isolated process, consider integrating it with other types of testing by testing in parallel.
Using CDRouter parallel testing to add stability test cycles to your strategy
CDRouter's parallel testing capabilities allow you to run multiple test instances simultaneously, reducing the overall test execution time. This works great with a Continuous Integration/Continous delivery system that utilizes CDRouter’s automation API to start tests, track the results, and trigger further testing or action. Using parallel testing, a single CDRouter platform can be maximized to cover all four of the major testing types, as shown in this infographic:
In this example, four physical versions of the same product (Model A) run different types of tests in parallel. The first three are builds of Firmware #1, with DUT #1 possibly incorporating changes on a per-commit basis through a CI/CD pipeline. DUT #4 is also hardware model A, running Firmware #2 to compare it to results from Firmware #1.
Even when one CDRouter system is maximized with five devices running simultaneously, additional CDRouter systems can be used to test other products (Models B, C, D, etc.), all automated through the same CI/CD system, if desired.
Overall, CDRouter's parallel testing capabilities enable you to run testing on multiple devices concurrently, allowing you to maximize your test environment's efficiency. By using parallel testing, you can:
- Minimize test execution time by running multiple test instances simultaneously
- Easily scale up your test environment to accommodate more test cases and execute different types of testing at the same time (e.g., stability testing)
- Test multiple devices simultaneously that use different WAN technologies and connection modes
- Test multiple configurations of a single DUT simultaneously
Continuously monitor and improve your test strategy
You’ve successfully incorporated stability testing into your testing cycles. What do you do when youuncover issues that arise from stability testing? You’ll want to isolate the issues and reproduce them efficiently and quickly so developers can fix them. Look for future articles on stability testing to address this critical next step!
