Videos

What to do with your stability test results

Stability testing is an extremely valuable process you should add to your overall automated testing strategy. In this video, Matt summarizes what to do once you’ve got some stability test results: what to look for, how to find it, and what to do next to help improve your products.

Tips for analyzing stability test results

In summary, Matt suggests the following tips once you have some stability test results available:

1. Look at your performance results

Use the Performance Visualization tool within CDRouter to review the performance over time. Spot any unexpected dips or drops can highlight potential problems that need further investigation.

2. See if you can reproduce the issue

If an anomaly is detected, the first action is to verify its consistency. Rerun the test package to see if the issue can be reproduced. Reproducibility is important in validating the stability issue and not dismissing it as a one-off event.

3. Look at your functional results

Check the functional results from each loop of the test to ensure consistency. This step involves looking at two aspects:

  • Analyze pass/fail test results: Establish a test package where all functional tests are expected to pass. This approach makes it easier to identify any issues if they arise, giving you clear indicators of functional issues.
  • Monitor the test durations: Analyze the duration of individual tests across different loops. Over time, protocol interactions may start to take longer as state builds up, which could be a signal of underlying problems.

4. Look for Correlations

Often (but not always), functional failures and performance degradation are related. An increase in the duration of individual tests, or anomalies in log data, may hint at the source of the performance issue. With these correlations, you can drill down to find clues into the underlying cause of the problem and give you insight into what smaller duration test runs you can perform to isolate and resolve the issue.

5. Ramp up your stability testing

If your test results don't reveal anything, it's an opportunity to expand your stability testing. This can include:

  • Extending the test duration: Increase the duration of the test. Let it run for a week, two weeks, or even longer, to uncover any issues that might take more time to surface.
  • Adding more functional test cases: Incorporate additional test cases into your stability package. Choose common protocol interactions such as DHCP, DNS, and ARP to simulate a real-world network environment.
  • Increasing the number of LAN clients: If you initially tested with a certain number of LAN clients, consider increasing that number. This expansion can also include mixing different types of clients, such as Ethernet and wireless.
  • Run stability tests on different configurations: Test the stability package across different device configurations. For example, run the stability tests in different WAN modes, such as DHCP, PPPoE, or over IPv6.

6. Run testing in parallel

Use CDRouter's parallel testing to conduct multiple stability tests at once. This strategy can be useful for testing different device configurations simultaneously, offering a comprehensive view of the system's overall stability.

Putting this all together

Stability testing is a critical part of the product development cycle that plays a pivotal role in ensuring the robustness and reliability of devices over extended periods of use. Stability testing using CDRouter ensures consistent product performance but also helps to uncover potential issues that might otherwise go unnoticed. The power of this testing lies in the ability to stress the device with real-world network environments and interactions quickly and repeatably, finding issues that will affect the quality of your product and the end-user experience long before they appear in the field.

Find the rest of our series on stability testing here: