As we’ve said many times in our testing strategy guide and best practices guide for Wi-Fi router testing, stability testing is a vital quality assurance vector to really understand how your products will behave in the field. While many test solutions and test processes focus on maximizing performance numbers in ideal (or even controlled impairment) situations, this is still not a complete picture of your product’s robustness.
Before we get started discussing what stability testing is, let’s share our video with Matt Langlois where he covers an incredibly insightful use case of stability testing in CDRouter:
The summary of Matt’s story is that we had a CDRouter customer, a major ISP, that was experiencing very bizarre, sudden drops in performance of their Wi-Fi CPE. They asked if we had any interesting ideas of how to reproduce it, so we turned to long-term stability testing to examine the problem.
What is stability testing?
Testing protocols, features, and performance are all fairly obvious. What may not be obvious is how protocols and features interact and how those interactions can lead to unexpected or poor behavior and performance in other areas. This is why it is important to test protocols, features, and performance together and not just in isolation.
There are few features of stability testing that set it apart from regular performance testing:
1. Stability tests performance in the presence of real network functionality.
Stability testing combines functional testing (testing the behavior of protocols, applications, and other services on your device) and performance testing together. This is done by queuing up a set of functional tests (for example, DHCP addressing, DNS requests, Wi-Fi association/deassociation), repeating them a good number of times, and then running performance tests to see if there has been any effect.
2. Stability testing uses performance tests as a heartbeat, not a final metric.
The purpose of performance tests during stability testing is to uncover deltas. Did the performance change after running other tests? Do other tests fail after running performance tests? Rather than trying to achieve the highest possible performance in an ideal situation, stability testing is meant to push the limits of your device and find issues you may never discover otherwise.
3. Stability is run over a longer period of time than other testing.
Stability testing should be looped and repeated as long as you can tolerate for your test cycles.
As we outline in our whitepaper, How to Build an Automated Test Strategy for Broadband CPE, Wi-Fi, and Smart Home Products, different types of testing should be done at different phases of your development cycle. Run stability tests during longer, overnight cycles or even over several days when possible.
The results of stability testing can be dramatic
In our example with that large ISP from before, it was discovered that a large number of Wi-Fi associations and deassociations was affecting performance. When running performance alone for 2 weeks, we saw no issues:
But after about 200 Wi-Fi associations, the performance dropped dramatically:
This occurs because functional behavior can build up unseen state in the device until something unexpected happens. This may be memory leaks, fragmentation, table/arrays that exceed allocated memory or other unpredictable states the device might get into from bugs that are very difficult to uncover and replicate.
While this was one of our best examples, we encounter this sort of problem often, from subtle performance differences to outright unusable product states. But what seems like intermittent bugs can be replicated consistently using automated stability testing.
Automation will reveal stability issues way faster than in the field
The main benefit of using CDRouter to run this kind of testing is that, in addition to the full capability to perform thousands of feature and protocol tests interwoven with throughput testing, easy automation is required to really get the most out of stability testing.
Stability issues are pernicious and may not appear in the field after a long time of regular use. Automated stability testing looped hundreds of times over a longer testing cycle can reveal these issues in hours or days in the lab rather than weeks or months in the field.
Use parallel testing to add stability to your overall strategy easily
Stability testing is so important, but it can definitely take up a lot of testing time in your overall testing strategy. CDRouter can run testing on up to five devices in parallel not only to maximize the testing you can accomplish during your test cycles but to allow you to use each instance for diverse purposes. For example, with all five parallel tests active, you could dedicate one instance to running long-term stability while leaving the remaining interfaces available for per-commit, per-build, and ad-hoc testing.
Duplicating your devices with identical firmware and then using CDRouter’s Parallel Testing instances will let you continue to do your regular development testing while setting aside time for stability testing or even your regular throughput testing, getting more done in less time!
