Altibox is one of the largest Internet Service Provider partnerships in Norway, and was the first to offer fiber broadband to the Norweigian private market. With over 800,000 subscribers, they have been named the highest in customer satisfaction for over 12 years. They are also one of our most active CDRouter customers with a solid testing and automation strategy that has paid off in reduced support costs and high-quality customer experiences.
We recently sat down with Rafal Wierzbicki, the test lead responsible for quality assurance on Altibox’s CPE fleet, to ask him about the importance of stability testing to the broadband user experience and how he’s making use of CDRouter and its parallel testing capabilities to do more stability testing, faster.
QA Cafe (QA): Hi Rafal! We understand you do an awful lot of testing for Altibox. Can you tell us a little about their overall testing situation and strategy?
Rafal Wierzbicki (RW): Hello! As you mentioned, Altibox has over 800,00 subscribers. This has come from years of success and growth, and the result is a broad inventory that comes from many different vendors. Some of these CPE are built for different access technologies, some are deployed because they support services specific to the needs of the subscriber. Others still are devices from the existing installation base that Altibox updates to give the latest services to their subscribers. To support this diverse ecosystem, Altibox is committed to a rigorous test process to ensure a quality end-user experience regardless of the CPE they have.
QA: You’ve been in testing for a while now, as well as using CDRouter in particular. How has your testing strategy evolved over time?
RW: While CDRouter has coverage for hundreds of protocols for doing straightforward RFC compliance and validation, there’s much more that needs to be done that CDRouter can also do. We’ve moved on from this type of validation (our initial “smoke test” to make sure there’s nothing clearly wrong with the device) to include performance, regression, and above all - stability testing - to our overall strategy.
QA: So let’s talk about stability testing, as it’s not something that everyone understands well, but you’ve made it a key component of your strategy. What is it, and how do you approach it?
RW: We perform stability testing to look for issues that may appear in the field due to regular use of the products. For us, IPv6 stability was most important; it’s part of Altibox’s network strategy to support it for subscribers, so it has to work well and work consistently. You can’t suddenly have IPv6 functionality die after 3 months - that would cause a lot of complaints!
We do this by creating a test package in CDRouter to cover all critical aspects of IPv6 behavior that would result from end-user activity. The important part here is to find a mix of tests that you know will pass every time, and then run it continuously for as long as you can without rebooting the Device Under Test (DUT). If something suddenly starts failing, you know you have to investigate further. Luckily, CDRouter makes that easy too, both by providing full logs and packet captures and by summarizing what appears in the packets in the test logs. This feature is gold! It takes so much of the manual effort away and saves us significant time and money.
QA: That’s great to hear! What other testing is part of your strategy?
RW: In addition to our IPv6 stability testing, we have a total of nine CDRouter test package that we run against our devices. This is a combination of protocol validation and performance that covers all aspects of the products and the subscriber’s service.
On the performance side we do both WAN to LAN throughput as well as LAN to LAN throughput. For the former, we like to increment the amount of traffic being pushed through the device in 20 Mbps increments, all the way up to the full capabilities of the device. For the latter, we also look for data duplication on the other LAN ports (which can be hard to catch without good packet capture data).
QA: That sounds like a lot of testing. How do you get it all done?
RW: Yes, it can take a while to get through all nine test packages on a device. Recently we’ve been able to drastically increase the amount of testing we can do by adding CDRouter’s parallel testing capabilities, which lets us run multiple test packages simultaneously on different ports. Sometimes we use this to test multiple devices at once; other times we will duplicate a device (hardware + firmware) and run our stability testing on one instance while we run our regular testing on another. It’s really increased the amount of testing we can get through overall, and lets us cover more devices and firmware versions in the same amount of time.
QA: Excellent. Is there any other advice you would give to other testers out there?
RW: Yes - the main point of stability testing is to uncover issues that would take a long time to appear in the field in less time by repeating them very frequently in a shorter period of time. So it may not seem realistic to have that much IPv6 behavior (for example) repeated so quickly, but the idea is to replicate the long-term behavior very quickly, so that you can uncover in a matter of hours or days what might take weeks or months to appear in the field. You can never do enough testing!
