As the television industry moves towards streaming and OTT services, the video set-top-box (STB) has evolved from a broadcast television decoder to one that operates as an independent computing platform with Internet Protocol (IP) connectivity. The functionality an STB provides may now be included in a smart TV itself, or through third-party devices like Roku, Google Chromecast, or Amazon Fire stick. In many cases, these devices connect via Wi-Fi, to allow consumers to place them anywhere in their home or business without any further installation work.
Because these devices are so dependent on the consumer’s network, testing their connectivity, performance, and stability is critical. Consumers hate nothing more than interrupted video services. It is in the best interests of operators and manufacturers alike to ensure that all of the technologies involved in their function will work well, beyond simply their capability to decode video streams.
Here are some of the most important things to consider when testing set-top-boxes, smart TVs, and other IP video devices.
Test for delays, latency, and failures in DNS
Test that your devices will correctly configure their Domain Name Service (DNS) servers via DHCP, make successful DNS Queries, work with a gateway that is acting as a DNS proxy, and successfully handle failover to secondary DNS servers.
Almost all network-enabled consumer products rely on connectivity to many different services to function properly or even connect in the first place. Since DNS is responsible for translating the domain provided in a URL into an IPv4 or IPv6 address, is one of the most heavily used application protocols on any product.
Delays, latency, and failure in DNS Queries, and the processing of Responses, are one of the most likely sources of connectivity problems. Whether the device is changing “channels,” looking up video resources, logging in, or using any web service, a DNS problem almost always translates into a dreaded spinning circle for the end-user.
Test that your product supports IPv6 completely
Test that your products fully support IPv6, including addressing, discovery, DHCPv6, and DNS over IPv6.
The end-user network that is using any sort of modern Wi-Fi gateway will usually support IPv6. Operators support IPv6 in their networks by using transition mechanisms that tunnel the IPv6 traffic of the end-user network to directly interact with the Internet applications that deploy services over IPv6.
However, even off-the-shelf IPv6 networking stacks need to be tested to make sure that they will be interoperable with common gateway implementations. STBs and smart TVs that implement code to support IPv6 are no exception to this rule. Moreover, IPv6 interoperability is vitally important when dealing with the end-user devices used to interact with your products, like smartphones, tablets, and personal computers using a brower.
Validate that your set-top-box or smart TV will stay stable in the long term
Test that your products will remain stable and still perform well after frequent use or turbulent protocol activity.
One of the things we often encounter with connected devices is that device stability can be at risk over long periods of time, and the problems that cause them can be hard to identify. DHCP address assignment, Wi-Fi association/disassociation (especially over different security methods or legacy protocols), and more can cause memory leaks or memory fragmentation issues to build up and result in devices performing unpredictably or not at all. In some cases, these issues can expose security vulnerabilities as well.
The test methods used to uncover these stability issues involve repeatedly performing these functions at a greater frequency than would occur in the field. This can reveal things in a matter of days or hours that the customer might not experience for weeks or months. Coupled with the unpredictability of the effects this might cause, it can result in end-users thinking that the product they purchased isn’t of top quality after all.
Invest in pre-designed tests that are fully automated
While developers may focus testing on video stream performance and UI/UX testing, IP-enabled connected devices, connectivity testing is often not very well understood or prioritized. The good news is that tools exist with pre-defined test cases for these standardized protocols and applications, saving you the time and effort it takes to develop them on your own.
Moreover, it’s important to invest in tools that automate this kind of testing and can work with your Continuous Integration/Continuous Delivery (CI/CD) environment. Manual testing is expensive and is not conducive to the repeatable examination of device quality. This is especially important for stability testing, which must be run by an automation engine to be performed at all.
Testing these top set-top-boxes and smart TV issues in PassPort
PassPort by QA Cafe is designed specifically to assist developers building these kinds of devices, as well as the service providers and cable operators that are deploying their own set-top-boxes. It contains hundreds of test scenarios to cover the connectivity and stability testing mentioned above that can be run easily by testers with just a few simple steps:
1. Connect your device to PassPort and select it from the discovered devices
PassPort setup is easy: connect your set-top box or smart TV under test directly or via Wi-Fi. It will automatically appear in the list of available devices to test.
2. Select the tests you want to run
Check off the tests you want to run against your device. You'll get an estimate of how long it will take.
For the above issues, you should select at least:
- Basic IPv4and IPv6 connectivity testing
- DHCP test coverage for both DHCP for IPv4 and DHCP for IPv6
- DNS test coverage for both DNS for IPv4 and IPv6 addresses
- Stability testing that stresses the IP connectivity of the device
PassPort also has hundreds of Wi-Fi test cases covering various technologies and security modes that are extremely valuable for STB testing.
3. Start your testing
As your tests are completed, you'll get a live look at the status of your test run.
4. See what worked, and what didn't
PassPort has a complete set of results with clear pass/fail indicators, logs, and packet captures viewable with advanced tools to investigate issues.
It's that easy! Contact us to learn more.
