Let’s be honest: designing good smart home products is hard. The industry is still heavily dependent on DIY customers, users are heavily concerned about privacy and security, and when things don’t work, they are quick to abandon them or move on to another product after leaving a bad review.
There are many reasons for this, from quality assurance of the devices themselves to the underlying and possibly third-party and open-sourced components, all the way to the applications that get these complex systems to work together. As the CSA’s Matter protocol seeks to solve some of these problems, the reality of basic networking and connectivity issues continues to be the major and overlooked source of bad customer experiences.
At QA Cafe, we’ve put quite a few off-the-shelf smart home products through the test cases in our fully-automated PassPort test solution. Here are three of the most common problems we found and why they matter.
1. Smart home devices connecting to Wi-Fi
View the InfographicWi-Fi and IP-based smart home products are among the most common, and their ability to interact with the consumer’s home network determines whether they will work at all.
The number of combinations of Wi-Fi technologies, bandwidth channels, security modes, etc., make this an extremely complex environment for smart home devices. Moreover, predicting how a device will behave long-term is often difficult when some issues may not arise for weeks or months of regular use. Issues with Wi-Fi in the smart home include:
- Connecting to legacy Wi-Fi technologies
Like it or not, your customers probably won’t have the latest Wi-Fi technology on their home router. Legacy modes of operation as far back as 802.11g are still common in households today.
It’s important to test that your products can repeatedly connect to all Wi-Fi modes by forcing your test environment to offer only one or a few options. This lets you demonstrate your device’s capability to connect to legacy networks successfully and allows you to find critical edge cases.
- Covering all security modes
Secure communications are of utmost importance to the smart home. As WEP moved away to WPA, then WPA2, and WPA3, it’s still possible to run into older security modes and see interoperability issues when connecting over different 802.11 technologies.
Run testing against all of the Wi-Fi security modes, and test them over each radio technology from g to ax and beyond. This can uncover edge cases when an unexpected mismatch of a particular combination occurs.
- Association and disassociation over time
One of the most interesting and hard-to-replicate issues with Wi-Fi devices is the degradation of device functionality after connecting and reconnecting to Wi-Fi networks over a period of time. For example, after so many associations, a device may stop trying to connect or even stop functioning altogether.
In the field, this is hard to uncover as it may take weeks or months for a device to become unresponsive in this way. Leveraging your test environment to perform long-term stability tests for Wi-Fi association and disassociation can help uncover this much faster when done in a repeatable, automated manner, reducing the time to find these issues from weeks to days or even hours.
2. Smart home device use of DNS
The Domain Name System (DNS) is an integral part of the internet infrastructure that translates domain names into IP addresses. It’s also one of the most common sources of poor Internet experiences (so much so that it’s a frequent joke among networking experts). Problems with DNS mean devices can’t connect to the cloud services they require to onboard and operate.
Smart home devices may encounter problems with DNS resolution, causing slow or unresponsive connections. This can cause delays in device commands or prevent devices from communicating with each other. This can frustrate customers who expect their devices to function smoothly and seamlessly.
In addition to basic DNS resolution issues, the DNS architecture relies on endpoints to access a backup server for failover when the first DNS server is unresponsive. This is extremely important as DNS must be continuously available for almost any connected device.
In our testing at QA Cafe, we’ve found many smart home devices that don’t implement failover properly or at all. Test to make sure that your device will communicate with the secondary DNS server provided to it (either during setup or via DHCP) and that it will resume communication with the primary server when it becomes available or when the secondary server goes offline.
3. IP renumbering and MAC randomization
IP addresses and a device’s underlying MAC address on a Wi-Fi interface can all be used to identify and track smart devices and users. This introduces some very serious privacy and security issues in an industry that is particularly sensitive to them.
To address this, network gateways and a devices interface stack themselves implement different technologies to ensure that the IP addresses assigned to hosts will change - or renumber - after some time or when a new connection is established.
A DHCP server can be configured on the gateway side to hand out new addresses more frequently or when a device disconnects and reconnects from the network entirely. A smart device must accept and use this new address even if it possesses logic to try to remember the address it used the last time it connected.
In addition, many mobile devices implement MAC address randomization to protect user privacy. Rather than relying on the heuristic that a device’s MAC is a permanent identifier, MAC randomization will change the advertised MAC address after a period of time, usually several hours. When a device re-connects to a Wi-Fi network from which it had previously connected, it must be ready to ask for a new address and utilize the one it is assigned.
As this behavior is quickly becoming the norm and even required by regulatory bodies, testing that your smart home products will still communicate properly when their IP address is changed regularly is important. Some devices store this information and try to use it in future connections or as a data point for its features and functions. The assumption that IP and MAC addresses will remain stable is worked into many, many design choices all the way up to the applications themselves, but this assumption has become unreliable. The reality of changing addresses means this assumption can be a critical source of bad user experience.
Catching smart home product issues with automated testing
Though simple code issues can cause these problems (whether from an open-source stack or not), discovering them is difficult without the ability to automate your smart home device testing with standardized test scripts that can be run predictably and repeatedly within your development and quality assurance processes. QA Cafe’s PassPort is built for this very reason! Contact us to learn more.
