Research has shown that consumers list “fear of technical issues” as one of the primary concerns that cause hesitation in investing in the smart home, followed closely by privacy and security concerns. Testing is one of the most important ways to ensure your products will deliver a reliable end-user experience.
As a developer or manufacturer of smart-home products, it is in your best interest to test all aspects of your products and applications. However, testing is not always a top priority for smart-home companies. As a result, product managers have to make decisions with their testing budget about what kind of testing to focus on and how best to get it done.
Here are some of the things you should consider when allocating your smart-home product testing budget.
There are three major focus areas for testing smart-home and other IoT products.
Application/feature testing
The area that most startups focus on is application or feature testing. This type of testing makes sure that the product will deliver the functionality it’s designed to do in the way it is intended, and that it interacts with the services that are required to support it (validating API calls, data management, etc.).
Security testing
Creators of smart-home technology are under huge pressure to perform security testing. This testing takes the product’s threat assessment and builds test cases that encompass penetration testing (e.g., network port scanning), data validation of API calls and user interfaces, and validating access control schemes at the application level (i.e., user access management, privacy, and data integrity).
Connectivity and stability testing
Perhaps the most important but most misunderstood type of tests are those that exercise a smart-home or IoT product’s ability to properly connect to the network, and utilize the standardized protocols necessary to function and perform these operations reliably. This can include things like obtaining a network address, maintaining connectivity when sleeping/waking up, connecting over various Wi-Fi security modes and radio technologies, and not crashing or becoming inoperable when doing these operations frequently or over long periods of time.
The importance of this type of testing cannot be stressed enough. Failing or inconsistent connectivity and stability will trickle down into all other aspects of your product, affecting the end-user experience significantly regardless of how well tested the product’s features and applications are.
When making decisions about where to spend your testing dollars, you’ll ultimately need to decide whether to develop the testing on your own or seek a solution developed for you already. The answer depends greatly on the type of testing as outlined above, but can be reduced to two main questions:
Do we have the expertise?
Creators of smart-home technology should be able to focus their knowledge on the things that differentiate their products and solutions from others on the market. It's a harsh truth of the industry that having a focus on the user experience of consumer electronics is no longer sufficient knowledge to deliver quality products. However, developers are now expected to be experts in security and networking in order build complete, robust solutions.
When deciding how to spend your budget on test case development, consider your team’s expertise. It might make sense to do in-house test case development for application and feature testing, because your developers are the ones most familiar with your system and API (and may already be using test-driven development schemes). However, you will experience huge savings and efficiency gains by purchasing a test solution with test cases written by experienced professionals for standardized security and connectivity testing.
Can we afford the time and labor cost?
When considering whether to build testingon your own, definitely consider the time it takes to do so, and how expensive it is to dedicate your developers to this task. Especially today, your dev team is already squeezed for resources and spending a lot of time on test case development, regular maintenance, and test infrastructure means they aren’t focused on building product features. Only perform in-house test case development when there are no other options available and it makes the most sense to do so.
How you run testing as part of your overall development process is another factor in how you spend your testing budget. Investing in automation rather than manual or even outsourced testing will drastically reduce the time and cost of development. Things to consider here include:
Working with Continuous Integration/Delivery (CI/CD) systems
Most modern software and firmware development relies on agile processes that make heavy use of CI/CD environments with automated pipelines. Make sure that you spend your budget on tools that are designed to work seamlessly with your CI/CD system and that make it easy to automate testing and analyze feedback with clear pass/fail criteria.
Working with other tools
Testing products where user interaction is difficult to simulate ultimately requires that you are able to integrate all of your tools together to get value out of automation. For example, having tools that power-cycle devices on and off, automatically change wireless settings, and deliver useful logs and packet captures will make your overall testing more cost-effective.
Working with your team and partners
Very little development today is done by one person or even one team. Often, the supply chain involved in building a smart-home product will cross multiple companies with multiple teams from all over the world. Consider spending your budget on tools that will make it easy to collaborate on testing, standardize test procedures so that testing can be easily replicated, and share results among all members of your development team.
We talk a lot here about testing during development, and for many smart-home product creators who are trying to get a new product out to market, this is where the most testing dollars are spent. However, the reality of the smart home is that the support lifecycle is much longer than it ever was for legacy consumer electronics. Make sure you allocate your budget to include continuous regression testing as you develop new features, patch security issues, and create new versions of your product. The most important things to keep in mind for regression testing are:
Ultimately, you want to allocate your testing budget in a way that will discover issues early or quickly fix issues discovered in deployment. Finding the right tools, like PassPort by QA Cafe, will help you achieve that goal effectively while taking the burden off of your developers, letting them focus on what makes your products great.
Smart-home developers and product managers can benefit greatly from a test automation strategy. Download our free whitepaper to learn how to build an overall strategy across your entire team and supply chain.
