Incumbent vendors and startups are developing “smart” home routers, WiFi access points, or parental control systems that are meant to be managed using a web based application in the cloud. This presents some difficulties for network function QA, as it is difficult to ensure a controlled environment when your device needs to be connected to the Internet or other external resources.
What do we mean by “cloud managed”?
What kinds of devices are we talking about? These are network connected devices that, at some point, need to contact some nonstandard external resource. Some of these devices rely on the cloud for their bootstrapping process. They must connect to a remote server and register with a service, or receive configuration information, before they can even operate. This presents situations where there are devices that cannot operate when the WAN is down, but regardless, these are devices that need to connect to services in order to begin operation, even if they don’t have to communicate with the cloud service later.
Other devices may be remotely managed using a proprietary web interface, report usage statistics, or make API calls as part of a custom application. These devices will generate and receive application traffic that needs to reach or is received from the cloud during the period of normal operation.
Some examples of these devices managed WiFi - systems like Meraki, Aerohive, Eero, and others that allow you to connect to single web application to manage all of your access points and other network devices. This also includes smart routers - single endpoints that nonetheless rely on a user-facing web application for configuration. It also includes things like Smart Hubs; end devices that connect to a web service to function, like home monitoring systems.
The other devices in this category are those that the user doesn’t necessarily interact with directly (or they can in some cases), but the system is making API calls behind the scenes to operate some service. This includes things like parental control of network security monitoring applications, autoconfiguration or bootstrapping through a management protocol, etc.
Lab testing in a closed loop environment
Let’s look at a standard closed loop test setup. In this case, CDRouter is simulating ALL of the network services, WAN terminations, and LAN clients. It’s also simulating any standardized applications that a Device Under Test might interact with. This should be familiar for QA testers; the thing to note here is that the entire network is simulated, so the DUT never connects to the Internet during these tests.
This has its advantages of course. A closed loop is ideal because:
- It allows for isolation. In an isolated environment, you won’t see any extraneous traffic interfering with test results. This lets you clearly distinguish your control vs. experimental variables in testing.
- It provides for a consistent test setup. In a closed loop with simulated interfaces and services, you can be comfortable knowing that it is being generated the same way every time, rather than relying on an existing network setup that may be doing other things or may have been running for a period of time that has introduced unpredictable results.
- It’s easy to use. Since the WAN terminations and standardized services are simulated, there’s no reason to try to provision those services yourself in your lab or use organizational resources that might have to be shared
Translating closed loop benefits to cloud enabled devices
These are the benefits of closed loop testing, and things we want to preserve, so how do we pass on those benefits to cloud dependent devices?
It’s possible to do this by routing test traffic separately from the management and application traffic needed by the device under test. In CDRouter this is done with an add-on called “ICS”. ICS stands for “Internet Connection Sharing”, because it uses some of the same concepts as operating systems that share their network connections. With ICS, CDRouter is set up just as in closed loop, but any traffic that CDRouter knows that is not destined for itself - that is, traffic that is not part of our usual test cases or startup procedures - is routed through an interface attached to the external network (“application traffic” vs. “test traffic”). ICS does this by creating an externally facing NAT to separate the traffic.
The only caveat to this is that it is behind a NAT, so inbound connections are limited. You could use this setup to allow a device to communicate with a real-world ACS, for example, but it would require the usual systems for getting around NAT if you want Connection Requests to work - STUN, XMPP, etc.
Going in-depth
Watch the video above for a more in-depth look at how we test these kinds of devices, including some example captures with devices that contact multiple external services and websites before coming online.
