As operators race to fill the demands of end-users who are working, playing, and living more through their broadband connections, managed services enabled through device management have become a cornerstone of the overall broadband service package. Broadband Forum TR-069, and its successor, the User Services Platform (USP/TR-369) are key technologies used by operators to perform lifecycle management on their CPE, onboard customers, and provide these managed services like Wi-Fi optimization, performance monitoring, parental controls, and more.
CDRouter is the industry standard test platform for these technologies, and has thousands of test cases to validate core protocol functionality, data model interoperability, and security. It is also the official test platform for both TR-069 and USP certification. While running your products through these test cases and seeking certification is vital to their success, it’s also desirable to validate how a device will behave when executing the real-world TR-069 or USP scenarios it will perform in the field.
There are a number of ways to do this kind of testing with CDRouter, while still retaining its powerful abilities to automate and repeat testing.
Using TR-069 or USP scenarios
CDRouter contains special test modules for TR-069 and USP called “scenarios”. While they work a little differently for TR-069 vs. USP (you can read about them in the user guides here and here), the scenario modules allow you to easily build custom test scripts that will tell CDRouter’s simulated ACS/Controller to perform certain actions and wait for certain responses. You can use these modules to re-create the actual configurations and commands that a device is expected to perform in the field.
Scenario tests use “scenario files” written with a specific syntax. This syntax lets you write flexible TR-069 or USP test scripts that perform actions like calling an RPC or Request Message, waiting for a particular response, flagging pass/fail criteria, and more.
There are two different ways to use a scenario file, controlled in the CDRouter test configuration through two different testvars:
- cwmpScenarioPath/uspScenarioPath - This is used to point to the scenario file you want to use when running the cwmp-scenarios.tcl module or usp-scenarios.tcl module, respectively.
- cwmpScenarioBootstrap/uspScenarioBootstrap - This is used to point to the scenario file you want to use when CDRouter runs its Start procedure.
Configuring a device for testing
One of the most powerful uses of scenarios is the ability to set a script to run during the start procedure of your testing. This allows you to use TR-069 or USP as a mechanism to set up a Device Under Test (DUT) automatically before running the rest of your tests. For products that support remote management, this is significantly easier than manually changing a configuration in the device’s GUI!
The scenario script can also be used in between other tests in a package to change the DUT configuration before a set of tests. To do this, include the scenario test module before the tests for which you want a different configuration. You can add your own pass/fail metrics to scenario scripts, so you can use this as a check to make sure that the configuration worked correctly.
Replicating your onboarding process
You can also use scenarios to replicate your deployment’s bootstrap/onboarding process for your CPE. This lets you test that a CPE will be onboarded correctly when in the field.
For example, let’s say an operator wants to read some device info and configure a specific Wi-Fi SSID upon receiving a BOOTSTRAP event from a TR-069 client. You can build a cwmp scenario to do exactly that:
- Wait for a BOOTSTRAP event
- Perform a GetParameterValues RPC
- Perform a an AddObject RPC and/or SetParameterValues RPC
- Perform a GetParameterValues RPC to verify
You can do this as part of the Start procedure, or as an explicit use of the cmwp-scenarios/usp-scenarios modules in your test package. If done during the Start procedure, any failures will be noted in the log, but your test package will continue as planned to avoid interruption to your automation.
Reproducing issues seen in the field
Another great function of scenario testing is the ability to replicate a sequence of events that may have caused issues with a device in the field. For example, if a certain set of configuration actions causes a device to refuse the configuration or end up in an error state, you can use scenarios to perform the exact same commands that were run from the ACS or Controller in the field to see if the issue can be reproduced.
Both the CWMP and USP scenario syntax have mechanisms for doing this, though the options for USP are a bit more flexible. In either case, rather than setting an expected value to be returned, you can set expected error conditions to occur during your scenario test. Using pass/fail metrics will allow you to see if the issue was replicated exactly or not.
Using CDRouter Security for live network connectivity
The scenario tests above are built to use CDRouter’s simulated ACS or USP Controller as the server for your TR-069 or USP clients. While this gives a great amount of flexibility in testing, sometimes you’ll want to have your DUT connect to the real services it will be using in the field. When you’re able to do this from within your automated testing environment, it has a number of useful advantages.
Connecting to a live ACS/Controller
The CDRouter Security expansion includes the ability to separate test traffic from traffic that is destined for other networks. You can use this ability to allow a DUT to communicate with the services it needs to access during operation, including a TR-069 ACS, one or more USP Controllers, or related services like a bulk data collector.
Using this setup, a device connecting to CDRouter to begin testing can communicate with its ACS, receiving any initial bootstrap, performing critical setup, or even to provide remote firmware updates. Then CDRouter testing can continue normally. If the device has any other applications it needs to connect to during operation, it can reach those, too.
One caveat to this setup is that CDRouter Security works through Network Address Translation (NAT). This means that, for TR-069, you’ll have to use the XMPP Connection Request mechanism, or use STUN, to allow the DUT to receive Connection Requests. To get around this, you can also decrease the PeriodicInformInterval to communicate with the ACS more often.
Monitoring the connection
The other advantage of testing through the CDRouter Security setup is that all traffic is monitored in real time during testing. Using the Suricata Intrusion Detection framework, traffic that is bound for the Internet is compared to a set of pre-configured “rules” that can be used to alert testers. In addition to security threats, however, these rules can be used to catch all sorts of wanted or unwanted behavior.
For example, QA Cafe’s custom rules included with CDRouter Security monitor live TLS communication to ensure that proper certificates are used between the ACS and CPE and that sensitive information is not being sent unencrypted. You can also build custom rules for CDRouter Security using Suricata’s easy-to-use syntax, checking that the device is communicating with the right services in the right way.
Setting and connecting to bulk data servers
You can use TR-069/USP scenario tests and CDRouter Security in tandem, coupled with CDRouter’s host simulation capabilities, for some interesting exercises.
For example, you can create a TR-069/USP scenario to configure a bulk data profile in the device’s data model, setting the target of collection to be an external HTTP bulk data collector. If used to measure the reception of traffic statistics on a particular interface, running a CDRouter Performance test will trigger changes in those statistics which will exercise the reporting capabilities of both the device and the collection server, all as part of your automation.
Balancing isolated vs. real-world
While it’s important to run automated, repeatable tests in an isolated test environment, it can be equally useful to validate a device’s behavior as it would perform in the field. CDRouter’s unique test setups and scenario options allow this sort of testing and expands the number and types of validation you can perform on your devices to make sure they will work, and work well, when deployed in the real world.
