The Hunt For Comprehensive Test Solutions

Posted 07/2000

The Hunt For Comprehensive Test Solutions
Service Providers Face a Sea of Complications When Testing Quickly Evolving Networks
By Ken Branson

"Test range to target, one ping only, Vassily," says Sean Connery's Captain Ramius in The Hunt for Red October. That is test and measurement at its most dramatic. In a way, Ramius is engaging in test and measurement at its simplest, too: He knows what he wants to know, and how to interpret the data in real time.

In a competitive telecom carrier's world, test and measurement is rarely that dramatic (at least, not if you're doing it right), but the stakes are just as high. To get the wrong data, to read the right data the wrong way, or to have the rug pulled out from under a network or OSS all can spell disaster for a carrier. And even if test and measurement for telecom companies was never as simple as Ramius' instructions to Vassily, it was once a lot simpler than it is now.

"We deal with telecom-intensive customers, sophisticated customers, other telecom entities, ISPs, Fortune 500 companies," says John Barnicle, COO of Focal Communications Corp. (www.focal.com). "And we sell a variety of services, from basic circuits to private line to enhanced data. When you deal with the high end of the food chain, you deal with customers who demand service level agreements. It may be just basic uptime availability or traffic-related things; and on the data side, packet loss and other kinds of parameters that you need to be able to measure ... to show a customer what kind of performance they're getting themselves."

As networks grow and equipment evolves, and as technologies converge and carriers merge, it is more important than ever to make sure that everything works. That's a challenge because there is more equipment designed by a larger variety of vendors. Most carriers have networks and systems optimized for voice, but are developing data-centric networks. Since one of the key ways they differentiate themselves from competitors is QoS, they have to make sure that nothing gets lost in the handoff.

"Everyone in the industry at any given time is facing some sort of limitation on test and measurement," says Hamid Akhavan, CTO at Teligent Inc. (www.teligent.com). "The technologies are changing so fast that on the OSS side, we never get the features we want fast enough."

No matter what new bells and whistles may be attached to a carrier's network, competitive carriers still need to be able to test the existing systems.

"The first thing that comes to my mind is that physical part of the network--the unbundled copper loop--and the measurement we do of that loop, whether it's a traditional telephony measurement or DSL-related," says John Boersma, senior vice president of technology at Mpower Communications Corp. (www.mpowercom.com).

For Boersma--whose company is in the middle of a metamorphosis from its initial inception as a fairly standard first-generation CLEC to a DSL carrier--the blocking and tackling of test and measurement still includes the kinds of things that would have been familiar to someone in his job 20 years ago: capacitance, resistance, voltage and traffic loads. And yet, while the terms may be the same, the activity associated with them has changed. Consider the measurement of traffic loads, for instance.

"It's just in the nature of being a phone company, or a data company, for that matter," Boersma says. "You want to look at your capacity relative to peak load."

In traditional telephony, according to Boersma, that sort of measurement takes into account the percentage of calls completed during the busiest hours. But Boersma and his colleagues think of traffic loading in terms of peak bandwidth utilization. The information they take off their voice and data switches tells them they have more than enough bandwidth. "We supply 45mbps on our DSLAMs, and peak is 5mbps, so we know we've got plenty of capacity there," he says.

Vendors and carriers alike say the convergence of technologies and the interconnection with other networks--circuit and packet, voice and data--presents the greatest challenge to carriers, especially the small, newly competitive ones.

The test and measurement strategy carriers developed 18 months ago is most likely invalid now as a result of changes in network technology. "It's probably exceeded their assumptions. So now, they're not only dealing with their legacy systems, but how they will leverage their company's investment in the PSTN. How much of this can I use now in the packet space?" says Steven Murphy, president and COO of Minacom International Inc. (www.minacom.com), a software company that makes test and measurement tools.

Murphy says the networks have changed, and the customers have changed even faster. "Customers are demanding to be self-provisioned and self-tested--which is completely opposite to the prior world, where you planned it and built it and then delivered it," he says.

Even for a new player with an entirely packet-based network, the problems are not over, they're just different problems. Connectivity and interoperability of networks and systems with other networks and systems must be assured, if not guaranteed. There are very few customers who have a direct IP connection to the network, according to Stefan Pracht, product manager for fax and VoIP at Agilent Technologies Inc. (www.agilent.com), so even a carrier with the most modern network has to connect to what Murphy refers to as "the prior world."

"Interworking between traditional and new IP-based networks is a big especially--challenge the interworking with the SS7 network," Pracht says. The SS7 network "really contains all the intelligence necessary to route the call. You don't want to duplicate this just because you're using VoIP."

The increased interconnection between networks adds another layer of complexity to the picture.

"Every mom-and-pop shop out there and some of the big boys are able to connect to your network, and you have to open up to them," says Jim Ticknor, national manager of diagnostic product sales at Inet Technologies Inc. (www.inet.com). "If you've got someone who's screwed up the network ... causing problems on the network, it could take links, nodes and networks down."

Not surprisingly, Murphy, Pracht and Ticknor think they have just the thing for the carrier harassed by these problems.

Murphy's solution is Minacom's complete packet-based testing solution, a combination of its DirectTest and DirectQuality products. These, he says, "will be able to provide the necessary industry-based standards and algorithms that will support VoIP and some of the emerging data over wireless segments. And that is significant, because you're no longer testing a couple of testing points with packets. A traditional call would be maybe three or four test points. It could be 1,000 [trouble] tickets in the packet world."

By bridging DirectTest and DirectQuality (Minacom's quality assurance software), Murphy says, a carrier can remotely monitor test devices from multiple vendors and monitor QoS on interconnected networks. Minacom licenses the software to vendors, starting at $75,000 and going up to $200,000, "depending on how many test device drivers we need to provide," says Murphy.

Pracht stands behind Agilent's test and measurement products, the newest of which is Internet Advisor 11.4. Internet Advisor allows network managers to detect frame relay network problems and make sure that QoS standards meet the requirements of SLAs. Internet Advisor is a protocol analyzer for LANs and WANs, and, according to Pracht, "supports seven different VoIP signaling protocols under development or deployment." Pricing starts at $15,000 but depends on the kind of software and interfaces a carrier has, Pracht says.

Agilent's Telegra Voice Quality Tester 2.0, which measures the impact of echo noise distortion on QoS levels for IP-based network and services, is base priced at $39,000. For another $495, a carrier can add voice and fax over IP software that helps network engineers integrate the H.232 VoIP protocol into new IP-based products and services during the design and installation process.

Agilent also offers FASTest. This systematically tests SS7 and ISDN networks when new services and features are deployed. FASTest 3.1.1 includes automated controls to support Agilent's ATM test system remotely. The starting price is $55,000.

Ticknor and his colleagues at Inet offer the Spectra System, a hardware/software protocol analyzer that, he says, performs multiprotocol analysis simultaneously, which is particularly important in SS7/IP interoperability testing. Paul Renning, director of product technology at Inet, points out that large carriers bring new network elements into their labs to test them before deployment. "If you're little, you still have to validate your equipment to make sure it meets standards," he says. "Plus, you have to prove to Bell Atlantic Corp. [www.bell-atl.com], or whomever, that it works before you connect it to their network. Spectra will help you do that."

Carriers say their challenges are not just about convergence and QoS, but about testing and measuring networks, systems and equipment that are so new that their parameters of test and measurement are still fluid. For some of them, there aren't any ready-made solutions at the moment, so they're improvising.

The traditional telephone network tested and measured itself to a large degree. Switch vendors, DLC vendors and others built test routines and self-monitoring features into their products. Telcordia Technologies Inc. (www.telcordia.com), formerly Bellcore and known as "the research lab of the Baby Bells," built network monitoring systems that could tell network engineers when something bad was about to happen. The information that came out of these arrangements was "aggregated," telling network managers what was happening, or about to happen, with their networks as a whole. Focal, like many carriers, straddles the old world and the new, and most of the information Barnicle gets is still aggregated. But Barnicle and his colleagues find themselves trying to measure things on a "disaggregated" basis.

"Now we're forced to measure and test things on a disaggregated basis, so we can show the customer a report card," he says.

Having no system in place to do this, Barnicle and his colleagues do the best they can with what they have, and they lean on their equipment vendors for new test and measurement capabilities.

"It's a challenge," Barnicle says. "A lot of it is manual today. We're working with our vendors to come up with automated ways to take the output of their boxes and through some sort of web interface to show our customers what they've got at any point in time. Now we have to take the output, do some manual manipulation in desktop or Excel and then post the information for our customers. We're looking for vendors who can do that in a more automated way."

Most of the systems used at Focal are designed to let Barnicle and his colleagues see into their own network, but do not allow customers the same view. Some of his sales people have been proactive in filtering information obtained this way so that it can be meaningfully presented to their customers, but it isn't quite the same thing as having a direct window, he says.

At Mpower, Boersma isn't sure his customers want a direct window into the network. His company, unlike Focal, targets small and medium-sized businesses, and what they want is bandwidth.

"We don't have rigorous and complex SLAs," Boersma says. "We just have a lot of bandwidth, and they seem happy with that."

Customer self-service is a fine thing, he says, and Mpower is headed that way. But for now, Mpower's customers seem focused on having enough bandwidth at a reasonable price, rather than how the bandwidth is being used.

Even if no carrier can avoid touching the old circuit-switched world somewhere, there are carriers who touch it less than others, and for them, testing and measuring can be relatively simple matters up front.

For DSL provider Zyan Communications Inc. (www.zyan.com), much of the heavy-lift testing is done by its wholesale DSL partners, Covad Communications Co. (www.covad.com) and NorthPoint Communications Inc. (www.northpoint.net). They use test sets and toolkits purchased from the Harris Corp. (www.harris.com) to make sure the loops are up. Harris gets high marks from Nathalie Pongpanik, Zyan's provisioning manager. "Harris has been great for Covad and NorthPoint," she says. "When I started in provisioning, you had to roll a truck [to test the loop] every time an installation happened. The Harris stuff cut days off provisioning; you could just run a test."

Within 24 hours of receiving a customer's order, Pongpanik says, Zyan forwards the order to NorthPoint or Covad. Depending on the speed of the local ILEC's provisioning process, the confirmed order comes back to Zyan anywhere from one to eight days. Within 24 hours of the ILEC "dropping the circuit," Pongpanik says, NorthPoint or Covad runs a "Harris test" to make sure the ILEC completed the work. They get loop qualification, noise level, and hopefully, a signal. If there is no signal, they open a trouble ticket. Covad or NorthPoint installs the inside wiring, and then runs a "ping test." "If it comes back, if it's pinging, then it's talking to our router, and we're talking to that equipment, and we're ready to go."

Terry Lee, Zyan's chairman and founder, says his company also performs a "lot of IP level testing" using public domain software. But he can see the day coming when that software, and the testing done by his partners, won't be enough.

"We're actually evaluating retail products that do stuff like network monitoring," Lee says. "Our network is growing considerably. We have 12 PoPs now, and should have 50 operational by the end of the year. The basic utilities aren't good enough; we need a more comprehensive test and measurement system."

At a fixed wireless provider like Teligent, they have a different challenge.

"We were pioneers in point to multipoint, so we face the challenge of not even having any test and measurement equipment for the radio part of the network," Teligent's Akhavan says. "We were [using] IP and ATM, [and] the troubleshooting for IP and ATM is different. We needed more sophisticated equipment. There were no specific test measurements for what we were doing."

Teligent's technicians employ spectrum analyzers from various vendors to check for unexpected radiation in an area where they're about to set up a radio antenna. They measure the noise floor, trying to stay away from sites with too much ambient racket. They measure "link margins" to determine the strength of the signal, and finally they measure error rates, ATM cell loss, and delay.

Vendors say the worst move a carrier can make regarding test and measurement is to do nothing, or to assume that someone else will do it for them, such as their equipment vendor, their software vendor or their wholesale providers. Agilent's Pracht said the dumbest thing he has seen any carrier do was not to test at all. "We have seen service providers just connect multiple phones and faxes to a setup, try it out, and it seems to work, so they roll it out," he adds. "They don't really test under load and with new network components."

Akhavan says his biggest test and measurement headache comes from his OSSs, of which Teligent, like many competitive carriers, has quite a collection. Teligent's OSSs often come from its equipment vendors, and it has more than one of those.

The OSSs themselves are always missing some functionality, Akhavan says, which vendors supply on the next upgrade. By then, Teligent needs a new functionality, which the vendors supply on the next upgrade. And so it goes.