An ISP's infrastructure is built something like a wedding cake, in tiers. A broad base of users feeds into a large number of COs or PoPs, which in turn funnel up to a city or superPoP, and that in turn culminates at one or more national data centers, huge packet-processing nexus with server farms and gigabit routers.
"It started with what would now be considered a very small PoP and went to the tiered architecture very early on," says Ruth Chatterton, senior consultant with consulting firm TeleChoice Inc. (www.telechoice.com).
Unlike the real thing, though, this wedding cake is always growing. The base will get broader as more users demanding more bandwidth come online. The superPoPs and data centers grow in response. What was a superPoP or even a data center a few years ago may qualify as little more than a busy PoP today.
With scalability more important than ever before for ISPs, new solutions are appearing to deal with the issue. Some help small to medium-sized providers work with the legacy public switched telephone network equipment that is still the primary means of access and transport for many ISPs. Others provide unique solutions within PoPs themselves, aggregating traffic in new ways that cut costs and the number of boxes in data centers.
The lowest tier of the layer cake, the access layer, is the segment of the market with the greatest development and the greatest challenges, says Chatterton, "because there are many more things to manage there: figuring out where to put the aggregators and which combination of access speeds and technologies to use. There are a lot more decisions to make."
For smaller ISPs, with perhaps one superPoP and a number of dispersed PoPs, these are critical business decisions because of the low margins that have become their lot in life, as well as the inflexible bandwidths of telco private-line offerings. For Southwest Cyberport Inc. (www.swcp.com), an Albuquerque, N.M., ISP, the problem became acute in the links from its remote PoPs to its central location.
In the past, a T1 link from Santa Fe, for instance, would have sufficed, but with a significant number of customers now using DSL or fractional T1 in addition to dial-up, more bandwidth was needed. "We want six to eight megs for a given link," says Mark Costlow, co-owner. "But in our location you have to get from eight to 10 T1s of bandwidth before you hit that break-even point where a T3 is cheaper."
Southwest Cyberport adopted a new solution by Tiara Networks Inc. (www.tiaranet.com) that multiplexes T1s for IP and frame-relay networks. Unlike other "inverse multiplexing" solutions, Tiara uses multilink point-to-point protocol (MLPPP), a Layer 2 protocol that multiplexes the flows packet by packet rather than bit by bit. With headers intact, the equipment can use IP protocols for routing, forwarding and prioritization. It can also aggregate traffic before that traffic is sent to a switch or router, cutting down on the processing required of the router.
An unexpected advantage of the Tiara solution, Costlow says, is it can spread traffic over T1s from multiple services providers, enabling the ISP to look for the cheapest provider at any given time in a geographical area. Another player with a new approach to inverse multiplexing is Larscom Inc. (www.larscom.com). The company introduced its iPLEX 6000 in January, which it planned to make available in the first quarter of 2000. It provides multiplexing of T1s and soon will add DSL, but adds the extra capability of service delivery and provides frame relay, ATM and IP services. The four-port T1/E1 product will cost about $10,000.
The University of Texas System (www.utexas.edu) is experiencing similar growth pressures, says Wayne Wedemeyer, manager of networking and telecommunications. It is a medium-sized system, but still relies heavily on T1 connections to its customers, which are schools and community colleges.
Wedemeyer has found the Texas system can best create new connections to customers by aggregating them to a sufficient level to justify buying a DS-3. "We have found that we can do new connections with a variety of solutions, including high-density cards for routers that accept a DS-3. That way, we can bring in 28 new connections."
The challenge has been expanding existing connections from T1 to higher levels, and for this Wedemeyer has opted for T1 aggregation, sometimes using Tiara at the customer premises. "Until I have 28 T1s filled, I don't have to buy another piece of hardware. I can aggregate all that bandwidth in one DS-3."
The University of Texas System is at the point at which it could consider moving to the next level of device--a solution such as Cisco Systems Inc.'s (www.cisco.com) Cerent technology, which aggregates multiple traffic sources to a gigabit Ethernet stream on fiber. "We are excited about their approach, but I have to get a little bigger. I'm just below where Cerent starts to make a lot of sense," Wedemeyer says.
One of the truisms of IP PoP equipment is that nothing is ever thrown away. As the PoPs get bigger, larger, more capable equipment is brought into a data center or superPoP and older devices migrate outward. However, says TeleChoice's Chatterton, "at some point it becomes cheaper to throw away the older equipment that does not have as dense a population of ports on it, and bring in newer equipment that can combine many different kinds of access, which is a boon to ISPs."
This is where solutions such as Cerent begin to play a role. Cerent is part of a new generation of aggregation and switching equipment, sometimes called edge routers or edge aggregators, that is coming onto the market to meet this need. They provide a very broad range of ports for aggregation, down to T1 and lower, while feeding and terminating multigigabit fiber connections.
But this new generation is also "a challenge to Cisco and Cisco's domination at the edge of the network," says Jeff Wilson, director of corporate access, Infonetics Research Inc. (www.infonetics.com). "For the first time ... the multiprotocol router may not be the ideal device at the edge of the network. I look at it as like what Microsoft went through when they had to upgrade from DOS to Windows or to 32-bit."
The most recent entry to this arena is ECI Telecom Ltd.'s (www.ecitele.com) XDM, launched at the CeBIT convention in Hannover, Germany, at the end of February. Although designed as an optical aggregation platform that can feed up to terabit-level DWDM fiber connections, the XDM nevertheless can connect directly to subscribers at speeds from 64kbps to 100gbps. In a market where 90 percent of all business customers still con nect at T1 rates, aggregating those customers to DWDM backbone speeds has required layers of high-cost systems, such as edge data multiplexers and DWDM multiplexers. By combining many functions in a single platform, ECI believes it can cut network deployment costs significantly.
Another solution aimed at this market is the ERX edge router by Unisphere Solutions Inc. (www.unispheresolutions.com). A recent entrant to the field, Unisphere Solutions, a Siemens AG startup (www.siemens.de), calls its ERX platform an edge router, even while saying the terminology is inadequate for what the product can do. Other terms are multiservice access platforms or even broadband remote access servers (BRAS).
"It is a new kind of network element," says Jon Mischel, senior product manager, data division, Unisphere Solutions. "The traditional routers were developed to terminate T1 connections and provide access on and off the Internet. In the router that we are providing, the architecture is remarkably different from what has been done. We can terminate T1, T3, OC-3 and OC-12, and we will strip down to individual IP flows, yet maintain performance and speed."
It is an edge router, he says, "that has all the bells and whistles to provide next-generation services and do that at wire speed. This is so service providers can turn on all the services they want per user without degrading performance. That is combined with a management system to offer whatever services the subscriber is asking for." Unisphere has made the product access-agnostic, introducing a version for cable modem networks as well as more standard DSL or T1 access.
RMI.NET Inc. (www.rmi.net), a Denver-based ISP, has deployed the ERX-1400 to enable its nationwide rollout of DSL service. The ERX is placed between DSL access multiplexers and Internet backbone routers, providing termination and aggregation for as many as 32,000 DSL connections.
Another important part of the scalability equation is knowing when to scale. Clairvoyant Software Inc. (www.clairvoyant.com) announced in December its ForeCAST Resource Manager, which includes a module for Internet access metrics. Gregory Hampton, president and CEO, says ISPs need to know "when and where to add capacity in front of demand, but not so far in front that they have cash-flow problems."
Clairvoyant puts a PC at every PoP to monitor flows and tally when saturation occurs and for how long. The data is forwarded once a day or more to a central location, where it is evaluated. ISPs can decide how many busy minutes per day they want to impose on their customers. The product replaces many homegrown solutions the ISPs have created over the years with a vendor performance monitoring and planning solution.
