As more companies move their mission-critical applications to the web, data centers are beginning to resemble the central switching offices that telephone companies use to offer lifeline voice services. While the actual equipment between the two differs significantly, both are built with a major emphasis on reliability, scalability and security.
That means raised floors under which power and air conditioning can run, backup power, strict climate control, fire control, redundant equipment and connectivity, and controlled access.
"The most important part of a data center is all the environment," says Sam Mohamad, president of worldwide sales and international field operations at Exodus Communications Inc. (www.exodus.net). "You have to build them to such standards that there's absolutely no point of failure."
That's an expensive proposition. Depending on who you talk to, the minimum investment in a data center facility is somewhere between $5 million and $20 million. And that's just for the building itself. Add to that the costs of equipment and personnel to fill that building, plus the cost to connect the building to the rest of the world.
Mitch Ferro, director of hosting products at UUNET Technologies Inc. (www.us.uu.net), a WorldCom Inc. (www.wcom.com) company, says it costs UUNET about $10 million to construct a 50,000 square foot facility. The network equipment could cost another $10 million to $20 million, he says.
The Basic Utilities
According to Ferro, power is the most difficult challenge in a data center.
"A power failure is the most painful thing for a customer," he says.
Not only can it result in a customer being offline for a time, but it can also corrupt the company's database, he says. To avoid these possibilities, UUNET provides a redundant power supply. In its Atlanta data center, for example, the company has two separate connections to the power company, and each connection is to a different substation at that power company. UUNET also has a redundant uninterruptible power supply (UPS) for battery backup. And each UPS is backed up by redundant diesel generators, which can support the data center from
72 hours to one month, depending on how full the data center is with customers at that time. "Most customers couldn't afford to do that themselves," Ferro notes.
Qwest Communications International Inc. (www.qwest.com) has a similar setup. According to Kurt Cohen, vice president of Qwest's hosting and collocation services, each of the cabinets in the company's CyberCenters has redundant power-- distribution units backed by UPS backed up by redundant generators. And, Cohen notes, Qwest has contracts with fuel companies so if it had to use the generators, which can run for 10 hours at a stretch, it would have more fuel available within two hours.
Of course, data center operators also have to have cooling systems to keep the environment at an optimum temperature for the equipment. Ferro says UUNET even offers redundancy with its cooling towers. "We have water delivery, so if we had a city water outage, we have water delivery just in case," he says.
And to protect the data center in the event of a fire, UUNET has installed fireproof flooring and sprinkler systems. Because a broken water pipe or accidentally triggered sprinkler system could do as much damage as a fire, the pipes going to the sprinkler heads don't contain water, and the alarms must sound in two areas of the building and a piece on the sprinkler head must melt to switch on the sprinklers, he says.
Cohen says Qwest's advanced smoke detection systems are "so sensitive that if you smoke outside and breathe inside it will detect those particles and an alarm will go off." And when an alarm goes off, controls signal where in the building the smoke was detected. That way, Cohen says, an automated system can shut off that zone and extinguish the fire using gas or water.
Guarding the Goods
If a power failure or fire doesn't knock out service, a security breach could. Heeding Murphy's Law, many data center operators have elaborate access procedures.
For example, to get into UUNET's Dallas data center, individuals must first identify themselves as one of the people on its customer list. They must then place their hand in a biometric palm reading device, which measures the size of the hand, and enter a four-digit code. Once in the building, a security guard in the lobby checks driver licenses. People must then punch in their account number and password to open a box and get the keys to their cage or rack of equipment. They then have to do another palm check and code entry to get into the data center.
"There are multiple challenge response steps to get to your cage," says Ferro. "In Dallas we back that up with cameras" and onsite staff for security and customer assistance.
Qwest has a similar security procedure, according to Cohen.
In addition to the physical security, data center operators offer logical security in the form of firewalls and other technologies.
Plugging In
Data centers also typically have redundant connectivity to an Internet backbone provider or peering site.
TeleChoice Inc. (www.telechoice.com) senior consultant Ruth Chatterton says companies building a data center would want at least two T3 connections into the center--one for daily use and one for backup.
But some data center operators have much faster connections.
Qwest and UUNET note that they offer the added benefit of integrating their data centers with their large backbone networks, so customers can easily upgrade their bandwidth to the outside world.
"The centers are right on our core fiber backbone," says Cohen. "We have our backbone routers right in this center so there's no local loop involved, so performance is improved, and it's easier to upgrade. We have dual OC-48 today, but we could upgrade that easily."
UUNET also is "building these data centers into the network," says Ferro. For example, the company's hosting center in Dallas is in the same building as its Dallas megahub.
"So if a customer needs multiple OC-12s, [we can provide that quickly]," he says. "If we were an independent web-hosting company, we'd have to order that from another carrier. But UUNET can pull fiber from the megahub to the hosting area in the same building."
The Box Step
Then there's the small matter of the actual equipment.
The number and functionality of "boxes" found in a given data center varies based on the operator and the needs of its customers. But the typical makeup of a data center includes routers connected to Ethernet switches, which are connected to servers via Ethernet connections over fiber.
Jose Garcia, vice president of business service providers and emerging markets for IBM Corp.'s Enterprise Systems Group (www.ibm.com), explains there are three classes of servers found in a data center: back-end, transaction-based servers; web servers, which handle web pages or run applications such as e-mail; and specialized servers, which act as firewalls or TCP/IP servers, for example.
Network management tools also are becoming increasingly important tools for data center operators.
"We try to avoid success failures," says Ferro of UUNET, referring to how the popularity of an Internet site can be its downfall if it slows or crashes from all the attention.
To avoid that fate, UUNET uses Alteon load balancers, which enable it to load balance traffic across multiple web servers.
"If you have the same information on multiple servers, it does balancing and monitors health of servers and redirects [traffic] to other servers if there's outage," he explains.
Network management also is key for such popular new applications as streaming, notes Mohamad of Exodus.
"Streaming involves a much higher level of quality and commitment on the network itself because it's a stream that's continuous instead of an IP package that's in bits and bytes," he says.
Exodus recently acquired a company called Service Metrics, which has a hardware/software solution built on an Oracle database that mimics a stream. The data center operator or its customer can mimic any kind of an event on the network, and it's fed to 120 locations around the world, says Mohamad. Performance results of that event can then be reviewed immediately or read into an Oracle database.
The tool could be used to enable a data center operator to pinpoint faulty equipment on its network or the networks of its users--such as a problem with a domain name server (DNS), he says.
As more companies and applications are added to the network, unearthing problems and ensuring reliability are only becoming more difficult. But data center operators are certainly putting forth significant effort to meet the challenge.
TeleChoice's Chatterton says, "The biggest challenge is making it reliable layer upon layer upon layer," while still being able to reap a profit.
