The cost to build and operate a modern data center (DC) continues to increase. This Total Cost of Ownership (TCO) includes capital and operational expenses. The good news is that compute capabilities in the DC are increasing at a much higher rate than the TCO. This means the actual cost per unit of compute performance is coming down.

One common belief is that the TCO of a new DC is lower with a low-density design because it doesn't require as advanced of a design, nor as elaborate of power delivery and cooling systems. To find out if this is true, Intel researchers compared the purchase costs of architectural space, power systems and cooling systems, as well as lifetime operational costs, of high-density DCs having densities of 1,000 watts per square foot of work cell to the same costs for modern low-density designs of 200 to 400 watts per square foot of work cell. The results were surprising and proved conventional thinking wrong. We also developed a list of considerations for successful construction and operation of high-density DCs.

APPLES TO APPLES

The first thing we had to establish was a solid basis for comparison. For our research, we used repeating rows of racks, side by side with alternating cold aisles and hot aisles, all over a raised floor providing cool air, good hot and cold air segregation, and well designed hot air returns. Note that liquid cooling was not used. (Our on-going research shows that supporting up to 30 kW racks with air cooling is feasible and provides a lower TCO than liquid cooling.) "High density" was defined as racks at 14 kW and above with a work cell of nominally 16 to 20 sq. ft. The metric we use is "watts/sq ft of work cell." A "work cell" is a repeating unit of cold aisle, rack and hot aisle. (See Figure 1.) Work cells represent the square footage directly attributable to a specific rack of server. For DC-to-DC benchmarking, "watts/sq ft of work cell" provides a more specific metric, than the more frequently used but less precise "watts/sq ft."

A TALE OF TWO DATA CENTERS

For our comparison, we used a requirement of 10,000 1U dual processor servers. (Equipment racks are measured in U, with a typical rack being able to hold 42 1U servers. 1U = 1.75 inch height.) We determined the low-density configuration using the median of a survey of 28 new or retrofit data projects being considered by a leading DC design/build firm. The median was between 125 and 150 watts/sq ft, which represents a 6.6 kW rack. For the high-density DC the goal was full racks and minimizing square feet. Thus, we fit 42 of the selected servers in a rack, creating a power requirement of just under 17 kW per rack.

Figure 2 shows the major differences between the two DCs. The low-density DC requires 625 racks and 28,571 sq. ft. of raised floor. The high-density DC requires just 238 racks and 10,880 of raised floor.

Comparison of the initial setup of both data centers.

Now let's consider cooling. The total airflow is driven by the number of servers, not the space density. Thus each DC is cooled by the same 39 cubic feet per minute (CFM) of cooling airflow, plus a 20 percent safety factor for leakage and bypass. A value of 35 percent might be more appropriate in either DC if less care is given to design, computational fluid dynamics (CFD) analysis, and operational expertise.

WHAT'S DIFFERENT ABOUT HIGH DENSITY

A high-density DC does require some extra consideration, but it also has some real advantages. Below are a few things to consider when building and operating a high-density DC.

A high-density design requires a higher building height to accommodate a higher raised floor. Instead of the 18" floor required for the low-density design, a 30" raised floor is necessary. This is to handle the higher per-rack flow rate for the high-density cabinets. The 30" raised floor raises the cost by approximately $1 per square foot.
A high-density design takes less land. In this case, the high-density design requires nearly half of an acre less land. So land costs are less and more than offset the cost of the higher building. Plus, since permitting and fees for any project are often based on square footage, this can mean additional savings. How much depends on the area.
Overall civil, structural and architectural (CSA) costs are less. In fact, in our comparison, an approximately $4 million savings is possible because of the fewer square feet.
Lower lighting costs. Less building, less space to light. And less bulbs to replace over time.
Fewer racks. Racks cost money and the low-density design pays dearly for only filling them each partially - it has 387 more of them in our example.
Lower cooling costs. This is a surprising advantage. It turns out the central cooling plant is the same for both, but the major difference in cost comes from the power required to move the air. Both have the same total flow rate, but there's less static pressure required to drive the proper airflow in the high-density design because of the 30" raised floor and the use of grates that are more than 50 percent open. The low-density design, with its 18" raised floor and 25 percent open perforated tiles needs more static pressure to maintain the proper airflow. (Using more open grates with the low-density design to reduce the work for the fans wouldn't help here because the lower raised floor and reduced CFM would affect the uniformity of the airflow.)

COST COMPARISONS AND MAJOR COST DELTAS

In the end, the high-density DC is measurably less to build over the comparable low-density design. As Figure 3 shows, the overall savings totaled $5.2 million in the areas where the major differences occur.

Data Center TCO comparison and major cost deltas.

High-density DCs do require some different methods in design and operations, which we will briefly touch on now. But, weighed against the lower TCO they give you, high-density DCs are well worth the extra work.

HIGH-DENSITY DESIGN AND CONSTRUCTION CONSIDERATIONS

As you might expect, airflow is extremely important in high-density designs. It's not so much that you need more (you don't), it's that the cool air has to be delivered precisely where you need it. High-density DCs also should be equipped with more robust computer room air conditioners (CRACs) that provide monitoring, reliability, redundancy and capacity tailored to their heat ratio loads.

One smart tactic is to zone a high-density DC into homogenous server types. This makes cooling easier to perfect. (Non-uniform loading requires greater planning.)

OPERATIONAL CONSIDERATIONS (WILL IT BE TOO HOT?)

People often erroneously believe that in a high-density design the hot aisle will get too hot for personnel. The truth is, at the ranges being discussed here, the hot aisle temperature is independent of density. Whether there are 42 or 16 servers in a rack, the servers will pull the needed amount of air per server from the cold aisle and discharge it in the hot aisle. If the hot aisle is not hot in either design, then cooling capacity, energy and money are being wasted. Here, low-density designs are often more prone to such air mismanagement since there is a larger margin for error - and consequently waste.

Airflow velocity will be higher in a high-density DC. In the design discussed here, it will be 500 feet per minute (fpm), the middle operating range of a typical grate with about 56 percent open area. This is well within the reasonable range for these types of airflow.

SHOULD YOU MOVE TO HIGH DENSITY?

High-density DCs do require more careful design. Raised floors of 30 inches and grates instead of perforated tiles are necessary for the higher volumes of air. These higher floors will preclude some retrofits, but in other situations where you can make the conversion or build new, the benefits of a smaller DC with the same computing performance at a lower TCO are hard to pass up. ENS