Designing High-Density Data Centers in 6SigmaRoom

6 September, 2018

Will Standard Cooling Methods Suffice in High-Density Designs?

In this blog, we’ll explore whether standard cooling infrastructure designs utilizing simple cooling capacity and airflow requirement calculations are appropriate for very high-density facilities. We’ll use an example model to demonstrate the importance of understanding airflow delivery when dealing with such a scenario, and conclude with a proposal that virtual modeling is the only cost-effective and viable method of doing so. 

Virtual Model of a high-density data center in 6SigmaRoom

The Virtual Facility (VF) model we’ll use for this example includes a cooling capacity that is more than twice that of the installed IT.

Data center rack-level power definition plot

The room details are as follows: 

  • Data Hall = 8611 sq/ft
  • IT Power Density  = 240W/sq ft 
  • Number of Racks = 216
  • Cooling Capacity = 21x250kW units = 5250kW
  • Room IT Load = 2060kW (Average of 9.54kW/Rack)
  • PDU Load = 54kW
  • CRAH Max Airflow = 500,000cfm
  • Equipment Max Airflow = 316,500cfm
  • IT Critical Temperature = 100F
  • Cooling Control = Return Temp of 90F

Baseline Simulation Results

The baseline model’s overheat plot shows the need for a well-designed cooling delivery strategy when operating a high-density data center. In this case, despite the over-provisioned cooling infrastructure, only 25% of the power load is being sufficiently cooled.  

Baseline model rack-level overheat plot

The simulation results highlight many areas where the cooling strategy’s effectiveness could be altered, but the most obvious issue arises from the control system. Using return temperature control is dangerous in many capacity loading scenarios, and high-density loading compounds this problem. While some of the CRAC units are running at maximum cooling capacity, others are only running at 10%; this is due to a huge variance in return temperature that originates from asymmetrical room loading. As a first step, we need to ensure that the control system is not inhibiting the cooling infrastructure’s capabilities. The simplest way to achieve this is by switching to a supply control strategy.

Supply Control

  • Cooling Control = Supply Temperature of 65F

The supply control strategy provides an immediate improvement, but it is still insufficient: 45% of the power load is now being sufficiently cooled. On closer review of the simulation results, we can see that many of the issues lie within the higher-loaded racks in the center of the room. This suggests that there are airflow issues causing recirculation in these regions. To combat this, there are two effective methods that can be considered: hot and cold aisle containment. The Virtual Facility allows us to test both methods for resilience before any deployment decisions are made. 

Supply control rack-level overheat plot

Supply Control Model - Recirculation

 

Recirculation over and around racks in an overheating row

As expected, there is a large amount of recirculation occurring in this row. A deficiency between the IT airflow requirement and the floor grille flow rate causes low-pressure regions in the cold aisle. This results in air being drawn over, through and around the racks, which increases inflow temperatures. Installing containment is an effective way to combat this recirculation. While this does not rectify the flow deficit, the recirculation flow path above and around the racks is blocked so the amount of hot air reaching the IT inflows is reduced. Again, the effect of this installation can be simulated and analyzed.

Cold Aisle Containment (with supply control)

Cold aisle containment rack-level overheat plot

Looking at the simulation results, it’s immediately clear that containment increases the effectiveness of the airflow delivery: 95% of the cooling load is now being sufficiently cooled. Adding cold aisle containment separates the cold and warm airstreams, which reduces mixing and increases efficiency. However, this reduced mixing results in higher air temperatures in the IT exhaust regions, so even a small amount of recirculation through well-sealed containment and rack blanking could result in overheating IT equipment. This issue is apparent in all forms of containment. Due to the over-specification of cooling infrastructure in this room, another form of containment should also be considered: a flooded room, hot aisle containment scenario.

Hot Aisle Containment (with supply control)

Hot aisle containment rack-level overheat plot

Installing a hot aisle containment system requires more retrofitting than the previous scenarios, including CRAH unit ducting and a dropped false ceiling. However, this solution results in a sufficiently-cooled room, with all the IT equipment receiving air 9F or more below the critical temperature of 100F

Conclusion

The results highlight the ineffective nature of designs using simple heat addition and rejection calculations. These calculations miss vital details: while at room level the cooling strategy can satisfy thermal limits, without modeling it is impossible to know whether this is reflected at cabinet or IT level. 

The Virtual Facility made it possible to install a 2MW load in a facility with less than 9000sqft of white space. While many modeling caveats must be considered when applying this methodology to a real scenario, it is undeniable that the VF gives engineers a scientific foundation to build from without requiring costly testing or inaccurate assumptions. Following these compliance-centered simulations, you can use the VF to explore different ways of increasing the facility’s efficiency. It should be noted that installing containment, either hot or cold, is a significant investment and should not be considered as a panacea for airflow issues. Additionally, the simulations described here do not include redundancy considerations; you can easily investigate these scenarios in the VF by switching off the most highly-loaded CRAH units.


Blog written by: Mike Eccles, Consultant Engineer

Other Recent Posts


Managing Energy Consumption for the Surging Data Center Market

Data center operations today are not the same as they were 10 years ago. Between the rapid growth o…

Read More


HF Lenz speeds up design process with 6SigmaRoom and Rescale

Engineering firm uses 6SigmaRoom and Rescale to Deliver Fast and Accurate Results to Demanding Clie…

Read More