In a recent report published by Research and Markets, the data center cooling industry is projected to rise from a value of $9.27B to $18.39B by 2024 at a CAGR of 12.3%. To meet performance demands, owner-operators are having to incorporate more blade servers and storage devices, sometimes producing heat exceeding 30kW-50kW/server rack (Research and Markets). Data center owner-operators will have to invest significantly in new cooling technologies to ensure maximum uptime and protect the long-term capacity of the data center. This is especially true in the case of high-density designs.
The digital twin provides owner-operators a model with which to manage these new and growing demands. The digital twin is a three-dimensional, physics-based, virtual representation of the physical data center that encompasses the entire data center, including IT equipment, cabinets and the supporting power/cooling infrastructure. The digital twin virtually replicates the physical behavior of the data center—from design to operations. This blog illustrates the power of the digital twin when it comes to managing a high-density data center.
Will Standard Cooling Methods Suffice in High-Density Designs?
In this blog, we’ll explore whether standard cooling infrastructure designs utilizing simple cooling capacity and airflow requirement calculations are appropriate for very high-density facilities. We’ll use an example model to demonstrate the importance of understanding airflow delivery when dealing with such a scenario, and conclude with a proposal that virtual modeling is the only cost-effective and viable method of doing so.
Figure 1. Digital Twin Model of a high-density data center in 6SigmaRoom
The digital twin model we’ll use for this example includes a cooling capacity that is more than twice that of the installed IT.
Figure 2. Data center rack-level power definition plot
The room details are as follows:
Baseline Simulation Results
The baseline model’s overheat plot shows the need for a well-designed cooling delivery strategy when operating a high-density data center. In this case, despite the over-provisioned cooling infrastructure, only 25% of the power load is being sufficiently cooled.
Figure 3. Baseline model rack-level overheat plot
The simulation results highlight many areas where the cooling strategy’s effectiveness could be altered, but the most obvious issue arises from the control system. Using return temperature control is dangerous in many capacity loading scenarios, and high-density loading compounds this problem. While some of the CRAC units are running at maximum cooling capacity, others are only running at 10%; this is due to a huge variance in return temperature that originates from asymmetrical room loading. As a first step, we need to ensure that the control system is not inhibiting the cooling infrastructure’s capabilities. The simplest way to achieve this is by switching to a supply control strategy.
Supply Control
The supply control strategy provides an immediate improvement, but it is still insufficient: 45% of the power load is now being sufficiently cooled. On closer review of the simulation results, we can see that many of the issues lie within the higher-loaded racks in the center of the room. This suggests that there are airflow issues causing recirculation in these regions. To combat this, there are two effective methods that can be considered: hot and cold aisle containment. The digital twin allows us to test both methods for resilience before any deployment decisions are made.
Figure 4. Supply control rack-level overheat plot
Supply Control Model - Recirculation
Figure 5. Recirculation over and around racks in an overheating row
As expected, there is a large amount of recirculation occurring in this row. A deficiency between the IT airflow requirement and the floor grille flow rate causes low-pressure regions in the cold aisle. This results in air being drawn over, through and around the racks, which increases inflow temperatures. Installing containment is an effective way to combat this recirculation. While this does not rectify the flow deficit, the recirculation flow path above and around the racks is blocked so the amount of hot air reaching the IT inflows is reduced. Again, the effect of this installation can be simulated and analyzed.
Cold Aisle Containment (with supply control)
Figure 6. Cold aisle containment rack-level overheat plot
Looking at the simulation results, it’s immediately clear that containment increases the effectiveness of the airflow delivery: 95% of the cooling load is now being sufficiently cooled. Adding cold aisle containment separates the cold and warm airstreams, which reduces mixing and increases efficiency. However, this reduced mixing results in higher air temperatures in the IT exhaust regions, so even a small amount of recirculation through well-sealed containment and rack blanking could result in overheating IT equipment. This issue is apparent in all forms of containment. Due to the over-specification of cooling infrastructure in this room, another form of containment should also be considered: a flooded room, hot aisle containment scenario.
Hot Aisle Containment (with supply control)
Figure 7. Hot aisle containment rack-level overheat plot
Installing a hot aisle containment system requires more retrofitting than the previous scenarios, including CRAH unit ducting and a dropped false ceiling. However, this solution results in a sufficiently-cooled room, with all the IT equipment receiving air 9F or more below the critical temperature of 100F.
Manage High-Density Designs with the Data Center Digital Twin
The results highlight the ineffective nature of designs using simple heat addition and rejection calculations. These calculations miss vital details: while at the room level the cooling capacity appears to be high enough to satisfy thermal limits, without modeling it is impossible to know whether this is reflected at cabinet or IT level.
Following these compliance-centered simulations, you can use the digital twin to explore different ways of increasing the facility’s efficiency. It should be noted that installing containment, either hot or cold, is a significant investment and should not be considered as a panacea for airflow issues. Additionally, the simulations described here do not include redundancy considerations; you can easily investigate these scenarios in the digital twin by switching off the most highly-loaded CRAH units.
The digital twin made it possible to install a 2MW load in a facility with less than 9000sqft of white space. While many modeling caveats must be considered when applying this methodology to a real scenario, it is undeniable that the digital twin gives engineers a scientific foundation to build from without requiring costly testing or inaccurate assumptions.
This blog was originally published on June 27, 2018 and updated on July 18, 2019.*
Blog written by: Mike Eccles, Consultant Engineer
Other Recent Posts
What’s Causing Sleepless Nights for Data Center Professionals?
We explored several trends in digital transformation and how this transformation is impacting data…
Digital Transformation's Impact on Data Centers
Will the next twelve months see the same pressures placed on IT infrastructures and the data center…
18 July, 2019
