Richard T. Ungar ABB Data Center Automation Toronto, Canada, richard.t.ungar@ca.abb.com
For the last six years, ABB has been pioneering automation that can replace traditional control and monitoring solutions for data centers. Now, many leading data center builders and operators rely on ABB’s systems and expertise to ensure their facilities run efficiently and reliably.
But what is ABB AbilityTM Data Center Automation and how is it an improvement on traditional methods? This article explains the differences between the two approaches and how the concept of converged, hardened, industrial systems will be applied to all data center facilities in the future.
A brief history of data centers
The multi-megawatt, custom-built data center behemoths of today grew out of the “computer rooms” and “server closets” that were simply rooms set aside as part of existing facilities. Power to these early “data centers” was provided through a dedicated infrastructure and, when specified, an electrical power monitoring system (EPMS) was provided by the vendors of the electrical gear. Typically, these systems were closely tied to the particular vendor and it was difficult (and often impossible) to mix and match equipment from different vendors.
To get rid of waste heat, the heating, ventilation and air conditioning (HVAC) and the building management system (BMS) – or building automation system (BAS) – of the host facility were simply extended to include the data center cooling equipment.
Management and monitoring of the data center was largely a function of the IT organization. If there were concerns about hotspots, standalone temperature monitoring was put in place. Although they often provided detailed views of the data center temperature profile, these devices were not tied into the BMS/BAS. If hotspots or uneven air distributions were identified, the responses were typically manual: computer room air conditioner (CRAC) setpoints turned up or down, floor vents moved around to change airflow, or fans positioned to redirect cold air.
Similarly, if IT personnel had concerns about power consumption, they would put the appropriate monitoring equipment in place. If detailed electrical observation was required, branch circuit monitoring (BCM) or in-rack monitoring methods were used. Again, these were typically standalone arrangements. Often, there was no power monitoring in place and consumption was calculated by totaling up the estimated consumption of each server.
Throughout all this, the divide between IT staff and facilities management remained rigidly in place. The facilities team monitored the system enough to ensure the cooling systems functioned and sufficient electrical power was delivered. IT consumed the power and cooling and did its best to optimize its use.
Dedicated data center construction
As data center requirements grew, dedicated buildings started to appear. Inevitably, power consumption skyrocketed and the corresponding increase in waste heat drove more sophisticated cooling designs. These new data centers bore little resemblance to the server rooms of the past, yet three facts did not change:
• The building HVAC remained a dedicated, standalone system, managed by the same BAS/BMS that was used to manage HVAC in other types of buildings.
• Electrical monitoring was still provided by the vendors of the electrical equipment.
• IT staff still ignored the BMS and EPMS and installed their own systems, when required.
Why was this? It was because these three conventions derived from the tried-and-tested models used for constructing the commercial buildings that formerly housed data centers. These models involved, for the most part, traditional builders contracting with traditional vendors to equip traditional commercial, non-mission-critical buildings (such as office buildings or shopping malls) and not modern data centers with quite different requirements.
Band-aid solutions
Modern data centers do not resemble traditional commercial buildings in any way; they are large, purpose-built, power-hungry, mission-critical pieces of infrastructure, with much more in common with industrial facilities than with commercial ones. Because many data centers are still built using traditional methods, they adopt design strategies that make implicit assumptions about control system failures, often at considerable cost. For example, complicated, multi-tier cascading BMS designs, where upper tier and lower tier controls perform the same action using parallel communication channels, each assuming the other can fail at any time, are common. Equally common are management systems that do not actually perform any management and in which individual subsystems manage themselves and exclude any possibility of mutual coordination.
These band-aid solutions all have the same underlying assumption: the control system is unreliable and cannot be counted on except for basic visibility or, at best, high-level coordination.
Silos
Traditional data center construction often causes the individual subsystems to be divided into standalone systems or “silos.” These silos correspond to the scope of supply of an individual vendor or subcontractor: the electrical contractor supplies the EPMS, the mechanical contractor the BMS and so on. No overall system supplier is tasked with unification or consolidation. The result is a duplication of material and effort since each system contractor will need to install, wire, program and commission separate systems that share common actions: read data from devices, move the data around the network, display the data on screens, inform users when the data is not what it should be and send data back to the devices when it needs to change.
Revolution rather than evolution
The evolutionary and piecemeal approach to data center control and monitoring described above scales up extremely poorly. Data centers are not just larger, more power-hungry office buildings. In fact, data centers manage critical data for global communications, financial transactions, government services, business operations and entertainment. Facility sizes of 50,000 m2 (>500,000 sq. ft) and power consumptions of
50 MW are no longer uncommon →01.
From a size and criticality perspective, modern data centers have much more in common with industrial facilities – power plants and oil refineries, for instance – than they do with commercial buildings. Given this fact, it seems natural that industrial-grade automation systems should be used to monitor and manage data center infrastructure. Industrial automation systems are robust and highly hardened systems that can scale from small to extremely large and from simple to extremely complex. They are designed to talk to all kinds of equipment using many different protocols and have been designed for continuous operations over long periods, even while being upgraded. Furthermore, industrial systems are cyber-secure, needing to comply to the rigorous standards set in the various industrial environments in which they operate.
ABB AbilityTM Data Center Automation systems are simply industrial automation systems adapted for use in data centers. They eliminate the requirement for band-aid solutions in the facility since they are designed to run continuously and reliably via built-in redundancy and are designed for scalability. Since they perform the functions of all the various siloed systems they replace, they are also cheaper to install, as only a single system is necessary to cover all building management and electrical monitoring tasks.
Advanced functionality: optimize operations and reduce downtime
By consolidating all the facility information in the ABB AbilityTM data center, Data Center Automation systems also become the natural platform for advanced functionality. Since all the data on power consumption, cooling, performance metrics and status are managed, in real time, by one system, advanced prediction and optimization can be performed. This data consolidation is especially important for energy saving and sustainability, as demonstrated in the case of the Lakeland Community College implementation →02.
Condition-based monitoring provides a good example of cost-savings technology. Here maintenance is triggered using predictive indicators, rather than after a set time interval. Health information is collected from the electrical and mechanical systems, aggregated, analyzed and compared to historical data to provide advanced warning of degrading equipment performance or impending failure. This approach optimizes operations, reduces the risk of downtime and eliminates waste associated with premature or unnecessary maintenance.
Predictive maintenance, energy optimization, dynamic load forecasting, etc., are possible only through consolidation of data center facility information – and consolidation is only possible when robust, reliable, industrial technologies are used →03.
Intelligent data centers
Technologies such as 5G and the Internet of Things are pushing data center designs to new levels of complexity. IT loads will become far more dispersed and fluid and the facilities that power them will have to be much more adaptable. Old efficiency metrics like power utilization effectiveness (PUE) will have little meaning in a world where IT load can transition from 0 to 100 percent and back to 0 percent over the course of a day to accommodate, say, telemetry data from self-driving cars during rush-hour (after all, how efficient can a fully powered data center with zero IT load be?) To be truly efficient, the facility itself has to be “self-driving” and predictive so it can deliver energy and cooling to handle such transient loads. Only through Data Center Automation can truly intelligent data centers that are sophisticated enough for these, and other, as yet unthought of, tasks be realized.
BUILT FOR SUSTAINABILITY AND FLEXIBILITY: LAKELAND COMMUNITY COLLEGE DATA CENTER
In 2011, Lakeland Community College, Kirtland, OH, United States, moved the school’s data center to a new campus facility and out of what Chief Information Officer Rick Penny [1] described as “mostly just a closet with some servers and move-in cooling units.” The school needed an all-new, sophisticated data center with more space and more configurable flexibility [1]. The facility also had to be energy efficient and LEED-certified as a green building [2]. Lakeland chose ABB Ability™ Data Center Automation as its Data Center Infrastructure Management (DCIM) solution¹ →02a, “And now,” Penny [1] said, “10 years later, I don’t know what we’d do without it.”
Since Lakeland Community College mandates that all new campus buildings be LEED-certified, they chose ABB due partially to the evidentiary reports of energy usage provided. Between 2006 and 2018, by focusing on sustainability and redesigning the way the heating and cooling works in its campus buildings, Lakeland could increase facility size by 18 percent while reducing electricity use 40 percent, natural gas use 49 percent, and water/sewer usage 30 percent [3].
According to Penny, ABB Ability™ Data Center Automation has been instrumental in helping significantly lower energy use and reduce costs. For example, by analyzing cooling data, they determined that the data center could reduce air conditioning demand by adding containment walls to the server rows →02b. Penny estimates that there is a 20- to 30-percent temperature differential inside the walled-off server containment areas as compared to outside these rows.
“We were able to shut down a big 10-ton AC unit that allowed us to save even more money than we were expecting,” Penny [1] said. By 2014, Lakeland’s new data center rated Silver LEED-certification status. Nearly a decade after moving into the new facility, Penny credits the ABB Ability™ Data Center Automation with helping reduce the facility’s energy usage by more than 53 percent.
Cool and cost effective
The increased operational visibility they got with the ABB solution, integrated with FNT Command from FNT Software, an ABB partner, has enabled Lakeland’s data center operations to more efficiently add servers and take advantage of emerging hyper-converged infrastructure (HCI) technologies. While converged hardware is typically smaller, it runs considerably hotter than traditional hardware, with power supplies often rated above 1,000 W, eg, a full rack of 2U-high HCI boxes could be 25-30 kW, while typical 1U servers are about 350-500 W each [4]. The school’s former data center was too small to support the additional air conditioning that HCI required.
Moving into the new facility with ABB Ability™ Data Center Automation enabled Lakeland to more efficiently arrange the data center layout and plan cooling configurations to better control energy usage without compromising uptime→02c. The college could take advantage of emerging data center trends and technologies, including shifting many of its servers to the cloud, reducing the number of servers they needed to cool, ultimately lowering energy cost.
Increasing uptime
David Levine, Associate Director of Administrative Technologies for Lakeland, thinks that from an operations and maintenance point of view, ABB Ability™ Data Center Automation with FNT Software, has been a game changer for its monitoring, alarm capabilities and planning efficiency.
“Our air conditioner works off the water temperature and if the water temperature gets too high, it alerts us. I have temperatures of everything, including how many KW every row and every rack consume,” Levine [1] said. “Plus, the DCIM has a water sensor around the racks so if there’s any liquid that gets on the floor there, it will alert us.”
“The best thing is that our server uptimes are almost 100 percent. Things happen in the data center just because there are physical components. But now, we get alarm notifications and can resolve issues before these things can become a disaster.” Penny [1] explained. Plus, Levine noted that his operators can more efficiently track inventory of all data center racks to understand the impact of adding something new.
What’s next for the Lakeland Community College data center? Penny and Levine plan to capitalize on the efficiencies they get with the ABB Ability™ Data Center Automation system and HCI to reduce the data center footprint by 50- to 66-percent of its current size. "We are looking forward to the energy savings," said Penny [1].
Footnote
1) The original name for ABB’s solution purchased by Lakeland Community College was ABB Decathlon® for DCIM, Education Edition.
References
[1] GrowthPoint, “Lakeland Community College data center: Built for sustainability and flexibility”, ABB report, March 2020, pp. 1-7.
[2] United States Government, 2011, Available: https://www.usgbc.org/leed [Accessed: May 5, 2020].
[3] Ohio Department of Higher Education, “The 2018 Efficiency Advisory Committee Report,” 2018, Available: https://www.ohiohighered.org/ [Accessed: May 5, 2020].
[4] R. McFarlane, et al, “Find the right data center cooling systems for hyper-converged”, in TechTarget Blog, March 23, 2017, Available: https://techtarget.com [Accessed: May 5, 2020].
ABB’S DATA CENTER AUTOMATION FOR ERICSSON: AN ALL-IN-ONE AUTOMATION SYSTEM FOR EFFICIENT, SUSTAINABLE RELIABILITY
ABB Ability™ Data Center Automation is being used by Ericsson, one of the world’s largest telecommunications network equipment suppliers, for its Global Information and Communication Technology (ICT) Center in Rosersberg, Sweden →04a.
The Global ICT Center is of critical importance to Ericsson as company engineers around the world use the facility remotely to test products and services, before releasing them to clients. The reliability of the site and its ability to operate efficiently without interruption are crucial to Ericsson’s success.
The vast data center spans over 20,000 m2 but all three of the site’s control subsystems – the BMS, smart power management system (PMS) and energy management system (EMS) – can be monitored and controlled from one room, thanks to ABB Ability Data Center Automation →04b. As a result, Ericsson has been able to reduce energy usage by 40 percent while reducing operational and capital spending.
In this project, Ericsson leveraged ABB’s expertise in powertrain technologies and automation, including magnetic flow meters (for cooling-water flow), substation transformers and medium-voltage switchgear – all controlled and monitored by the automation system.
Energy management is key
ABB’s technology enabled Ericsson to automate and control operations not only across hardware and software systems but also across power, cooling and energy management systems. This involves integrating data from equipment supplied by at least six different manufacturers.
The BMS manages ventilation and cooling. Heat removed from the Global ICT Center is recycled to provide heat and hot water to some 20,000 local residences. In return, Ericsson obtains cold water from the city to help cool the center’s servers →04c. Efficient management of these thermal transfers is possible through ABB’s control system, enabling a more sustainable and profitable operation for both Ericsson and the region.
The PMS is designed to support the 15 MW IT load and the BMS load, which includes monitoring of: the uninterruptible power supply (UPS) battery package, auxiliary power supply, generator-set fuel system and automatic transfer schemes, as well as advanced functionality to respond to failure situations. When the backup power is activated, load-shedding functions are triggered so that UPS power is available for critical loads →04d. All loads can be controlled from one screen and non-critical loads can be configured at a lower demand status. These smart power solutions were designed into the software, enabling Ericsson to optimize their infrastructure, thus reducing the quantity and size of their UPS and gensets.
Finally, the EMS collects power and energy measurements from all the meters within the facility. This gives Ericsson insights into their energy consumption and where they can improve efficiency. As data centers grow in size and number, owners and customers have a major incentive to manage their energy use wisely. It is estimated that energy accounts for up to 40 percent of the total cost of ownership of a data center. If centers do not become more efficient and innovative, their growth could be constrained by overloaded national power grids.
