Gary Bernstein, Director of Network and Communications Services (NCS) at McGill University, presented these slides as part of the Cybera Summit 2010 session "Ultra-Efficient Data Centres: Design and Applications". For more information, visit http://www.cybera.ca/ultra-efficient-data-centres-design-and-applications
Ultra-Efficient HPC Data Centre - Gary Bernstein, McGill University
1. Ultra Efficient HPC Data
Center
Natural Low Energy Cooling Conceptual Design
10/8/10 1
2.
3. Project Funding
Canada's Advanced Research and Innovation
Network (CANARIE)
Canada California Strategic Innovation Partnership
(CCSIP)
• ISTP Canada
• University of California
• McGill University
4. Site Selection
Three candidate locations in Quebec
• McDonald Campus of McGill University in St. Anne de Bellevue
• Campus of the Institut de recherche d’Hydro-Quebec (IREQ) in
Varennes
• IREQ campus in Shawinigan
McDonald Campus of McGill University selected as
site for project
All enjoy
Cold climate
Renewable energy resource (hydroelectric)
Inexpensive electricity
5. The System Approach: An Overview
Goals: Most Efficient Class One Data Center
Climate Evaluation
Define Loads and How to Best Serve Them
• Water cooled equipment
• Medium temperature chilled water (65F, 75F)
Optimize Heat Rejection for Climate and Loads
• Evaporative free cooling – Primary cooling
• Seasonal ice storage – Top up cooling
Backup Approach
Results
6. Goals
Provide ASHRAE TC9.9 Class 1 Datacenter
No compressor based cooling
Lower construction cost
Lower operating cost
Best efficiency
Environmentally friendly
Construction materials
Recycle heat, water
7. Proposed Annual Electrical Costs
Comparison
$10,000,000
$9,000,000
$8,000,000
$7,000,000
$5 Million/yr Annual
$6,000,000 Savings Target
$5,000,000
$4,000,000
$3,000,000
$2,000,000
$1,000,000
$0
San Diego (1.35 PUE) Montreal (1.06 PUE)
at $0.09/kWh at $0.05/kWh
8. Aerial Perspective – “Farm” at
McDonald Campus
Cooling Towers and
Mechanical Infrastructure 20,000 SF Phase 1
8 MW IT Load
VA
Hospital
Fuel Tanks
Cooling Ice Pond
Stormwater Detension
Office, Shipping/
Receiving Electrical
Infrastructure
9. Climate: Free Cooling Analysis with 65F CHWS
0.030
Data Source: Government of Canada - National Climate Data & Information Archive
0.028
Data Set: WMO #71627, Montreal/Pierre Elliott Trudeau Airport, Typical Year
0.026
Elevation: 118 feet
Humidity Ratio (lbs H2O per lbs dry air)
0.024
Air Pressure: 14.633224 psia
0.022
0.020 Auxillary Cooling
80 hrs/yr
0.018
0.016 Partial Free Cooling
1234 hrs/yr
0.014
0.012
0.010
0.008
Full Free Cooling
0.006 7446 hrs/yr
0.004
0.002
0.000
-30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Dry Bulb Temperature (F)
10. Climate: Free Cooling Analysis With 75F CHWS
0.030
Data Source: Government of Canada - National Climate Data & Information Archive
0.028
Data Set: Montreal/Pierre Elliott Trudeau Intl Airport, Typical Year
0.026 Elevation: 118 feet Auxillary Cooling
Air Pressure: 14.633224 psia 0 hrs/yr
Humidity Ratio (lbs H2O per lbs dry air)
0.024
0.022
0.020 Partial Free Cooling
0.018 114 hrs/yr
0.016
0.014
0.012
0.010
0.008
Full Free Cooling
0.006 8646 hrs/yr
0.004
0.002
0.000
-30 -25 -20 -15 -10 -5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
Dry Bulb Temperature (F)
11. Load Strategies
Climate analysis shows higher temperature chilled
water offers many more hours of free cooling
Highly concentrated heat loads
• A single high density rack
can put off as much
waste heat as a
VW Beetle (40kW)
• Air exiting racks
typically exceeds
90F
12. Load Heat Collection Strategies
Higher temperature Chilled Water Supply (CHWS)
offers many more hours of free cooling: Design to
use 75F and 65F CHWS
• Direct water based cooling most efficient
• Hot aisle / cold aisle for minority of load
14. Primary Cooling Strategies: Medium
Temp. Cooling Water and Free Cooling
Design to cool with 65F/18C and 75F/24C water
• 90% of 65F load served with cooling tower provided free cooling;
99.3% of 75F load
• 590,000 ton-hrs (2,100 MWh) Top up Cooling Required
15. Supplemental Cooling: Seasonal Ice
Storage Slush Pond System
Fill in winter with plowed snow collection
Melt water cools data center
16.
17. Slush Pond
Paved collection basin, 75,000 ft3 (2,100 m3)
Drive-in slope on one side for plow loading
Lightweight, waterproof insulating cover or roof to protect
from warm rains
Extensive drain system to collect meltwater
Berms for sides, or dig into ground
21. Slush Pond System – Pumping and
Filtration
Mature waste water handling technology
Remove gravel, wood, grit from melt water
Remove oils and road chemicals prior to release as required
Filter
Select heat exchangers for highly corrosive fluid
Maintain complete separation between pond water and
building loop water
Integrate settling tank to also serve as emergency
storage
22. Key approaches
Keep storage pond simple
Leverage local snow removal program if possible
Collect snow dumpage fees?
Provide appropriate maintenance
Provide for pile grooming, drain clearing, filter cleaning, etc
Use in lieu of chillers to save cost
Consider emergency chiller rental for backup
Design properly
Simple concept but careful design required
23. Office Approaches
Much lower load
Design for comfort and optimal use of medium
temperature water
24. Backup – Do Not Invest in Chillers 'Just
in Case'!
Pay for it only
when (if) you
ever need it
Design for
portable
air-cooled
chillers
to connect in
an emergency
25. Results
McGill-USCD HPC Data Center PUE Itemization
Fans; 1.5%
CRAH Fans; 0.0%
Humidifier; 0.0%
CHW Plant; 2.1%
Transformer Loss;
0.5%
UPS Loss; 0.6%
Racks; 94.0% PDU Loss; 1.0%
Data Center
Lights; 0.2%
Power Usage Effectiveness (PUE) = Total Energy / Rack Energy = 1.06
26. Results
Supply Temperatures Annual Energy Use Mechanical Cooling Needed Water Usage
Hours of Free
Cooling / year PUE Additional Load at
Air Water Cost Evaporation +
Cooled Cooled
Energy
( $0.058/kWh) Hours Extreme Weather Carry Over
per Year (wetbulb = 68.7°F)
°C °F °C °F hrs/yr % of yr MWh/yr $ tons gallons
23.9 75.0 23.9 75.0 8,532 97% 1.06 74,567 $4,325,000 228 0 30,100,000