Presented at the Data Center Efficiency Summit, San Jose, CA, October 14, 2010

1. Project GreenLight: DC Modular Datacenter for Improved Energy Efficiency Dr. Greg Hidley California Institute for Telecommunications and Information Technology, UCSD http://greenlight.calit2.net

2. The NSF-Funded GreenLight ProjectGiving Users Greener Compute and Storage Options PI is Dr. Tom DeFanti $2.6M over 3 Years to construct GreenLight Instrument Start with Sun Modular Data Center Sun Has Shown up to 40% Reduction in Energy Measures Temperature at 5 Levels in 8 Racks Measures power Utilization in Each of the 8 Racks Chilled Water Cooling input and output temperatures Add additional monitoring at every receptacle Add web and VR interfaces to access measurement data Populate with a variety of computing clusters and architectures Traditional compute and storage servers GP GPU arrays and specialized FPGA based coprocessors DC powered servers Turn over to investigators in various disciplines Measure, Monitor and Collect Energy Usage data

3. The GreenLight Project: Instrumenting the Energy Cost of Computational Science Focus on 5 Communities with At-Scale Computing Needs: Metagenomics Ocean Observing Microscopy Bioinformatics Digital Media Measure, Monitor, & Web Publish Real-Time Sensor Outputs Via Service-oriented Architectures Allow Researchers Anywhere To Study Computing Energy Cost Enable Scientists To Explore Tactics For Maximizing Work/Watt Develop Middleware that Automates Optimal Choice of Compute/RAM Power Strategies for Desired Greenness

4. UCSD is Installing Zero Carbon EmissionSolar and Fuel Cell DC Electricity Generators UCSD 2.8 Megawatt Fuel Cell Power Plant Uses Methane Available Late 2011 San Diego’s Point Loma Wastewater Treatment Plant Produces Waste Methane 2 Megawatt of Solar Power Cells Installed

5. Zero Carbon GreenLight Experiment:Direct DC-Powered Modular Data Center Concept—Avoid DC to AC to DC Conversion Losses Computers Use DC Power Internally Solar and Fuel Cells Produce DC Both Plug into the AC Power Grid Can We Use DC Directly (With or Without the AC Grid)? DC Generation from Renewable Can Be Intermittent Depends on Source Solar, Wind, Fuel Cell, Hydro Can Use Sensors to Shut Down or Sleep Computers Can Use Virtualization to Halt/Shift Jobs Experiment Planning Just Starting Collaboration with Oracle, LBNL, EPRI, Direct Power, Intel, Delta, Emerson NSF GreenLight Year 2 and Year 3 Funds CEC and PIER support DC Fuel Cell 2800kWatts Sun Box <200kWatt

6. Two Rack DC Experiment now Operational Proof of concept experiment within context of Project GreenLight Uses Emerson rectifier as source of DC power => 400VDC Uses Emerson DC PDUs Drives 400VDC custom server power supplies (from Delta and Emerson) Drives 13 servers 6 SunFire X4270 (Nehalem) servers 6 Intel 2600 (Nehalem) servers 1 SunFire x4540 storage servers Transformer Rectifier PDU Servers 480VAC 400VDC 400VDC

7. Potential Energy Efficiency Improvements using DC Power Anticipate reductions in all levels of energy chain from FC to servers using 400VDC Fuel Cell Inverter (DC to AC) DC-DC conversion losses (adjust to grid voltage) 480VAC distribution line losses 480VAC to 208VAC or 120VAC step down losses UPS (AC to DC to charge batteries and back to AC) conversion losses PDU’s (AC voltage stepped down to server requirements) Inside Power Supplies (rectification and transformation losses) Anticipate reduction in cooling loads given heat dissipation of conversions Anticipate reduction of power utilization by 25% or better (power and cooling) Anticipate cooling efficiencies of 33% or better

8. END

9. Capabilities sneak peak… GLIMPSE Decision Support System D E M O 2010.08.20

10. Situational Awareness Dashboard interface “Tap” for details Power utilization Enterprise reach Multiple perspectives

11. Datacenter vitals Input/Output temperature sampling Live/average Fan speeds Live/Average data Live Temperature Heat Exchangers Environmentals

12. Domain specific views Control elements Real-time heatmap Realistic models

13. Airflow dynamics Live fan speeds Airflow dynamics 13

14. Heat distribution Combined heat + fans Realistic correlation

15. Heat Trends Heat exchangers Hotspot identification Trends over past 24h

16. Past changes in airflow Fan slices rpm Heat distribution changes Potential for failures Trends over past 24h

17. Power spikes IT assets Computation zone Unused asset Average load Peak computation 1 minute resolution

18. Zoom-in Analysis 2010.08.20 History over several days. Zoom on desired time range. Hint on each sample point. Automatic average area. Multiple sensors per asset with up to 1 min samplingresolution. Calit2/UCSD [http://greenlight.calit2.net]

19. Summary GLIMPSE – decision support system just a sliver of entire GreenLight Enterprise capabilities critical to understand the massive amounts of data provide real-time situational awareness monitor & control datacenter assets GreenLightEnterprise Solution: federates datacenters improves performance/watt by selecting most appropriate architectures improves energy utilization by scheduling computation based on heat, energy usage, and load reduces cooling needs and improves equipment lifetime