SlideShare uma empresa Scribd logo
1 de 24
Baixar para ler offline
Where Does the Power Go
 & What to do About it?

             James Hamilton
  Conference on Innovative Data Systems Research
                     2009/1/7
               Amazon Web Services
              e: James@amazon.com
            w: mvdirona.com/jrh/work
          b: perspectives.mvdirona.com
Agenda

• Services Inevitable
• Where does the power go?
      – Power distribution systems &
        optimizations
      – Mechanical systems & optimizations
      – Cooperative, Expendable, Micro-Slice
        Servers
      – Critical Load Optimizations
• Summary

2009/1/7                 http://perspectives.mvdirona.com   2
Services Economies of Scale
• Substantial economies of scale possible
• Compare a very large service with a small/mid-sized: (~1000 servers):

                                                 Large Service [$13/Mbps]: $0.04/GB
                  Networking                     Medium [$95/Mbps]: $0.30/GB (7.1x)



                                        Large Service: $4.6/GB/year (2x in 2 DC)
                  BLOB Storage          Medium: $26.00/GB/year* (5.7x)



                                                 Large Service: Over 1.000 servers/admin
                  Administration                 Enterprise: ~140 servers/admin (7.1x)

                                                                                      2006 Chart data
• High cost of entry
     – Physical plant expensive: 15MW ~$200M (infrastructure only)
• Summary: significant economies of scale but at high cost of entry
     – Small number of large players likely outcome
2009/1/7                       http://perspectives.mvdirona.com                                    3
Services Different from Enterprises
• Enterprise Approach:
      – Largest cost is people -- scales roughly with servers (~100:1 common)
      – Enterprise interests center around consolidation & utilization
           • Consolidate workload onto fewer, larger systems
           • Large SANs for storage & large routers for networking
• Internet-Scale Services Approach:
      – Largest costs is server & storage H/W
           • Typically followed by cooling, power distribution, power
           • Networking varies from very low to dominant depending upon service
           • People costs under 10% & often under 5% (>1000+:1 server:admin)
      – Services interests center around work-done-per-$ (or joule)
• Observations:
     • People costs shift from top to nearly irrelevant.
     • Expect high-scale service techniques to spread to enterprise
     • Focus instead on work done/$ & work done/joule
2009/1/7                        http://perspectives.mvdirona.com                  4
Agenda

• Services Inevitable
• Where does the power go?
      – Power distribution systems &
        optimizations
      – Mechanical systems & optimizations
      – Cooperative, Expendable, Micro-Slice
        Servers
      – Critical Load Optimizations
• Summary

2009/1/7                 http://perspectives.mvdirona.com   5
PUE & DCiE
• Measure of data center infrastructure efficiency
• Power Usage Effectiveness
      – PUE = (Total Facility Power)/(IT Equipment Power)
• Data Center Infrastructure Efficiency
      – DCiE = (IT Equipment Power)/(Total Facility Power) * 100%




                                                                       Advanced Data Centers

http://www.thegreengrid.org/gg_content/TGG_Data_Center_Power_Efficiency_Metrics_PUE_and_DCiE.pdf
2009/1/7                            http://perspectives.mvdirona.com                               6
Power & Related Costs Dominate
•   Assumptions:
      –    Facility: ~$200M for 15MW facility (15-year amort.)
      –    Servers: ~$2k/each, roughly 50,000 (3-year amort.)
      –    Average server power draw at 30% utilization: 80%
      –    Commercial Power: ~$0.07/kWhr
                        Monthly Costs
           $284,686
                                                  Servers
$1,042,440

                                                  Power & Cooling
                                                  Infrastructure
                         $2,997,090               Power
           $1,296,902

                                                  Other Infrastructure

             3yr server & 15 yr infrastructure amortization

•   Observations:
      •      $2.3M/month from charges functionally related to power
      •      Power related costs trending flat or up while server costs trending down
                        Details at: http://perspectives.mvdirona.com/2008/11/28/CostOfPowerInLargeScaleDataCenters.aspx
2009/1/7                                  http://perspectives.mvdirona.com                                      7
Fully Burdened Cost of Power
 • Infrastructure cost/watt:
       – 15 year amortization & 5% money cost
       – =PMT(5%,15,200000000,0)/(15,000,000)
         => $1.28/W/yr
 • Cost per watt using $0.07 Kw*hr:
       – =-0.07*1.7/1000*0.8*24*365=>
         $0.83/W/yr (@80% power utilization)


 • Fully burdened cost of power:
       – $1.28 + $0.83 => $2.11

2009/1/7                 http://perspectives.mvdirona.com   8
Where Does the Power Go?
• Assuming a pretty good data center with PUE ~1.7
      – Each watt to server loses ~0.7W to power distribution
        losses & cooling
• Power losses are easier to track than cooling:
      – Power transmission & switching losses: 8%
           • Detailed power distribution losses on next slide
      – Cooling losses remainder:100-(59+8) => 33%
• Data center power consumption:
      – IT load (servers): 1/1.7=> 59%
      – Distribution Losses: 8%
      – Mechanical load(cooling): 33%
2009/1/7                       http://perspectives.mvdirona.com   9
Agenda

• Services Inevitable
• Where does the power go?
      – Power distribution systems &
        optimizations
      – Mechanical systems & optimizations
      – Cooperative, Expendable, Micro-Slice
        Servers
      – Critical Load Optimizations
• Summary

2009/1/7                 http://perspectives.mvdirona.com   10
Power Distribution
                                                                               IT Load (servers, storage, Net, …)
       High Voltage
                                 8% distribution loss
    Utility Distribution
                                   .997^3*.94*.99 = 92.2%

                                2.5MW Generator (180 gal/hr)
           115kv




                                                                                                     208V
                                                                                                            ~1% loss in switch
                                       13.2kv




                                                                                                            gear & conductors
                              UPS:                                                               Transformers
                                                                    Transformers
                              Rotary or Battery
  Transformers


                                                           13.2kv
                     13.2kv                                                              480V




                                  6% loss
     0.3% loss                                                                                      0.3% loss
                                                                       0.3% loss
                       94% efficient, ~97% available
   99.7% efficient                                                                                99.7% efficient
                                                                     99.7% efficient
2009/1/7                                        http://perspectives.mvdirona.com                                      11
Power Redundancy to Geo-Level
• Over 20% of entire DC costs is in power redundancy
      – Batteries supply over 10 min at some facilities (~2 min sufficient)
      – N+2 generation (2.5MW) at over $2M each
• Instead use more, smaller, cheaper data centers
• Non-bypass, battery-based UPS in the 94% efficiency range
      – ~900kW wasted in 15MW facility (4,500 200W servers)
      – 97% available (still 450kW loss in 15MW facility)




2009/1/7                      http://perspectives.mvdirona.com                12
Power Distribution Optimization
• Two additional conversions in server:
     – Power Supply: often <80% at typical load
     – Voltage Regulation Module: ~80% common
     – ~95% efficient available & affordable
• Rules to minimize power distribution losses:
     1.        Avoid conversions (Less transformer steps & efficient or no UPS)
     2.        Increase efficiency of conversions
     3.        High voltage as close to load as possible
     4.        Size voltage regulators (VRM/VRDs) to load & use efficient parts
     5.        DC distribution potentially a small win (regulatory issues
•     Two interesting approaches:
     –         480VAC (or higher) to rack & 48VDC (or 12VDC) within
     –         480VAC to PDU and 277VAC to load
           •      1 leg of 480VAC 3-phase distribution
2009/1/7                           http://perspectives.mvdirona.com               13
Agenda

• Services Inevitable
• Where does the power go?
      – Power distribution systems &
        optimizations
      – Mechanical systems & optimizations
      – Cooperative, Expendable, Micro-Slice
        Servers
      – Critical Load Optimizations
• Summary

2009/1/7                 http://perspectives.mvdirona.com   14
Conventional Mechanical Design
Blow down & Evaporative Loss for
  15MW facility: ~360,000 gal/day




                                                                   Heat                          Primary
                                                                Exchanger                         Pump
                                    Cooling               (Water-Side Economizer)
                                     Tower



                                    CWS                      A/C                                              A/C
                                    Pump                  Condenser                                        Evaporator
                                                                                       A/C
                                                                                    Compressor
                                                                                                                        Secondary
                                                                                                                          Pump
                                                                        Diluted Hot/Cold Mix
       Server fans 6 to 9W each
                                             leakage
 Air-side
                                                          Hot




                                                                                                      Computer
                                                   fans
Economiz-                                                             Overall
                                                                                                      Room Air
  ation
                                                                  Mechanical Losses
                                            cold                                                       Handler
                                                                       ~33%

                                                                                                    Air Impeller
                                     Cold




                          2009/1/7                                     http://perspectives.mvdirona.com                             15
Mechanical Optimization
• Simple rules to minimize cooling costs:
      1.   Raise data center temperatures
      2.   Tight control of airflow with short paths
      3.   Cooling towers rather than A/C
      4.   Air side economization (open the window)
      5.   Low-grade, waste heat energy reclamation
•      Best current designs still use air but bring water near servers
      –    Lower heat densities could be 100% air cooled
•      Common mechanical designs: 33% lost in cooling
•      PUE under 1.0 within reach with some innovation
      –    Waste heat reclamation in excess of power distribution & cooling
           overhead (~30% effective reclamation sufficient for <1.0 operation)

2009/1/7                      http://perspectives.mvdirona.com               16
Agenda

• Services Inevitable
• Where does the power go?
      – Power distribution systems &
        optimizations
      – Mechanical systems & optimizations
      – Cooperative, Expendable, Micro-Slice
        Servers
      – Critical Load Optimizations
• Summary

2009/1/7                 http://perspectives.mvdirona.com   17
CEMS Speeds & Feeds
    •      CEMS: Cooperative Expendable Micro-Slice Servers
            – Correct system balance problem with less-capable CPU
                 • Too many cores, running too fast, for memory, bus, disk, …
    •      Joint project with Rackable Systems (http://www.rackable.com/)
                                                  CEMS V3                  CEMS V2                  CEMS V1
                        System-X                (Athlon 4850e)            Athlon 3400e)           (Athlon 2000+)
        CPU load%                    56%                      57%                         57%                   61%
           RPS                       95.9                      75.3                        54.3                  17.0
           Price                   $2,371                     $500                        $685                 $500
          Power                       295                        60                         39                     33
         RPS/Price                   0.04                      0.15                     0.08                     0.03
        RPS/Joule                    0.33                      1.25                     1.39                     0.52
         RPS/Rack                  1918.4                   18062.4                  13024.8                   4080.0


                                                                                •CEMS V2 Comparison:
                                                                                          •Work Done/$: +375%
                                                                                          •Work Done/Joule +379%
                                                                                          •Work Done/Rack: +942%

                                                                                Update: New H/W SKU likely
                                                                                will improve numbers by
                                                                                factor of 2. CEMS still a win.
2009/1/7                                    http://perspectives.mvdirona.com                                            18
Power Yield Management
• “Oversell” power, the most valuable
                                   Max utility power
  resource:                                                                                    10%
                                   Max de-rated power
      – e.g. sell more seats than airplane holds
• Overdraw penalty high:
      – Pop breaker (outage)




                                                                                                                                Dynamic yield mgmt
                                                                     Peak




                                                                                                            Static yield mgmt
      – Overdraw utility (fine)




                                                                            Max server label




                                                                                                                                     with H/W caps
                                                                 Average
• Considerable optimization possible If




                                                                                                Max clamp
  workload variation is understood
      – Workload diversity & history helpful
      – Graceful Degradation Mode to shed
        workload

                 Power Provisioning in a Warehouse-Sized Computer, Xiabo Fan, Wolf Weber, Luize Borroso
2009/1/7                          http://perspectives.mvdirona.com                                                                 19
Critical Load Optimization
• Power proportionality is great but full load or shut off is even better
      – Idle server consumes ~60% power of full load
      – Industry secret: “great” data center utilization around ~30%
      – All solutions require changing where a workload is executed
• What limits dynamic workload migration?
      – Networking constraints: VIPs can’t span L2 nets, ACLs static, manual config, etc.
      – Data Locality: Hard to efficiently move several TB
      – Workload management: Scheduling work over available resources
• Critical load optimizations, in order:
      – Use the servers: any workload with marginal value over power
      – Shut them off if you can’t use them
      – Those servers on should be fully loaded but not all resources fully consumed
        so use power management (e.g. dynamic voltage & frequency scaling)
• Efficient S/W algorithms as important as H/W

2009/1/7                           http://perspectives.mvdirona.com                         20
Resource Consumption Shaping
• Essentially yield mgmt applied to full DC
                                                                Egress charged at 95th percentile
• Network charged at 95th percentile:                                               3PM
      – Push peaks to troughs                                        4AM            PST
                                                                     PST                         The
      – Fill troughs for “free”                                                                 Pacific
                                                                                               Ocean is
           • e.g. Amazon S3 replication                                                          big.

      – Dynamic resource allocation
           • Virtual machine helpful but not needed
   – Charged for symmetrically so ingress
      effectively free
                                             David Treadwell & James Hamilton / Treadwell Graph
• Power also charged at 95  th percentile

   • Server idle to full-load range: ~65% (e.g. 158W to 230W )
   • S3 (suspend) or S5 (off) when server not needed
• Disks come with both IOPS capability & capacity
   • Mix hot & cold data to “soak up” both
• Encourage priority (urgency) differentiation in charge-back model
2009/1/7                          http://perspectives.mvdirona.com                                        21
Agenda

• Services Inevitable
• Where does the power go?
      – Power distribution systems &
        optimizations
      – Mechanical systems & optimizations
      – Cooperative, Expendable, Micro-Slice
        Servers
      – Critical Load Optimizations
• Summary

2009/1/7                 http://perspectives.mvdirona.com   22
Summary
• Current “good” data centers have considerable room
  for improvement
• Where do the power go?
     – 58% Servers and other IT equipment
     – 33% mechanical systems
     – 8% power distribution
• Lowest hanging fruit in servers & mechanical systems
• Server system optimizations
     – Utilization levels and general scheduling optimizations
     – Servers optimized for work done per joule & watt rather than raw
       performance
• CEMS takes only a small step forward but achieves better than
  3x improvement in work done/$ and work done/joule
2009/1/7                     http://perspectives.mvdirona.com             23
More Information
• This Slide Deck:
      –    I will post these slides to http://perspectives.mvdirona.com later this week

• Designing & Deploying Internet-Scale Services
      –    http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf

• Architecture for Modular Data Centers
     •     http://mvdirona.com/jrh/talksAndPapers/JamesRH_CIDR.doc

• Increasing DC Efficiency by 4x
     •     http://mvdirona.com/jrh/talksAndPapers/JamesRH_PowerSavings20080604.ppt

• Perspectives Blog
      –    http://perspectives.mvdirona.com

• Email
      –    James@amazon.com



2009/1/7                                 http://perspectives.mvdirona.com                 24

Mais conteúdo relacionado

Semelhante a CIDR 2009: James Hamilton Keynote

Wattminder exec brief
Wattminder exec briefWattminder exec brief
Wattminder exec briefSam Beal
 
Trilogy - Henk Groenendijk
Trilogy - Henk GroenendijkTrilogy - Henk Groenendijk
Trilogy - Henk GroenendijkHPDutchWorld
 
Clean power plans - the role of the smart grid
Clean power plans  - the role of the smart gridClean power plans  - the role of the smart grid
Clean power plans - the role of the smart gridPaul Alvarez
 
H&amp;M Power Conversion Segmented Inverter 2010 R1
H&amp;M Power Conversion Segmented Inverter 2010 R1H&amp;M Power Conversion Segmented Inverter 2010 R1
H&amp;M Power Conversion Segmented Inverter 2010 R1Sammy Germany
 
Cutting costs with pc power management approaches, pitfalls and best practices
Cutting costs with pc power management approaches, pitfalls and best practicesCutting costs with pc power management approaches, pitfalls and best practices
Cutting costs with pc power management approaches, pitfalls and best practicesInterop
 
Product: Power Factor & Harmonics: StacoSine: Technical
Product: Power Factor & Harmonics: StacoSine: TechnicalProduct: Power Factor & Harmonics: StacoSine: Technical
Product: Power Factor & Harmonics: StacoSine: TechnicalStaco Energy
 
16_Economic_assessment_of_PV_and_wind_for_energy_planning_Cairo_Egypt.pdf
16_Economic_assessment_of_PV_and_wind_for_energy_planning_Cairo_Egypt.pdf16_Economic_assessment_of_PV_and_wind_for_energy_planning_Cairo_Egypt.pdf
16_Economic_assessment_of_PV_and_wind_for_energy_planning_Cairo_Egypt.pdfMohamed Salah
 
Mme presentation customs
Mme presentation customsMme presentation customs
Mme presentation customsmme
 
IRJET- Modeling, Simulation and Implementation of D-STATCOM for Improveme...
IRJET-  	  Modeling, Simulation and Implementation of D-STATCOM for Improveme...IRJET-  	  Modeling, Simulation and Implementation of D-STATCOM for Improveme...
IRJET- Modeling, Simulation and Implementation of D-STATCOM for Improveme...IRJET Journal
 
IRJET- Automatic Power Factor Correction
IRJET- Automatic Power Factor CorrectionIRJET- Automatic Power Factor Correction
IRJET- Automatic Power Factor CorrectionIRJET Journal
 
2009 Itc Green It
2009 Itc Green It2009 Itc Green It
2009 Itc Green ItJayMNEA
 
IRJET- Automatic Load Sharing of Transformer
IRJET- Automatic Load Sharing of TransformerIRJET- Automatic Load Sharing of Transformer
IRJET- Automatic Load Sharing of TransformerIRJET Journal
 
High level explanation of smart grids and battery energy storage systems
High level explanation of smart grids and battery energy storage systemsHigh level explanation of smart grids and battery energy storage systems
High level explanation of smart grids and battery energy storage systemsMircea Gingu
 
Retail P P T Kvar Presentation 2
Retail  P P T  Kvar  Presentation 2Retail  P P T  Kvar  Presentation 2
Retail P P T Kvar Presentation 2Robert B. Luquette
 

Semelhante a CIDR 2009: James Hamilton Keynote (20)

Wattminder exec brief
Wattminder exec briefWattminder exec brief
Wattminder exec brief
 
Trilogy - Henk Groenendijk
Trilogy - Henk GroenendijkTrilogy - Henk Groenendijk
Trilogy - Henk Groenendijk
 
Flyer 4 page
Flyer 4 pageFlyer 4 page
Flyer 4 page
 
Clean power plans - the role of the smart grid
Clean power plans  - the role of the smart gridClean power plans  - the role of the smart grid
Clean power plans - the role of the smart grid
 
H&amp;M Power Conversion Segmented Inverter 2010 R1
H&amp;M Power Conversion Segmented Inverter 2010 R1H&amp;M Power Conversion Segmented Inverter 2010 R1
H&amp;M Power Conversion Segmented Inverter 2010 R1
 
ISLANDING DETECTION TECHNIQUE FOR DISTRIBUTED GENERATION SYSTEM
ISLANDING DETECTION TECHNIQUE FOR DISTRIBUTED GENERATION SYSTEMISLANDING DETECTION TECHNIQUE FOR DISTRIBUTED GENERATION SYSTEM
ISLANDING DETECTION TECHNIQUE FOR DISTRIBUTED GENERATION SYSTEM
 
IT Energy Waste - Green IT Expo 2009
IT Energy Waste - Green IT Expo 2009IT Energy Waste - Green IT Expo 2009
IT Energy Waste - Green IT Expo 2009
 
Cutting costs with pc power management approaches, pitfalls and best practices
Cutting costs with pc power management approaches, pitfalls and best practicesCutting costs with pc power management approaches, pitfalls and best practices
Cutting costs with pc power management approaches, pitfalls and best practices
 
Product: Power Factor & Harmonics: StacoSine: Technical
Product: Power Factor & Harmonics: StacoSine: TechnicalProduct: Power Factor & Harmonics: StacoSine: Technical
Product: Power Factor & Harmonics: StacoSine: Technical
 
Faronics Power Save
Faronics Power SaveFaronics Power Save
Faronics Power Save
 
16_Economic_assessment_of_PV_and_wind_for_energy_planning_Cairo_Egypt.pdf
16_Economic_assessment_of_PV_and_wind_for_energy_planning_Cairo_Egypt.pdf16_Economic_assessment_of_PV_and_wind_for_energy_planning_Cairo_Egypt.pdf
16_Economic_assessment_of_PV_and_wind_for_energy_planning_Cairo_Egypt.pdf
 
Mme presentation customs
Mme presentation customsMme presentation customs
Mme presentation customs
 
IRJET- Modeling, Simulation and Implementation of D-STATCOM for Improveme...
IRJET-  	  Modeling, Simulation and Implementation of D-STATCOM for Improveme...IRJET-  	  Modeling, Simulation and Implementation of D-STATCOM for Improveme...
IRJET- Modeling, Simulation and Implementation of D-STATCOM for Improveme...
 
IRJET- Automatic Power Factor Correction
IRJET- Automatic Power Factor CorrectionIRJET- Automatic Power Factor Correction
IRJET- Automatic Power Factor Correction
 
2009 Itc Green It
2009 Itc Green It2009 Itc Green It
2009 Itc Green It
 
2014 PV Distribution System Modeling Workshop: High Penetration PV Control Co...
2014 PV Distribution System Modeling Workshop: High Penetration PV Control Co...2014 PV Distribution System Modeling Workshop: High Penetration PV Control Co...
2014 PV Distribution System Modeling Workshop: High Penetration PV Control Co...
 
IRJET- Automatic Load Sharing of Transformer
IRJET- Automatic Load Sharing of TransformerIRJET- Automatic Load Sharing of Transformer
IRJET- Automatic Load Sharing of Transformer
 
High level explanation of smart grids and battery energy storage systems
High level explanation of smart grids and battery energy storage systemsHigh level explanation of smart grids and battery energy storage systems
High level explanation of smart grids and battery energy storage systems
 
Retail P P T Kvar Presentation 2
Retail  P P T  Kvar  Presentation 2Retail  P P T  Kvar  Presentation 2
Retail P P T Kvar Presentation 2
 
Maximize Minnesota Power Supply And Demand Presentation February 2010
Maximize Minnesota Power Supply And Demand Presentation February 2010Maximize Minnesota Power Supply And Demand Presentation February 2010
Maximize Minnesota Power Supply And Demand Presentation February 2010
 

Mais de infoblog

CIDR 2009: Jeff Heer Keynote
CIDR 2009: Jeff Heer KeynoteCIDR 2009: Jeff Heer Keynote
CIDR 2009: Jeff Heer Keynoteinfoblog
 
Claremont Report on Database Research: Research Directions (Le Gruenwald)
Claremont Report on Database Research: Research Directions (Le Gruenwald)Claremont Report on Database Research: Research Directions (Le Gruenwald)
Claremont Report on Database Research: Research Directions (Le Gruenwald)infoblog
 
Claremont Report on Database Research: Research Directions (Eric A. Brewer)
Claremont Report on Database Research: Research Directions (Eric A. Brewer)Claremont Report on Database Research: Research Directions (Eric A. Brewer)
Claremont Report on Database Research: Research Directions (Eric A. Brewer)infoblog
 
Claremont Report on Database Research: Research Directions (Rakesh Agrawal)
Claremont Report on Database Research: Research Directions (Rakesh Agrawal)Claremont Report on Database Research: Research Directions (Rakesh Agrawal)
Claremont Report on Database Research: Research Directions (Rakesh Agrawal)infoblog
 
Claremont Report on Database Research: Research Directions (Gerhard Weikum)
Claremont Report on Database Research: Research Directions (Gerhard Weikum)Claremont Report on Database Research: Research Directions (Gerhard Weikum)
Claremont Report on Database Research: Research Directions (Gerhard Weikum)infoblog
 
Claremont Report on Database Research: Research Directions (Beng Chin Ooi)
Claremont Report on Database Research: Research Directions (Beng Chin Ooi)Claremont Report on Database Research: Research Directions (Beng Chin Ooi)
Claremont Report on Database Research: Research Directions (Beng Chin Ooi)infoblog
 
Claremont Report on Database Research: Research Directions (Yannis E. Ioannidis)
Claremont Report on Database Research: Research Directions (Yannis E. Ioannidis)Claremont Report on Database Research: Research Directions (Yannis E. Ioannidis)
Claremont Report on Database Research: Research Directions (Yannis E. Ioannidis)infoblog
 
Claremont Report on Database Research: Research Directions (Donald Kossmann)
Claremont Report on Database Research: Research Directions (Donald Kossmann)Claremont Report on Database Research: Research Directions (Donald Kossmann)
Claremont Report on Database Research: Research Directions (Donald Kossmann)infoblog
 
Claremont Report on Database Research: Research Directions (Johannes Gehrke)
Claremont Report on Database Research: Research Directions (Johannes Gehrke)Claremont Report on Database Research: Research Directions (Johannes Gehrke)
Claremont Report on Database Research: Research Directions (Johannes Gehrke)infoblog
 
Claremont Report on Database Research: Research Directions (Alon Y. Halevy)
Claremont Report on Database Research: Research Directions (Alon Y. Halevy)Claremont Report on Database Research: Research Directions (Alon Y. Halevy)
Claremont Report on Database Research: Research Directions (Alon Y. Halevy)infoblog
 
Claremont Report on Database Research: Research Directions (Anastasia Ailamaki)
Claremont Report on Database Research: Research Directions (Anastasia Ailamaki)Claremont Report on Database Research: Research Directions (Anastasia Ailamaki)
Claremont Report on Database Research: Research Directions (Anastasia Ailamaki)infoblog
 
Database Research Principles Revealed (Small Size)
Database Research Principles Revealed (Small Size)Database Research Principles Revealed (Small Size)
Database Research Principles Revealed (Small Size)infoblog
 
Database Research Principles Revealed
Database Research Principles RevealedDatabase Research Principles Revealed
Database Research Principles Revealedinfoblog
 

Mais de infoblog (14)

CIDR 2009: Jeff Heer Keynote
CIDR 2009: Jeff Heer KeynoteCIDR 2009: Jeff Heer Keynote
CIDR 2009: Jeff Heer Keynote
 
Claremont Report on Database Research: Research Directions (Le Gruenwald)
Claremont Report on Database Research: Research Directions (Le Gruenwald)Claremont Report on Database Research: Research Directions (Le Gruenwald)
Claremont Report on Database Research: Research Directions (Le Gruenwald)
 
Claremont Report on Database Research: Research Directions (Eric A. Brewer)
Claremont Report on Database Research: Research Directions (Eric A. Brewer)Claremont Report on Database Research: Research Directions (Eric A. Brewer)
Claremont Report on Database Research: Research Directions (Eric A. Brewer)
 
Claremont Report on Database Research: Research Directions (Rakesh Agrawal)
Claremont Report on Database Research: Research Directions (Rakesh Agrawal)Claremont Report on Database Research: Research Directions (Rakesh Agrawal)
Claremont Report on Database Research: Research Directions (Rakesh Agrawal)
 
Claremont Report on Database Research: Research Directions (Gerhard Weikum)
Claremont Report on Database Research: Research Directions (Gerhard Weikum)Claremont Report on Database Research: Research Directions (Gerhard Weikum)
Claremont Report on Database Research: Research Directions (Gerhard Weikum)
 
Claremont Report on Database Research: Research Directions (Beng Chin Ooi)
Claremont Report on Database Research: Research Directions (Beng Chin Ooi)Claremont Report on Database Research: Research Directions (Beng Chin Ooi)
Claremont Report on Database Research: Research Directions (Beng Chin Ooi)
 
Claremont Report on Database Research: Research Directions (Yannis E. Ioannidis)
Claremont Report on Database Research: Research Directions (Yannis E. Ioannidis)Claremont Report on Database Research: Research Directions (Yannis E. Ioannidis)
Claremont Report on Database Research: Research Directions (Yannis E. Ioannidis)
 
Claremont Report on Database Research: Research Directions (Donald Kossmann)
Claremont Report on Database Research: Research Directions (Donald Kossmann)Claremont Report on Database Research: Research Directions (Donald Kossmann)
Claremont Report on Database Research: Research Directions (Donald Kossmann)
 
Claremont Report on Database Research: Research Directions (Johannes Gehrke)
Claremont Report on Database Research: Research Directions (Johannes Gehrke)Claremont Report on Database Research: Research Directions (Johannes Gehrke)
Claremont Report on Database Research: Research Directions (Johannes Gehrke)
 
Claremont Report on Database Research: Research Directions (Alon Y. Halevy)
Claremont Report on Database Research: Research Directions (Alon Y. Halevy)Claremont Report on Database Research: Research Directions (Alon Y. Halevy)
Claremont Report on Database Research: Research Directions (Alon Y. Halevy)
 
Claremont Report on Database Research: Research Directions (Anastasia Ailamaki)
Claremont Report on Database Research: Research Directions (Anastasia Ailamaki)Claremont Report on Database Research: Research Directions (Anastasia Ailamaki)
Claremont Report on Database Research: Research Directions (Anastasia Ailamaki)
 
Spot Sigs
Spot SigsSpot Sigs
Spot Sigs
 
Database Research Principles Revealed (Small Size)
Database Research Principles Revealed (Small Size)Database Research Principles Revealed (Small Size)
Database Research Principles Revealed (Small Size)
 
Database Research Principles Revealed
Database Research Principles RevealedDatabase Research Principles Revealed
Database Research Principles Revealed
 

Último

Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...HostedbyConfluent
 
Real-time Geospatial Aircraft Monitoring Using Apache Kafka
Real-time Geospatial Aircraft Monitoring Using Apache KafkaReal-time Geospatial Aircraft Monitoring Using Apache Kafka
Real-time Geospatial Aircraft Monitoring Using Apache KafkaHostedbyConfluent
 
Brick-by-Brick: Exploring the Elements of Apache Kafka®
Brick-by-Brick: Exploring the Elements of Apache Kafka®Brick-by-Brick: Exploring the Elements of Apache Kafka®
Brick-by-Brick: Exploring the Elements of Apache Kafka®HostedbyConfluent
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?Rustici Software
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent
 
How Do You Query a Stream? | Kafka Summit London
How Do You Query a Stream? | Kafka Summit LondonHow Do You Query a Stream? | Kafka Summit London
How Do You Query a Stream? | Kafka Summit LondonHostedbyConfluent
 
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveApache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveHostedbyConfluent
 
🎶🎵Bo-stream-ian Rhapsody: A Musical Demo of Kafka Connect and Kafka Streams 🎵🎶
🎶🎵Bo-stream-ian Rhapsody: A Musical Demo of Kafka Connect and Kafka Streams 🎵🎶🎶🎵Bo-stream-ian Rhapsody: A Musical Demo of Kafka Connect and Kafka Streams 🎵🎶
🎶🎵Bo-stream-ian Rhapsody: A Musical Demo of Kafka Connect and Kafka Streams 🎵🎶HostedbyConfluent
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent
 
Build Copilots on Streaming Data with Generative AI, Kafka Streams and Flink SQL
Build Copilots on Streaming Data with Generative AI, Kafka Streams and Flink SQLBuild Copilots on Streaming Data with Generative AI, Kafka Streams and Flink SQL
Build Copilots on Streaming Data with Generative AI, Kafka Streams and Flink SQLHostedbyConfluent
 
BODYPACK DIGITAL TECHNOLOGY STACK - 2024
BODYPACK DIGITAL TECHNOLOGY STACK - 2024BODYPACK DIGITAL TECHNOLOGY STACK - 2024
BODYPACK DIGITAL TECHNOLOGY STACK - 2024Andri H.
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent
 
Modifying Your SQL Streaming Queries on the Fly: The Impossible Trinity
Modifying Your SQL Streaming Queries on the Fly: The Impossible TrinityModifying Your SQL Streaming Queries on the Fly: The Impossible Trinity
Modifying Your SQL Streaming Queries on the Fly: The Impossible TrinityHostedbyConfluent
 
Technology Governance & Migration In The AI Era
Technology Governance & Migration In The AI EraTechnology Governance & Migration In The AI Era
Technology Governance & Migration In The AI Era2toLead Limited
 
Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactivestartupro
 
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...HostedbyConfluent
 
AsyncAPI v3: What’s New? | Kafka Summit London
AsyncAPI v3: What’s New? | Kafka Summit LondonAsyncAPI v3: What’s New? | Kafka Summit London
AsyncAPI v3: What’s New? | Kafka Summit LondonHostedbyConfluent
 
CERN IoT Kafka Pipelines | Kafka Summit London
CERN IoT Kafka Pipelines | Kafka Summit LondonCERN IoT Kafka Pipelines | Kafka Summit London
CERN IoT Kafka Pipelines | Kafka Summit LondonHostedbyConfluent
 
Tecnogravura, Cylinder Engraving for Rotogravure
Tecnogravura, Cylinder Engraving for RotogravureTecnogravura, Cylinder Engraving for Rotogravure
Tecnogravura, Cylinder Engraving for RotogravureAntonio de Llamas
 
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024BookNet Canada
 

Último (20)

Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
 
Real-time Geospatial Aircraft Monitoring Using Apache Kafka
Real-time Geospatial Aircraft Monitoring Using Apache KafkaReal-time Geospatial Aircraft Monitoring Using Apache Kafka
Real-time Geospatial Aircraft Monitoring Using Apache Kafka
 
Brick-by-Brick: Exploring the Elements of Apache Kafka®
Brick-by-Brick: Exploring the Elements of Apache Kafka®Brick-by-Brick: Exploring the Elements of Apache Kafka®
Brick-by-Brick: Exploring the Elements of Apache Kafka®
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
 
How Do You Query a Stream? | Kafka Summit London
How Do You Query a Stream? | Kafka Summit LondonHow Do You Query a Stream? | Kafka Summit London
How Do You Query a Stream? | Kafka Summit London
 
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveApache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
Apache Kafka's Common Pitfalls & Intricacies: A Customer Support Perspective
 
🎶🎵Bo-stream-ian Rhapsody: A Musical Demo of Kafka Connect and Kafka Streams 🎵🎶
🎶🎵Bo-stream-ian Rhapsody: A Musical Demo of Kafka Connect and Kafka Streams 🎵🎶🎶🎵Bo-stream-ian Rhapsody: A Musical Demo of Kafka Connect and Kafka Streams 🎵🎶
🎶🎵Bo-stream-ian Rhapsody: A Musical Demo of Kafka Connect and Kafka Streams 🎵🎶
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
 
Build Copilots on Streaming Data with Generative AI, Kafka Streams and Flink SQL
Build Copilots on Streaming Data with Generative AI, Kafka Streams and Flink SQLBuild Copilots on Streaming Data with Generative AI, Kafka Streams and Flink SQL
Build Copilots on Streaming Data with Generative AI, Kafka Streams and Flink SQL
 
BODYPACK DIGITAL TECHNOLOGY STACK - 2024
BODYPACK DIGITAL TECHNOLOGY STACK - 2024BODYPACK DIGITAL TECHNOLOGY STACK - 2024
BODYPACK DIGITAL TECHNOLOGY STACK - 2024
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
 
Modifying Your SQL Streaming Queries on the Fly: The Impossible Trinity
Modifying Your SQL Streaming Queries on the Fly: The Impossible TrinityModifying Your SQL Streaming Queries on the Fly: The Impossible Trinity
Modifying Your SQL Streaming Queries on the Fly: The Impossible Trinity
 
Technology Governance & Migration In The AI Era
Technology Governance & Migration In The AI EraTechnology Governance & Migration In The AI Era
Technology Governance & Migration In The AI Era
 
Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactive
 
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
 
AsyncAPI v3: What’s New? | Kafka Summit London
AsyncAPI v3: What’s New? | Kafka Summit LondonAsyncAPI v3: What’s New? | Kafka Summit London
AsyncAPI v3: What’s New? | Kafka Summit London
 
CERN IoT Kafka Pipelines | Kafka Summit London
CERN IoT Kafka Pipelines | Kafka Summit LondonCERN IoT Kafka Pipelines | Kafka Summit London
CERN IoT Kafka Pipelines | Kafka Summit London
 
Tecnogravura, Cylinder Engraving for Rotogravure
Tecnogravura, Cylinder Engraving for RotogravureTecnogravura, Cylinder Engraving for Rotogravure
Tecnogravura, Cylinder Engraving for Rotogravure
 
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
 

CIDR 2009: James Hamilton Keynote

  • 1. Where Does the Power Go & What to do About it? James Hamilton Conference on Innovative Data Systems Research 2009/1/7 Amazon Web Services e: James@amazon.com w: mvdirona.com/jrh/work b: perspectives.mvdirona.com
  • 2. Agenda • Services Inevitable • Where does the power go? – Power distribution systems & optimizations – Mechanical systems & optimizations – Cooperative, Expendable, Micro-Slice Servers – Critical Load Optimizations • Summary 2009/1/7 http://perspectives.mvdirona.com 2
  • 3. Services Economies of Scale • Substantial economies of scale possible • Compare a very large service with a small/mid-sized: (~1000 servers): Large Service [$13/Mbps]: $0.04/GB Networking Medium [$95/Mbps]: $0.30/GB (7.1x) Large Service: $4.6/GB/year (2x in 2 DC) BLOB Storage Medium: $26.00/GB/year* (5.7x) Large Service: Over 1.000 servers/admin Administration Enterprise: ~140 servers/admin (7.1x) 2006 Chart data • High cost of entry – Physical plant expensive: 15MW ~$200M (infrastructure only) • Summary: significant economies of scale but at high cost of entry – Small number of large players likely outcome 2009/1/7 http://perspectives.mvdirona.com 3
  • 4. Services Different from Enterprises • Enterprise Approach: – Largest cost is people -- scales roughly with servers (~100:1 common) – Enterprise interests center around consolidation & utilization • Consolidate workload onto fewer, larger systems • Large SANs for storage & large routers for networking • Internet-Scale Services Approach: – Largest costs is server & storage H/W • Typically followed by cooling, power distribution, power • Networking varies from very low to dominant depending upon service • People costs under 10% & often under 5% (>1000+:1 server:admin) – Services interests center around work-done-per-$ (or joule) • Observations: • People costs shift from top to nearly irrelevant. • Expect high-scale service techniques to spread to enterprise • Focus instead on work done/$ & work done/joule 2009/1/7 http://perspectives.mvdirona.com 4
  • 5. Agenda • Services Inevitable • Where does the power go? – Power distribution systems & optimizations – Mechanical systems & optimizations – Cooperative, Expendable, Micro-Slice Servers – Critical Load Optimizations • Summary 2009/1/7 http://perspectives.mvdirona.com 5
  • 6. PUE & DCiE • Measure of data center infrastructure efficiency • Power Usage Effectiveness – PUE = (Total Facility Power)/(IT Equipment Power) • Data Center Infrastructure Efficiency – DCiE = (IT Equipment Power)/(Total Facility Power) * 100% Advanced Data Centers http://www.thegreengrid.org/gg_content/TGG_Data_Center_Power_Efficiency_Metrics_PUE_and_DCiE.pdf 2009/1/7 http://perspectives.mvdirona.com 6
  • 7. Power & Related Costs Dominate • Assumptions: – Facility: ~$200M for 15MW facility (15-year amort.) – Servers: ~$2k/each, roughly 50,000 (3-year amort.) – Average server power draw at 30% utilization: 80% – Commercial Power: ~$0.07/kWhr Monthly Costs $284,686 Servers $1,042,440 Power & Cooling Infrastructure $2,997,090 Power $1,296,902 Other Infrastructure 3yr server & 15 yr infrastructure amortization • Observations: • $2.3M/month from charges functionally related to power • Power related costs trending flat or up while server costs trending down Details at: http://perspectives.mvdirona.com/2008/11/28/CostOfPowerInLargeScaleDataCenters.aspx 2009/1/7 http://perspectives.mvdirona.com 7
  • 8. Fully Burdened Cost of Power • Infrastructure cost/watt: – 15 year amortization & 5% money cost – =PMT(5%,15,200000000,0)/(15,000,000) => $1.28/W/yr • Cost per watt using $0.07 Kw*hr: – =-0.07*1.7/1000*0.8*24*365=> $0.83/W/yr (@80% power utilization) • Fully burdened cost of power: – $1.28 + $0.83 => $2.11 2009/1/7 http://perspectives.mvdirona.com 8
  • 9. Where Does the Power Go? • Assuming a pretty good data center with PUE ~1.7 – Each watt to server loses ~0.7W to power distribution losses & cooling • Power losses are easier to track than cooling: – Power transmission & switching losses: 8% • Detailed power distribution losses on next slide – Cooling losses remainder:100-(59+8) => 33% • Data center power consumption: – IT load (servers): 1/1.7=> 59% – Distribution Losses: 8% – Mechanical load(cooling): 33% 2009/1/7 http://perspectives.mvdirona.com 9
  • 10. Agenda • Services Inevitable • Where does the power go? – Power distribution systems & optimizations – Mechanical systems & optimizations – Cooperative, Expendable, Micro-Slice Servers – Critical Load Optimizations • Summary 2009/1/7 http://perspectives.mvdirona.com 10
  • 11. Power Distribution IT Load (servers, storage, Net, …) High Voltage 8% distribution loss Utility Distribution .997^3*.94*.99 = 92.2% 2.5MW Generator (180 gal/hr) 115kv 208V ~1% loss in switch 13.2kv gear & conductors UPS: Transformers Transformers Rotary or Battery Transformers 13.2kv 13.2kv 480V 6% loss 0.3% loss 0.3% loss 0.3% loss 94% efficient, ~97% available 99.7% efficient 99.7% efficient 99.7% efficient 2009/1/7 http://perspectives.mvdirona.com 11
  • 12. Power Redundancy to Geo-Level • Over 20% of entire DC costs is in power redundancy – Batteries supply over 10 min at some facilities (~2 min sufficient) – N+2 generation (2.5MW) at over $2M each • Instead use more, smaller, cheaper data centers • Non-bypass, battery-based UPS in the 94% efficiency range – ~900kW wasted in 15MW facility (4,500 200W servers) – 97% available (still 450kW loss in 15MW facility) 2009/1/7 http://perspectives.mvdirona.com 12
  • 13. Power Distribution Optimization • Two additional conversions in server: – Power Supply: often <80% at typical load – Voltage Regulation Module: ~80% common – ~95% efficient available & affordable • Rules to minimize power distribution losses: 1. Avoid conversions (Less transformer steps & efficient or no UPS) 2. Increase efficiency of conversions 3. High voltage as close to load as possible 4. Size voltage regulators (VRM/VRDs) to load & use efficient parts 5. DC distribution potentially a small win (regulatory issues • Two interesting approaches: – 480VAC (or higher) to rack & 48VDC (or 12VDC) within – 480VAC to PDU and 277VAC to load • 1 leg of 480VAC 3-phase distribution 2009/1/7 http://perspectives.mvdirona.com 13
  • 14. Agenda • Services Inevitable • Where does the power go? – Power distribution systems & optimizations – Mechanical systems & optimizations – Cooperative, Expendable, Micro-Slice Servers – Critical Load Optimizations • Summary 2009/1/7 http://perspectives.mvdirona.com 14
  • 15. Conventional Mechanical Design Blow down & Evaporative Loss for 15MW facility: ~360,000 gal/day Heat Primary Exchanger Pump Cooling (Water-Side Economizer) Tower CWS A/C A/C Pump Condenser Evaporator A/C Compressor Secondary Pump Diluted Hot/Cold Mix Server fans 6 to 9W each leakage Air-side Hot Computer fans Economiz- Overall Room Air ation Mechanical Losses cold Handler ~33% Air Impeller Cold 2009/1/7 http://perspectives.mvdirona.com 15
  • 16. Mechanical Optimization • Simple rules to minimize cooling costs: 1. Raise data center temperatures 2. Tight control of airflow with short paths 3. Cooling towers rather than A/C 4. Air side economization (open the window) 5. Low-grade, waste heat energy reclamation • Best current designs still use air but bring water near servers – Lower heat densities could be 100% air cooled • Common mechanical designs: 33% lost in cooling • PUE under 1.0 within reach with some innovation – Waste heat reclamation in excess of power distribution & cooling overhead (~30% effective reclamation sufficient for <1.0 operation) 2009/1/7 http://perspectives.mvdirona.com 16
  • 17. Agenda • Services Inevitable • Where does the power go? – Power distribution systems & optimizations – Mechanical systems & optimizations – Cooperative, Expendable, Micro-Slice Servers – Critical Load Optimizations • Summary 2009/1/7 http://perspectives.mvdirona.com 17
  • 18. CEMS Speeds & Feeds • CEMS: Cooperative Expendable Micro-Slice Servers – Correct system balance problem with less-capable CPU • Too many cores, running too fast, for memory, bus, disk, … • Joint project with Rackable Systems (http://www.rackable.com/) CEMS V3 CEMS V2 CEMS V1 System-X (Athlon 4850e) Athlon 3400e) (Athlon 2000+) CPU load% 56% 57% 57% 61% RPS 95.9 75.3 54.3 17.0 Price $2,371 $500 $685 $500 Power 295 60 39 33 RPS/Price 0.04 0.15 0.08 0.03 RPS/Joule 0.33 1.25 1.39 0.52 RPS/Rack 1918.4 18062.4 13024.8 4080.0 •CEMS V2 Comparison: •Work Done/$: +375% •Work Done/Joule +379% •Work Done/Rack: +942% Update: New H/W SKU likely will improve numbers by factor of 2. CEMS still a win. 2009/1/7 http://perspectives.mvdirona.com 18
  • 19. Power Yield Management • “Oversell” power, the most valuable Max utility power resource: 10% Max de-rated power – e.g. sell more seats than airplane holds • Overdraw penalty high: – Pop breaker (outage) Dynamic yield mgmt Peak Static yield mgmt – Overdraw utility (fine) Max server label with H/W caps Average • Considerable optimization possible If Max clamp workload variation is understood – Workload diversity & history helpful – Graceful Degradation Mode to shed workload Power Provisioning in a Warehouse-Sized Computer, Xiabo Fan, Wolf Weber, Luize Borroso 2009/1/7 http://perspectives.mvdirona.com 19
  • 20. Critical Load Optimization • Power proportionality is great but full load or shut off is even better – Idle server consumes ~60% power of full load – Industry secret: “great” data center utilization around ~30% – All solutions require changing where a workload is executed • What limits dynamic workload migration? – Networking constraints: VIPs can’t span L2 nets, ACLs static, manual config, etc. – Data Locality: Hard to efficiently move several TB – Workload management: Scheduling work over available resources • Critical load optimizations, in order: – Use the servers: any workload with marginal value over power – Shut them off if you can’t use them – Those servers on should be fully loaded but not all resources fully consumed so use power management (e.g. dynamic voltage & frequency scaling) • Efficient S/W algorithms as important as H/W 2009/1/7 http://perspectives.mvdirona.com 20
  • 21. Resource Consumption Shaping • Essentially yield mgmt applied to full DC Egress charged at 95th percentile • Network charged at 95th percentile: 3PM – Push peaks to troughs 4AM PST PST The – Fill troughs for “free” Pacific Ocean is • e.g. Amazon S3 replication big. – Dynamic resource allocation • Virtual machine helpful but not needed – Charged for symmetrically so ingress effectively free David Treadwell & James Hamilton / Treadwell Graph • Power also charged at 95 th percentile • Server idle to full-load range: ~65% (e.g. 158W to 230W ) • S3 (suspend) or S5 (off) when server not needed • Disks come with both IOPS capability & capacity • Mix hot & cold data to “soak up” both • Encourage priority (urgency) differentiation in charge-back model 2009/1/7 http://perspectives.mvdirona.com 21
  • 22. Agenda • Services Inevitable • Where does the power go? – Power distribution systems & optimizations – Mechanical systems & optimizations – Cooperative, Expendable, Micro-Slice Servers – Critical Load Optimizations • Summary 2009/1/7 http://perspectives.mvdirona.com 22
  • 23. Summary • Current “good” data centers have considerable room for improvement • Where do the power go? – 58% Servers and other IT equipment – 33% mechanical systems – 8% power distribution • Lowest hanging fruit in servers & mechanical systems • Server system optimizations – Utilization levels and general scheduling optimizations – Servers optimized for work done per joule & watt rather than raw performance • CEMS takes only a small step forward but achieves better than 3x improvement in work done/$ and work done/joule 2009/1/7 http://perspectives.mvdirona.com 23
  • 24. More Information • This Slide Deck: – I will post these slides to http://perspectives.mvdirona.com later this week • Designing & Deploying Internet-Scale Services – http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf • Architecture for Modular Data Centers • http://mvdirona.com/jrh/talksAndPapers/JamesRH_CIDR.doc • Increasing DC Efficiency by 4x • http://mvdirona.com/jrh/talksAndPapers/JamesRH_PowerSavings20080604.ppt • Perspectives Blog – http://perspectives.mvdirona.com • Email – James@amazon.com 2009/1/7 http://perspectives.mvdirona.com 24