High Performance Computing: State of the Industry

High Performance Computing
State of the Industry

Anil Vasudeva
Principal Analyst & President
anil@imexresearch.com IMEX
RESEARCH.COM
408-268-0800
©2000-2005 IMEX Research
IMEX

• HPC - Markets Drivers & Industry Dynamics
• State of HPC Technologies
Clustering, Interconnects, Storage, Blade Servers

• Industry Issues/Technology Hurdles
Standards, Management, Fabrics, Thermals, IP Acceleration

• Futures Trends - Computing & Grids
• Holy Grails
Volume Driven Economics, IP Evolution, TF/Desktop
HPC in our Daily Lives (Academia > Hollywood > Wall Street)

IMEX

Server Architectures: Competitive Technologies
Price/Performance Performance: SMP vs. Clustering
Serial vs.Parallel Computing
Multinode
Clusters

Performance - tpc
SMP
w/8 way
MPP
w/4 way
Serial
Price

16 32
MF 8
N-way SMP
6
4

2

Clustering i

Performance Number of Processors

Scalable Perf. vs. Availability
Scalable Performance

MPP
Clustering

SMP FT

UniP
Availability
IMEX

Market Segments by Applications
OLTP
Transaction
10 K Processing
eCommerce

Data DSS (RAID - 0, 3)
(RAID - 1, 5, 6)
1K Warehousing
(Latency)

Visual DB
IOPs

100 Scientific Computing HPC
HPC
Imaging
TPC
10 Audio Streaming
Streaming
HPC
Video
1
1 5 10 50 100 500
MB/sec
*IOs per sesond for a required response time ( ms)
IMEX

HPC – 2 decades of rapid progress

1991 1998 2005 2010
System Cray Y-MP C916 Sun HPC10000 Shuttle@newegg Blade
24x 333 MHz, Ultra 4x 2.6 GHz, x64
Architecture 16x Vector, 4 GB, Sparc II, 24 GB,S- Bit, 4 4x 10GHz, x64 bit,
Bus Bus GB, GbE 16GB, 10GbE
Win Server 2003
OS
UNICOS Solaris 2.5.1 SP2 LongHorn/Linux
Gflops ~10 ~10 ~10 ~100
Price $40 million $1 Million <$4000 ~$1000
Price Reduction 1 1/40 1/10,000 1/400,000
Government Small>Large
Target Audience
National Labs Large Enterprises Businesses Every Professional
Bioinformatics,
Applications Classified, Climate, Manufacturing, Materials Architects,Wall
Physics Energy,Finance Sciences Street, Hollywood

IMEX

The rapid rise of Clusters in HPC

Rise of Cluster Computing

350

# Clustered Computers in Top 500.org
300
250

200
150

100
50
0

1999e

2000

2001

2002

2003

2004e
IMEX

HPC Market Drivers

• Availability of industry standard HW making
HPC mainstream
• HSV Servers @$200, Storage @25c/GB, Add-On GbE NICS at $10, GbE Switches
at $2-5/port
• Migration to Embedded ASICs on Blades

• Rapid procurement, installation & System
Integration simplifying acquisition
• Cluster ready apps accelerating market growth
• Engineering (LSTC, Ansys..)
• Bioinformatics (Blast, Gaussian..)
• Finance (Matlab, Excel…)
• Oil & Gas (Eclipse, Promagic..)
• Government (Uexplore, NOAA Hysplit..)

IMEX

HPC Servers Rev - 15% of Total Server Market

WW HPC Rev ($M)
$9,000

$8,000

$7,000

$6,000
Departm ental
Rev $M

$5,000

$4,000

$3,000
Divisional
$2,000
Enterprise
$1,000
Capability
$-
2003 2004 2005 2006 2007 2008

IMEX

HPC Market Leaders 2004

Market Leaders

IMEX

Rise of Industry Standard Architecture in HPC

X86 Architecture (Intel & AMD)
using Linux and Microsoft
based clusters dominate in
High Perrformance Computing
Architecture.

TeraFlop HPC Using
Off-the-Shelf Modules
Intel servers with PCI Express, Infiniband
HBAs & Switches, SATA/SAS HDD or
AMD Opteron Servers, GbE HBA/Switches,
SATA

IMEX

Factors Affecting Performance

• Applications Configurations affect Linpack
significantly
(3 metrics have the most affect)
• Problem Size
• Size of Blocks
• Topology

• Tightly Coupled MPI Applications
• Very sensitive to network performance characteristics
- internodal communications delay
- OS Network Stack is a significant factor
• Very sensitive to mismatched node performance
- Random OS activities can add msec delays to
usec type communication line delays

IMEX

HPC Architecture & Challenges
Architecture Goals & Criteria
Goals
• Maximize compute time
HPC Communications Stack
vs. messaging time
• High constant cross-
Entire Stack
sectional throughput Needs Attention
• Reduce or hide latency Data Application
Metrics Upper Layers

• Bandwidth TCP/IP (or Offload I/F)

• Latency OS

• System Interface CPU
• SW Stack Memory

Chipset

Challenge is to move data from I/O interconnect
Network Interface
application to the network and back (w or w/o TOE)
with minimum latency
Data

IMEX

Application Application Application

Application Application
Memory Memory

Control
Control
Host Host

Host
Socket RDMA
CPU
RDMA
Adapter
TCP TCP CPU CPU

IP IP
Data Data
Kernel Kernel
Traditional Memory Memory
Adapter Ethernet Ethernet

RDMA Adapter RDMA Adapter

Network

IMEX

Interconnect Price/Performance Hierarchy

High MB/s,
Low Latency,
Price/Port Quadrics

Myrinet

InfiniBand

10 GbE w/ RDMA, TOE

10 GbE w/ TOE

1 GbE
Market Size
Microsoft Chimney Software 2004-07

IMEX

State of Interconnect Fabrics

EHERNET - LAN/SAN/MAN/WAN

FE
GbE

ATM IBM/SP
Latency/hop

MPP

SMP/NUMA
Servernet Dolphin
T3E
Myrinet SGI Mellanox IB
GbE
w/TOE Quadrics
FJ/Synfinity
GbE
w/RDMA
10GbE
w/RDMA

Bandwidth/link

IMEX

Interconnects: Industry Progress

IMEX

InfiniBand Clustering
• InfiniBand and Industry Standard Servers
– Displacing Mainframe Computers in the Data Center
with Three Standard Hardware Components
– Powerful Low Cost Combo Out Performs SMP
12-Nodes (24-CPUs) Up to 72 GFlops of
CPU Power; 24+ GB Memory

Industry Standard Server

InfiniBand Switch
=>
InfiniBand HCA

Enterprise Availability:
With Fabric Failover
IMEX

InfiniBand rears up its head

• Low Cost IB 10G Silicon & Adapters

– Very Low Cost 10G Host Adapters
• Host Channel Adapter ~ $69 in volume
• Switch Si at less than $30 a port in volume

• Compelling price/performance
• InfiniBand vs. 10GE Advantages in
the Short Term
– 15X – 20X price advantage vs 10GE
– 3X – 10X latency advantage
– 10X – 20X CPU utilization advantage
– 3X-5X throughput advantage
• Adoption by most majors
• Products
– InfiniBand to IP
* – InfiniBand to FC
– Advanced Management
IMEX

Ethernet - TCP + iSCSI Offload Engines

Typical 1GbE NIC
File System File System

Host
File System

SW
Volume Manager Volume Manager
Volume Manager
SCSI Device Driver SCSI Device Driver
SCSI Device Driver
iSCSI iSCSI iSCSI

Adapter
TCP TCP TCP

IP IP IP

Ethernet Ethernet Ethernet

Host CPU

HBA Firmware

HBA Hardware

IMEX

Interconnects: Applicability
Grids
100 Mbit 1 Gig 10 Gig 40 Gig
Data Center Ethernet Ethernet Ethernet Ethernet
LAN

SAN Fibre Channel
Storage RDMA NICs with iSCSI
NAS
Storage RDMA NICs

Clusters
Myrinet
InfiniBand
Blade Quadrics
Chassis
Network

SMP PCI Express & Advanced Switching
Backplanes SunFire Link, IBM Federation

<1 Gigabit 1 Gigabit 10 Gigabit >10 Gigabit
IMEX

InfiniBand Adopters
• Major providers of InfiniBand
Solutions
• Enterprise Class Products
– InfiniBand to IP
– InfiniBand to FC
– Advanced Mgmt
• Platform support includes
– Intel (IA32 + 64EMT, Itanium),
AMD Opteron, Power PC, Sparc
• Operating Systems
– Linux, Windows, AIX, HPUX,
Solaris, MAC OS X, VxWorks

IMEX

HPC Interconnect – Leaders

Myrinet for the highly
latency sensitive
applications
and GbE for the
majority applications
Dominate the
Interconnect for HPC.
Infiniband is rearing
to dominate at the
Midrange latency
applications

IMEX

HPC Tools Availability

IMEX

Data Center Environmentals Prep

Temp F

Speed fpm

Sources:
Fluent Airpak,
Ansys CFD,

IMEX

HPC embraces a broad spectrum of markets ..

High Performance
Computing
World

IMEX

3 Tier Computing Infrastructure
Application
End-Users  Uniform and ubiquitous physical
Internet Connectivity – “Wire Once”
 Any user to any server via Internet
Tier –1  Any server to any server via the LAN
Network Services:  Any sever to any storage via the SAN
- Web Servers,  Dynamic Logical Binding of…
- Firewalls,
- Load balancers  server network identification
LAN  server OS version
 server application assignment
Tier-2
 application data volumes
Application Services:
- Application Servers  Resulting in
 Application Mobility across servers
 Data mobility across storage
Tier-3
SAN
Data Base Services
- Database Servers
- Storage

Storage
IMEX

E2E Internet & 3 Tier Computing
HA, Secure Data Center

FC SAN IP SAN NAS NAS

IP Storage Network
Web
App
Internet Web DB
Edge Core Web
App
Optical
Web
DB
Web
App
Web

Web
DB
App
Web

Edge Applications Data Center
Directory Security Policy Management

Software OS Platform

IMEX

Fabric based Clusters Architecture

Ethernet,
Network Fabric Wi-Fi.

System Fabric
Quadrics,
Management

Myrinet,
SNMP, SCI
Fabric

IETF/ InfiniBand,
CIM, Ethernet / IP,
SMI-S, Ethernet IP
SMASH. w/TOE,
Ethernet IP
w/TOE and
RDMA.

SCSI,
Fiber
Channel,
ISCSI.

IMEX

Fabric based Grid Applications
Storage Systems

IMEX

Scale Up (Consolidated SMP Servers)
Server Platforms - Migrations

Large SMP Clusters
Parallel Sysplex High Availability/
- Scalable Nodes, Hi Perform.Clusters
- Dynamic LPAR - Linux/AIX/Solaris Clusters
- SW Provisioning/VMWare - HPC Clusters
IBM: i890/p690/z990 DELL: PowerEdge 6650
SUN: SunFire 4800, 12K ORACLE: 9iRAC/PE6650

Blade Servers
High Density Rack Mount
- Highly Integrated Dense Form Factor
- Rapid Deployment, Flexible Architect.
RLX: ServerBlade 300i, 800i, HP: 40b, p
FUJITSU: BX300, DELL:1655
IBM: HS20, Sun:

Scale Out (Clustered Blade Servers)
IMEX

Blades Servers - Infrastructure
Management Modules Remote Mgmt+KVM over IP

Midplane Networking: Gbit Ethernet Switches
W/Connectors Storage: IP NAS or FC SAN
To Blades & Back
Switch
Cooling N+1Fans/Cooling Modules
Processor Blades Modules

(6-24 typically) Power: N+1Power Supplies

Blade
Control
Panel

USB Ports,
CD/Floppy I/F

~ GbE Switch supports
Trunking/Port Aggregation
Flow Control
Gbit QoS Packet Prioritization
Memory Ethernet SNMP/RMON
Systems I/F
Micro- DDR w IGMP/BOOTP/TFTP
Monitor
Processors ECC ……
Module
© 2003 Source: IMEX Research

IMEX

Management – Blade Servers

Ultra-dense HW
• Density

20 %
• Power Consumption
20% • Cable Management

Management
• Platforms integration:
 Hardware
 Software

80%
 Network
 Management
• Scalability
80 %

• Security
• Manageability
• Lowest TCO

IMEX

Blade Servers Market Opportunity
$12
WW Blade Servers
$10 Market Opportunity
(cum 2004-2008)
Factory Rev $B

$8 Revenues ~ $24 Billion
Units ~ 9 Million
$6

$4

$2

$-
2003 2004 2005 2006 2007 2008

Data: IMEX Research 2005
IMEX

Blade Servers: Vendor Positioning Index
(As of 31 Mar 2004 - See IMEX Blade Servers Industry Report 2005 for new data)
Strategy (Potential)

IBM

Fujitsu Siemens

Hitachi HP
Linux Intel

© 2003-05 IMEX Research
NEC
Networx
Dell
Nexcom Tatung
Sun Appro Egenera
Penguin
Rackable Sys
HPC Syst Verari
Delivery (Execution) IMEX

iSCSI on roadmap of storage vendors

SAN iSCSI NAS
(Fibre
COST

(IP) (IP)
Channel)
•iSCSI delivers low-cost Networked Storage
–Higher availability,Increased usage
High End
–Centralized management,Storage functionality
FC & iSCSI

Mid-Tier
FC & iSCSI
Mid-Tier
NAS & iSCSI

Entry
FC & iSCSI
SERVICE LEVEL

IMEX

System Cost vs. High Availability

z/390
$10 M Sysplex
Avg. S/390
S/390
Syst z/390 MVS
MVS
IBM
IBM
Price
Propl
Propl
$1 M Prop.
Sun-Solaris
Sun-Solaris
HP-UX
HP-UX
IBM
IBM IBM-AIX
IBM-AIX
$100 K Clustered NCR-SVR4
NCR-SVR4 Bull
Bull Dell
Dell
UNIX AS400 Fujitsu
Fujitsu HP
HP
IBM
IBM NEC
NEC
Siemens
Siemens Stratus
Stratus
UNIX
Cluster Win Win
$10K Linux
Clustered Linux

0.1 1 10 100 Downtime
Hrs./Syst/Yr
99.999% 99.99% 99.9% 99.0%

System Availability
IMEX

The iSCSI SAN

Client
Workstations
Client requests data from App Server
LAN
Ethernet Header IP TCP DATA CRC
Ethernet
Switches

Application &
DB Servers
(iSCSI Initiator) iSCSI-equipped App Server (“initiator”)
requests data from an iSCSI storage (“target”)
SAN Ethernet CRC
IP TCP iSCSI SCSI DATA
GigE Header
Switches

Storage Arrays
(iSCSI Target)
To Other LANs,
Servers or IP SANs

IMEX

Tiered Storage by Price/Performance

Desktop PC Servers Large Enterprise
Entry NAS Midline Storage SAN & NAS
Workstations Mainstream NAS/SAN

PC SAS SAS HBA/RAID FC RAID
Chipset HBA
Drive
Interface SATA SAS SAS

Drive
Types 5400 RPM 10K RPM
7200 RPM 15K RPM

SATA Drive SAS Drive FC Drive
Source:Maxtor

IMEX

Cost Driven Data Storage Markets

100%

90%
SATA
80%

70%

60%
SAS
50%

40%

30%
FC
20%

10% P_SCSI
0%
2003 2004 2005 2006 2007 2008

IMEX

Grid Computing

• Grid is a Catch-all Marketing Term
• Means different things to different constituencies
• Desktop Cycle-Stealing
• Manahed HPC Clusters
• Virtualization of Data Center Resources
• Outsourcing to “Utility Data Centers”
• Multiple, opposing requirements
• Compute-intensive applications want to ship data tp idle processor resources
• Data-Intensive applications want to ship computations to appropriate large data
repositories

• An evolving entity with many pulls & pushes
• Economics, Feasibilities, management of hetero vs homogeneous HW

IMEX

Grid Challenges: Hype vs. Reality

• Cost of Local Grids (LAGs) – ala Blades
cheaper than Wide Area Grids (WAGs)
• Computing Cost $1000 > 1 cpu day (10 Terops) =$1
• 10 TB network transfer Cost = $1
• Internet BW cost $100/mbps/mo > 1GB network xfr cost ~$1
• Result: Local HPC Cluster is 10,000 cheaper than WAN Communication

• MPI Style Apps work well in LAN Clusters but
uneconomical in WAN
• Data analysis best done by moving programs to data not data to programs
• Small data, high compute applications woek well across internet
• Internet is not the CPU back plane

IMEX

Grids – Homogeneous vs Heterogeneous

IMEX

Grid Services Architecture & Standards

IMEX

Grid Computing – Across 3 Tiers

Grid Control
Storage
Databases
Application Servers

IMEX

Utility Computing/Web Services
Autonomics
• Self Healing
• Self Optimizing
• Universal Identity
• Single System Image
• ….

Internet Internet

Network Switching Pool
• HP - Utility Data Center
Utility Controller

Virtualization
Load Balancer Pool
• Sun - N1/SunONE/Orion
Firewall Pool
Server
Virtualization
• Microsoft - .Net
Server Pool

NAS Pool
• IBM - OnDemand/Autonomics
Storage Switching Pool

Virtualization
Storage Pool

IMEX

Future Computing: Processes Driven

Initiator Initiator Initiator

Old model New model
Hardware is the initiator Processes are the initiators
HBA ID
Initiator

Allows applications to move storage
wherever they are - storage no longer
cares where they are

Target

IMEX

Application-Driven Storage Virtualization


Moving from storage ...to storage associated
associated with with application
servers... processes

Old Model New Model
Server Hardware is the Allows applications to
initiator. Storage move storage from the
Drives are the Target virtualized pool, wherever
they may be

IMEX

Future: Embedded10Gb,TOE, iSCSI

CPU CPU CPU
CPU CPU
CPU CPU
CPU CPU
CPU CPU CPU
CPU CPU CPU CPU
TOE

MC MC
MC MC
MC MC
MC
MC DMA
DMA

PCI Exp PCI Exp PCI Exp PCI Exp
PCI Exp
I/O Uses I/O Uses
I/O 1GbE
Bridge Bridge Syst Bridge Syst 10 GbE
Mmry Mmry 1GbE

TOE GbE TOE 1GbE
GbE 1GbE
PCI-X
Ext
TOE Mmry PCI PCI-X TOE PCI PCI-X
with
10GbE
2GbE

Ext.Mmry

2x1Gb TOE NIC TOE chipset 10 Gb TOE NIC Embedded

2003 2004 2005 2006 onwards

IMEX

Future: IP Storage Management on a chip

Host Services Integration Storage
Provisioning
File system monitoring Storage provisioning Win, LINUX, Solaris Layer

IP SAN Management
Management Management of MultiPath IO Supp Security
IP SAN
Console iSCSI HBAs and Failover (iSNS, CHAP, SRP) Management
Layer
Virtualization Mirroring Snapshot Fail-Over

iSCSI Target Management Appliance
LVM, Error Handling, SCSI Daemon, API Interoperability
Service
Layer
HW Acceleration: TOE, iSCSI Offload, IPsec

IMEX

Future Intelligent IP Infrastructure

Present Data Center Infrastructue Future Infrastructue

Clients

LAN Ntwk
Mgmt WS

Application & Intelligent
DB Servers Uniform
IP-based
Infrastructure

Mgmt
SAN Storage WS
Mgmt WS

Storage
Sub Systems

IMEX

Future - IP Based Infrastructure

Servers Storage

IP
Networks Infrastructure Telecom

Virtualization
Provisioning
Automation

IMEX

HPC – From Academia to Wall St to Hollywood

High Performance Entertainment
High Performance Commercial Bioinformatics Decision-Support Entertainment
Commercial BioInformatics Decision
Audio/Video OnDemand
Computing
Computing Visualization
Visualization Support Syst Audio/Video On Demand
Systems
Data: IMEX Research & Panasas

100+ Teraflops Rendering (Texture & Polygons) Data rate & capacity
Throughput = 100 GB/s Throughput = 1.2 GB/s Throughput : DSL/Cable
IMEX
©2004--05 IMEX Research.com

HPC: From Homeland Security to Bioinformatics

IMEX

HPC From Academia to Hollywood

IMEX

State of Interconnect Fabrics

Latency/hop

1 ms EHERNET - LAN/SAN/MAN/WAN
FE GbE IB - CLUSTER
ATM IBM/SP MPP
100 us
Servernet Dolphin SMP/NUMA
T3E
Myrinet SGI
GbE Quadrics Mellanox
w/TOE
10 us FJ/Synfinity
Synfinity-1
GbE 10GbE
w/RDMA Scali w/RDMA

1us
100 Mb/sec 1Gb/sec 10 Gb/sec
Bandwidth/link

IMEX

High Performance Computing: State of the Industry

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (8)

Destaque

Destaque (11)

Semelhante a High Performance Computing: State of the Industry

Semelhante a High Performance Computing: State of the Industry (20)

Mais de IMEX Research

Mais de IMEX Research (15)

Último

Último (20)

High Performance Computing: State of the Industry