Sun sparc enterprise t5120 and t5220 servers technical presentation

Sun SPARC Enterprise
T5120 and T5220
Rack Server TTT
Peter A. Wilson
Technical Marketing
Sun Microsystems, Inc.

1

Agenda
• Introduction
• Overview of UltraSPARC T2
• Sun SPARC Enterprise T5x20 servers
> Comparison to Sun Fire T1000/T2000
> System Architecture and Tours
> Memory, Networking and IO Expansion
> Disks, Fans and Power Supplies
> Solaris for T5x20
• ILOM 2.0
• LDOMs for T5x20
• CRUs/FRUs and Configuration Options
• Tuning and Performance
• Tools, Information and References
• Q & A ...
Sun Internal and Approved Partners Only 2

CMT Processor Strategy
First True System on a Chip

• Scaling performance with frequency is
a dead-end
• Multi-core/Multi-thread processors can
exploit application parallelism
• Able to Aggressively attack memory
latency
• Provide rich availability feature set
• Integrate networking, I/O and security
capability
• Virtualize the architecture

Agenda
• Introduction
> Comparison to Sun Fire T1000/T2000
> Solaris for T5x20
• ILOM 2.0
• LDOMs for T5x20
• Q & A ...

UltraSPARC T2: • Up to 8 cores @1.2GHz or 1.4GHz
• Up to 64 threads per CPU
TrueSystem On a Chip • Up to 16 FB-DIMMs, 4 memory
controllers
> Up to 64GB memory (4GB DIMMs)
FB DIMM FB DIMM FB DIMM FB DIMM
> 2.5x memory BW of UltraSPARC T1
FB DIMM FB DIMM FB DIMM FB DIMM • 8 x fully pipelined Floating Point
units / core, 1 per core
MCU MCU MCU MCU • Dual 10Gbit Ethernet and PCI-E
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$
integrated onto chip
Full Cross Bar
• 4MB L2 (8 banks) 16 way
C0 C1 C2 C3 C4 C5 C6 C7
associative
FPU FPU FPU FPU FPU FPU FPU FPU • Enhanced MAU/Security co-
MAU MAU MAU MAU MAU MAU MAU MAU

DMA Sub-System
processor per core
> DES, 3DES, AES, RC4, SHA1,
NIU Sys I/F
SHA256, MD5, RSA to 2048 key,
PCI-E
(E-net+) Buffer Switch Core ECC,CRC32
Power ~95W • Advanced Power saving features
2x 10GE Ethernet <1.5W/thread x8 @2.5GHz
4GBytes/s bi-directional • 65nm process technology

UltraSPARC T2 Processor
L2 Data L2 Data
Bank 0 Bank 4

L2B0 SPARC SPARC SPARC SPARC L2B4

L2 Data Core 0 Core 1 Core 5 Core 4 L2 Data

Bank 1 Bank 5

L2B1 L2B5
L2 L2 L2 L2
MCU0 MCU2
NCU

TAG0 TAG1 TAG5 TAG4 FSR

EFU
MCU1 MCU3

SIO
SII
L2B3 CCX L2B7
FSR CCU
L2 Data L2 Data
Bank 3 Bank 7
L2 L2 L2 L2
L2B2 L2B6
TAG2 TAG3 TAG7 TAG6
L2 Data L2 Data

Bank 2 Bank 6

SPARC SPARC SPARC SPARC
DMU
Core 2 Core 3 Core 7 Core 6 RDP TDS

PEU

RTX
PSR ESR FSR MAC

For a video tour with Rick Hetherington, go to http://www.youtube.com/watch?v=B8DNexYcsa4

UltraSPARC T2 Core Block Diagram
• IFU – Instruction Fetch Unit
> 16 KB I$, 32B lines, 8-way SA
TLU IFU > 64-entry fully-associative ITLB
• EXU0/1 – Integer Execution Units
> 4 threads share each unit
> Executes one integer
EXU0 EXU1 instruction/cycle
• LSU – Load/Store Unit
MMU/ > 8KB D$, 16B lines, 4-way SA
HWTW > 128-entry fully-associative
DTLB
• FGU – Floating/Graphics Unit
SPU FGU LSU • SPU – Stream Processing Unit
> Cryptographic acceleration
• TLU – Trap Logic Unit
> Updates machine state, handles
exceptions and interrupts
Gasket
• MMU – Memory Management Unit
> Hardware tablewalk (HWTW)
xbar/L2 > 8KB, 64KB, 4MB, 256MB pages

Floating Point Unit
• Each Core has its own Floating Point Unit. This removes
completely the 2% FP restriction of T1.
• Only a 6 cycle penalty to the FPU
• Fully-pipelined (except divide/sqrt)
> Divide/sqrt in parallel with add or multiply operations of
Data
other threads
• Full VIS 2.0 implementation
• FPU performs integer multiply, divide, population count
• Multiplier enhancements for modular arithmetic operations
in the Crypto Unit
• Performance is impressive...testing and tuning still
happening


Niagara 2 Crypto Performance
• Supports ten most common ciphers and
Comment
secure hashing functionsCycles/
Iterations/ Block
Block (bytes) Block GB/S
2 rounds/cycle > RC4, DES, 3DES, AES-128, AES0192, AES-256, MDS, SHA-1,
2 rounds/cycle SHA-256
16 rounds 8 8/9+ 12.0
3 overlapped cycles 48 rounds 8 25/24 4.0
Data • On-chip crypto accelerators and built-in 10GbE vs.
4 words into 44 words 1 1 37683 12.0
accelerator cards
6 words into 52 words - - 32 -

with UltraSPARC T2
8 words into 60 words - - 37 -

New Ciphers
>
2 column over two cycles <36% crypto performance 41
- - -
2 column over two cycles 10 rounds 16 30/30 6.36
>
2 column over two cycles
>23% CPU overhead
12 rounds 16 36/36 5.28
2 cycles per step
• Leading cryptography performance
2 cycles per step
14 rounds
80 steps
16
64
42/42
160/160
4.56
4.80
2 cycles per step > Over 30x greater RSA1024 performance than 2-socket
64 steps 64 128/128 6.00 IBM p510
32 bits per cycle 64 steps 64 128/128 6.00
• Best SSL Performance
1 4 1 48.00

> 2.2x better performace/watt than 4-socket HP DL585G2
> 1.74x better performance/watt than 2-socket HP DL380G5
> 15.6x better AES128 performance than crypto acclerator card

FB-DIMM Memory Interface

Feature UltraSPARC T1 UltraSPARC T2
Require Pins 1000 440
Available Bandwidth 23GBytes/second 50Gbytes/second
Data Up to 64x DIMM slots
Memory Capacity up to 16x DIMM slots
T5x20 uses 16 DIMM slots
Memory Controllers 4 4
DIMMS/Channel/Controller 02/02/04 08/02/04
Memory Latency 96ns 126ns
DIMM speeds 400MHz 667MHz
DIMM Power 7watts/DIMM 12.5Watts/DIMM


10GbE Network Interface Unit
• Fully Integrated Network Interface Unit
> Eliminates PCI-E chipset latencies
> 30% better performance vs PCI-E NIC
– Mainly through reduced latency
• Two 10 GbE ports (XAUI)
• 16 Rx and 16 Tx DMA channels for virtualization
Data
• Packet classification
> Incoming flows are classified at layer 2 ,3 and 4 and put
into the Rx DMA buffer according to the classification
rules that matched the flow
• nxge device driver, source base shared with Atlas
• Performance estimates, testing/tuning ongoing
– 1 port TX/RX = about 9.4Gbps in each direction with about
only 40-50% CPU utilisation
– 2 port TX/RX = 12GbpsTX, 18GbpsRX with 70-80% CPU
utilisation (still 20 threads available for application)

Agenda
• Introduction
> Comparison to SunFire T1000/T2000
> Solaris for T5x20
• ILOM 2.0
• LDOMs for T5x20
• Q & A ...

Sun SPARC Enterprise T5x20 Extreme Rackmount Density
UltraSPARC T2 Platform • 1/2U chassis, 28” depth
• Up to 8 cores @ 1.2 - 1.4GHz
> Up to 64 threads
• 16 FB-DIMM memory slots
> 64GB of memory, (4GB DIMMs)
High Reliability
• 4 to 16 hot plug SAS 2.5” disk
• Optimized for best price / drive (RAID 0,1)
performance & performance / watt • N+N Redundant, hot-swappable
PSUs and Fans. DC PSU option
• Scale-Out of web & network for 1U systems only
infrastructure Expandable
• Extension to T1000/T2000 • 3/6 PCI-E expansion slots
(including 2 x 10 GBE I/0 slots)
product line • 4 x 1GbE Ethernet as standard
• 4 USB ports
Available Now
Sun Internal Only 13

Sun Fire T1000 / T5120 Differences

Feature Sun Fire T1000 SPARC Enterprise T5120
Form Factor 1U, 18¾” deep 1U, 28” deep
UltraSPARC T1 UltraSPARC T2
CPU
1.0 GHz, 32 threads 1.2/1.4 GHz 64 threads
DDR2 32GB Max FB-DIMM 64GB MAX
Memory
8x slots 16x slots
Network 4x GbE 4x GbE + 2x 10GbE
1x 3.5” SATA, non-hot-swap
Internal Storage Up to 8 x 2.5” SAS, hot-swap
OR 2x 2.5” SAS
Removable Media None 1x DVD-ROM
Serial 1x RS-232, No USB 1x RS-232, 4x USB
PCI Express slots 1x (low profile) 3x(low profile)
1x 300W AC 2x 650W AC & DC option
Power Supply
Non-Redundant Redundant/Hot-Swap
Fans Non-Hot-Swap Redundant/Hot-Swap

Sun Fire T2000 / T5220 Differences

Feature Sun Fire T2000 SPARC Enterprise T5220
Form Factor 2U, 24” deep 2U, 28” deep
UltraSPARC T1 UltraSPARC T2
CPU
1.0 - 1.4GHz 32 threads 1.2/1.4 GHz 64 threads
DDR2, 64GB MAX FB-DIMM, 64GB MAX
Memory
16x Slots 16x Slots
Network(10/100/1000) 4x GbE 4x GbE + 2x 10GbE
Internal Storage 4x 2.5” SAS, hot-swap Up to 16 x 2.5” SAS, hot-swap
Removable Media 1x DVD-ROM 1x DVD-ROM
Serial 1x RS-232, 4x USB 1x RS-232, 4x USB
PCI Express slots 3x(low profile) 6x(low profile)
PCI-X slots 2 x (low profile) None
2x 450W AC 2x 750W AC
Power Supply
Redundant/Hot-Swap Redundant/Hot-Swap

Front Panel Dual USB 2.0 ports

Locator LED
/Button DVD-ROM Disk Drive Map
Fault LED Top Fault LED
Status LED DISK 0 DISK 1 DISK 2 DISK 3 PSU Fault LED
Power Button Temp Fault LED

Serial Number label
DISK 1 DISK 3 DISK 5 DVD-ROM
DISK 0 DISK 2 DISK 4 DISK 6 DISK 7

Locator LED
/Button
Fault LED Top Fault LED
Status LED PSU Fault LED
Power Button Temp Fault LED
Disk Drive Map
Serial Number

Dual USB 2.0 ports
DISK 3 DISK 7 DISK 12 DISK 15
DISK 2 DISK 6 DISK 11 DISK 14

Rear Panel
PCI-E Slot 0/XAUI 0 PCI-E Slot 1/XAUI 1 PCI-E Slot 2
x8 Slot/x4 Electrical x8 Slot/x4 Electrical x8 Slot/x8 Electrical

PCI/XAUI 0 PCI/XAUI 1 PCI 2
PSU PSU 0 PSU 1
Status
LED's
Chassis
Status LED's Quad Gigabit Dual POSIX DB-9
and Locator Button Ethernet Ports USB 2.0 Serial Port
SP SP Ports
Serial Network
Port Port

PSU 1
PCI 3 PCI 4 PCI 5
PCI/XAUI 0 PCI/XAUI 1 PCI 2
PSU 0

Chassis
PSU Status LED's Quad Gigabit POSIX DB-9
Status Dual
and Locator Button Ethernet Ports USB 2.0 Serial Port
LED's SP SP Ports
Serial Network
Port Port


Architecture Block Diagram - T5120
16 x FB-DIMM 4 Memory Controllers, 2 Channels, 2 DIMMS
(DDR2-667)

Disk Chassis
1RU

SSI
x8 FPGA
10GbE
PCI-E 10GbE
x4 LSI x4 Switch x8 MPC885
SAS links x4 1068E ILOM
PLX 8533 Service
DVD x4
x4 Processor
PCI-E
Switch
PCI-E
USB 2.0 PCI-E x1 Switch PLX 8533
to IDE x8
2.0
to 10GbE
USB PLX 8517 x4 SerDes
BCM8704
USB 2.0 x4 x4
Hub 10GbE
USB Intel Intel Cu PHY
2.0
Dual Dual BCMxxxx
GbE GbE XFP

Front Panel
10GbE
0 1 2 3 Fibre
Plugin
USB Quad GbE PCI-E PCI-E PCI-E Serial Network POSIX
Rear Panel Connectors x16 x8 x8 Mgt Mgt Serial DB-9

Architecture Block Diagram - T5220
16 x FB-DIMM 4 Memory Controllers, 2 Channels, 2 DIMMS
(DDR2-667)

Disk Chassis
1RU 2RU/8

SSI
x8 FPGA
10GbE
x4 PCI-E 10GbE
LSI x4 Switch x8 MPC885
SAS links x4 1068E ILOM
PLX 8533 Service
DVD x4
x4 Processor
PCI-E
x8 Switch
PCI-E
USB 2.0 PCI-E x1 Switch PLX 8533
to IDE x8
2.0
to 10GbE
USB PLX 8517 x4 x4 x4 SerDes
BCM8704
USB 2.0 x4 x4
Hub 2RU Only 10GbE
USB Intel Intel Cu PHY
2.0
Dual Dual BCMxxxx
GbE GbE XFP

Front Panel
10GbE
0 1 2 3 Fibre
Plugin
USB Quad GbE PCI-E PCI-E PCI-E PCI-E PCI-E PCI-E Serial Network POSIX
Rear Panel Connectors x16 x8 x8 x8 x8 x8 Mgt Mgt Serial DB-9

Internal View (T5120) Fault Remind System Configuration
Button Card (SCC)EEPROM
Service Processor

FB1/FM0
FB-DIMMs PCI-E Slot 2
x16(m)/x8(e)

FB-DIMMs
FB0/FM1

FB1/FM1
PCI-E Slot 1
UltraSPARC x8(m)/x4(e)
or
T2 XAUI 1

Disk
Chassis FB-DIMMs
FB1/FM2

PCI-E Slot 0
x8(m)/x4(e)
FB-DIMMs or
XAUI 0

PSU1
Power Power
Board Supplies
PSU0

SAS x4 cable Power BUS Bars Chassis Intrusion Reed
to 4 Fan Assemblies are required Switch X4 SAS Connector 0 (cable attached)
Disk Chassis RED = +12Volts (magnet on underside of lid) X4 SAS Connector 1 (not used in 1RU)
(chassis can take up to 8) BLUE = Ground


Internal View (T5220) Fault Remind System Configuration
Button EEPROM
Service Processor

PCI-E Slot 2

FB0/FM0
FB-DIMMs x8(m)/x8(e)
x16(m)/x8(e)

FB-DIMMs

PCI-E Slot 1
UltraSPARC x8(m)/x4(e)
T2 x8(m)/x4(e)
FB0/FM1

or
XAUI 1
Disk
Chassis FB-DIMMs
PCI-E Slot 0
x8(m)/x4(e)
FB-DIMMs x8(m)/x4(e)
or
XAUI 0
FB0/FM2

Power PSU's
Board Power Vertically
Supplies Stacked

Two Power BUS Bars Chassis Intrusion Reed
SAS x4 cables 3 Fan Assemblies are required Switch X4 SAS Connector 0 (cable attached)
to RED = +12Volts (magnet on underside of lid) X4 SAS Connector 1 (cable attached)
(chassis can take up to 6) BLUE = Ground
Disk Chassis


Memory Physical Layout
Chassis Chassis
Front Rear
Channel1/DIMM 1/Branch 1 3
Memory Population Rules
Channel0/DIMM 0/Branch 1 1 1. All Branches must have at least one
DIMM populated
2 - min config is 4 x 1GB DIMMS
Channel1/DIMM 0/Branch 0
Channel0/DIMM 1/Branch 0 3 2. Populate Branches starting with the
nearest slot to the CPU first
- See “1” in the diagram
C0D0B0, C0D0B1, C0D0B2, C0D0B3

UltraSPARC 3. All DIMMS must be the same size
T2 4. In Branch all DIMMS must be indentical
(SUN Part No.)
- Upgrading from 4 to 8 DIMM
configs means moving DIMMS to
Channel0/DIMM 0/Branch 2 1 ensure sameness
Channel0/DIMM 1/Branch 2 3 5. 12 DIMM configurations are NOT
Channel1/DIMM 0/Branch 2 2 supported.
Channel1/DIMM 1/Branch 2 3 6. Only 1GB, 2GB and 4GB DIMMS are
1 supported
Channel0/DIMM 0/Branch 3
Channel0/DIMM 1/Branch 3 3 7. Memory is FB-DIMM DDR2-667, there
is no clocking down of memory in any
Channel1/DIMM 0/Branch 3 2 config.


Memory Configuration Rules
• Populate at least 4 DIMMS at a time
• First
> Fill DIMM0 in Channel 0 in each Branch
• Second
> Fill DIMM0 in Channel 1 in each Branch
• Third
> Fill remaining DIMM slots.
• All DIMMS must be the same size(i.e all 1GB, 2GB or 4GB, no
mixing)
• In a Branch all DIMMS must have the same SUN Part No.
• 12 DIMM configurations are NOT supported.
• No clocking down of memory in any config.

Memory Population Examples
4 DIMM
configuration
8 DIMM
Channel1/DIMM 1/Branch 1 3 configuration
Channel1/DIMM 0/Branch 1 2 16 DIMM
Channel0/DIMM 1/Branch 1 3 Channel1/DIMM 1/Branch 1 3 configuration
Channel0/DIMM 0/Branch 1 1 Channel1/DIMM 0/Branch 1 2
Channel1/DIMM 1/Branch 0 3 Channel0/DIMM 1/Branch 1 3 Channel1/DIMM 1/Branch 1 3
UltraSPARC Channel0/DIMM 0/Branch 0 1
T2
UltraSPARC
T2
Channel0/DIMM 0/Branch 2 1 UltraSPARC
Channel0/DIMM 1/Branch 2 3 T2


Network Ports and Options

XAUI 0 XAUI 1

Device NET 0 NET 1 NET 2 NET 3 XAUI 0 XAUI 1
Driver e1000g0 e1000g1 e1000g2 e1000g3 nxge0 nxge1
PCI Device PCI 0/0/1/0/2 PCI 0/0/1/0/3 NIU@80/0 NUI@80/1

NOTES
NET 0 and NET 1 share a common Intel Ophir chip, as do NET 2 and NET 3.
XAUI Adapter cards are required to take advantage of the built-in 10GbE networking
There are two variations, Fibre and Copper
Fibre XAUI cards take one of three current 10GbE transceivers
Short Reach(SR), Long Reach(LR) and Long Reach Multimode(LRM)
Copper versions of the XAUI card are expected Post-RR
XAUI Cards consume the same physical space in the chassis as a PCI card, so are mutually
exclusive with PCI cards in the same location

10GbE XAUI Option Cards
• Interface to the outside world for the Onboard 2 x 10GbE interfaces
from the UltraSPARC T2
• Cards each consume a PCI-E slot if installed
• Attach to dedicated XAUI sockets on special risers
• 10Gbe XAUI Card
– SESY7XA1Z
> Requires Fibre Tranceivers
– 10GbE Short Reach(SR)
– SESY7XT1Z
– 10GbE Long Reach (LR)
– SESY7XT2Z
– 10GbE Long Reach multimode (LRM)
– part No. TBD

• Copper connector XAUI versions to follow post-RR

PCI Riser Cards
• Various Riser Cards are used
• Risers should not be moved between slots
• All slots accommodate low Profile PCI-Express cards
• T5120 – 25 Watts per card, 75 Watts Max (3 Slots)
• T5220 – 25 Watts per card, 102 Watts Max (6 Slots)

> Riser Card Mechanical and Electrical widths for T5120 and T5220
T5120 Slot 0 Slot 1 Slot 2
Only Slot x8/XAUI(m) x4(e) x8/XAUI(m) x4(e) x16(m) x8(e)

T5220 Slot 1 Slot 1 Slot 2
Top Slot x8(m) x4(e) x8(m) x4(e) x8(m) x8(e)


Disk Cage
• Disk chassis is a FRU
– 4 screws and disconnect cables
• Contains DVD-ROM/Dual USB module as a
sub-FRU/CRU
– Standard in all default configs
– remove disk 3/7 to access release latch, pre-RR
• Non-DVD module available as X-Option Remove Disk and locate
– Still provides Dual USB ports latch beneath DVD
Press up and pull DVD out

• T5120 will be available initially with only a 4
disk chassis, 8 disk in qualification
> Only uses one or two, of the x4 SAS links
• T5220 will be available initially with only an 8
disk chassis, 16 disk in qualification
> Uses both of the x4 SAS link cables


Disk Cage - T5120
• Chassis accommodates up to 4 or 8 SAS or SATA 2.5”
drives
– No mixing in a chassis
– No chassis changes needed
• Disk Chassis requires only a single cable
Fan Paddle x4 SAS cable
• Connects to Fan Paddle board with press fit connector Board press-
fit connector
and connector

• Mounting ensures fan to disk vibration isolation

Chassis
Screws

Chassis
Screws
Locator Tabs

'T' head
locator bolts

Disk Cage - T5220
• Accommodates up to 8 or 16 SAS or SATA
2.5” drives
• Requires both SAS cables be connected
• Two x4 SAS connector on chassis
• Even 16 disks provides lots of airflow
Paddle Board
press-fit
• Cage and Fan install ensures vibration connector x4 SAS cables
isolation and connectors

Chassis
Screws

Chassis
Screws
Locator Tabs

'T' head
locator bolts

Disk Drives
• Disk drives are all 2.5” form factor
• Chassis support both SAS and SATA (post-RR)
• SAS Drives
– 73GB @ 10K RPM
– 146GB @ 10K RPM
Improved
– 73GB @ 15K RPM (post-RR) Airflow above
and below
disks
– 146GB @ 15K RPM (post-RR)
• SATA Drives
– 200GB @ 5400 RPM (soon after RR)
– 300GB @ 5400 RPM (H1CY08) Disk LED's
Ready to
Remove
– 500GB @ 7200 RPM (end CY08) Fault
Status
• New mounting bracket “Marlin”
> Allows better cooling in denser chassis


Disk Controller Options
• T5x20 has embedded LSI1068E on the motherboard
– JBOD up to 8 disks
– Controls internal disks only
– RAID 0 (striping) and RAID 1(mirroring)
– Hot-plug supported
– Hot-swap in non-RAID configs is done with 'cfgadm'
• Optional Extended RAID functionality cards (available Q4CY07)
> 'Cougar' - SGXPCIESAS-R-INT-Z
– 8 port SAS RAID 0,1,1E,5,5EE,6,10,50,60 (depending on No. disks available)
– Adaptec Card and Firmware with Intel SunRiseLake chipset
– Battery Backed Write Cache (BBWC) of 256MB
– BBWC battery at least 72 hours (much longer when new/fully charged)
– Only for internal disks, replaces cables to route to standard disk chassis
> 'Prometheus' – SGXPCIESAS-R-EXT-Z
– External port version of 'Cougar', for external disk arrays only (no internal ports)

Fans and Cooling
• T5120
> Requires 4 Fan modules

FBD1
FBD1

FBD0
FBD0
FM3
PSU's

FM3

FM3
FM3
> Two cooling groups
– Group 1 – N2 cooling

Disk Chassis
Disk Chassis
FBD1

FBD0
FBD1

FBD0
FM2

FM2
FM2

FM2
– FANBD0/FM1
– FANBD1/FM1

FBD1

FBD0
FBD1

FBD0
FM1
Motherboard

FM1
FM1

FM1
– Group 2 – FB-DIMM cooling

FBD1

FBD0
FBD1

FBD0
– FANBD1/FM0

FM0

FM0
FM0

FM0
– FANBD1/FM2
– Blank Module in FANBD1/FM3 Back Front

• T5220 PSU's

FBD1

FBD0
FBD1

FBD0
FM2

FM2
> Requires 3 Fan Modules

Disk Chassis
Disk Chassis
> Single group for N2 and FB-DIMM

FBD1

FBD0
FBD1

FBD0
FM1

FM1
– FANBD0/FM0 Motherboard
– FANBD0/FM1
– FANBD0/FM2

FBD1

FBD0
FBD1

FBD0
FM0

FM0
FM0

FM0
• Higher disk densities may require Internal andfans Partners Only
Sun
more Approved 33

Fans modules and boards
• Fans are speed controlled by the Service
Processor
• Each fan has an Amber FAULT LED
– if lit the fan module is faulty
• 1RU modules
Fan Fault
– 2 Fans at 15K RPM Max. Amber LED
Lit if fan
• 2RU modules faulty

– 2 Fans at 12K RPM Max
• Monitored through ILOM/IPMI/Web

Vibration
Absorbing
Silicon
Dampers


Power Supplies
• PSU's are high efficiency
• Output 12V (and 3.3v standby) PSU LED's
Ready to
• Most DC-DC conversion done in MBD Fault
Remove

• Shared with Doradi/o, Tucani/a AC
Present

• Integrated Fan
• T5120
> Side-by-Side
> AC 650Watts
> DC 660 Watts option coming post-RR
> DC or AC, no sharing
• T5220
> Vertically Stacked
> 750Watts (may require the 1050Watt Psu for the 16 disk chassis, pending quals)
> No DC option
> BEWARE... Same form factor as SunFire X4200 PSU's ... DO NOT MIX

Rack Mounting Options
• Two Rack mounting options are available..
• Standard Rack Mount (4 post)SESY9RK1Z
> Same as Galaxy servers, T2000, etc..
> Standard configs contain this kit
– ATO : SESY9RK1Z, X-ATO : SESY9RK1Z
• Tool-less Rack Kit (4 post)
> Optional kit, purchased seperately
> Snaps into most 3rd party Racks
> Requires no tools to fit or remove
– ATO : SESY9RK2Z, X-ATO : SESX9RK2Z


Solaris for Sun SPARC Enterprise T5x20
• T5x20 servers will require Solaris10 8/07 (update 4)
• This incorporates several UltraSPARC T1/T2
optimisations for CMT servers in general
• Also includes some specific T5x20 optimisations
> UltraSPARC T2 Crypto optimisations
> nxge driver for 10GbE devices
> UltraSPARC T2 performance counters
> Processor grouping, page coloring and default sizing and
bcopy/memcpy


Solaris FMA for T5x20
• Now aware of CPU Core oflining
• PCIe Fabric diagnostics improved
• Crypto Unit offlining
• Network interface Units (10GbE)
• POST memory CE tolerance improved
• FRU identity info improved in fault reports (part/ser #)
• Diagnostic Engines (DE)
– Existing DE have been extended for T5x20
– New Eversholt DE used for HostBridge, PCIe and NUI diagnostics
• Limitations...
– Can't isolate Uncorrectable Errors to a single DIMM
– PCIe riser cards not in topology
– XAUI card and XFP not in the topology
– SouthBridge chip is notand Approved Partners Only this time (DVD, USB,...)
Sun Internal
diagnosed at 38

Agenda
• Introduction
> Solaris for T5x20
• ILOM 2.0
• LDOMs for T5x20
• Q & A ...

ILOM Features at a Glance
• Hardware Control, Inventory, Monitoring
• Host Configuration and Administration
• SP Configuration and Administration
• Logging, Auditing and Alerting
• Authentication :
– LDAP, RADIUS, Active Directory, Local Users
• Services :
– SSH, HTTP(S), NTP, Syslog, SMTP
• Out-of-Band
– Via Serial port, or dedicated Ethernet port
• In-Band
– Via Host OS, firmware upload, Explorer, Syslog

ILOM – Industry Standard Manageability
• ILOM enables Sun systems to be easily integrated into
third party management infrastructures
• Key functionality
> IPMI 2.0
> SNMP V1, V2c, V3
> SSH 2.0
> LDAP, RADIUS & MS Active Directory authentication
> Browser UI, SMASH DMTF CLP + ALOM style CLI
• NOTE : No RKVMS functionality like Galaxy ILOM
> No Java plugin required
• Browsers tested include
– Mozilla 1.7 on Solaris 10, IE 6.0 on W2K Server,
Firefox 1.0 and 1.5 on W2K server

ILOM – Industry Standard Manageability
• Default user is 'root' with password 'changeme'
> Use for ILOM WEB GUI access and SSH access and
IPMI
• The ALOM 'admin' user is not defined by default, create
this user with an ALOM CLI
• Lost 'root' password and ILOM config is reset using
Motherboard Jumper J6801
• Note... 'sunservice' user shares the password of the
'root' user ONLY FOR SERVICE ACCESS*
• Passwords must be 8 to 16 chars (used to be max 8)
• Max of 10 users allowed (used to be 16)

ILOM Common Tasks
• Setting up an ALOM 'admin' user from the ILOM CLI
> SUNSP00144F4641BD login: root
Password: changeme
Waiting for daemons to initialize...
Daemons ready
Sun(TM) Integrated Lights Out Manager
Version 2.0.4.0
Copyright 2007 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Warning: password is set to factory default.
->create /SP/users/admin role=Administrator cli_mode=alom
Creating user...
Enter new password: ********
Enter new password again: ********
Created /SP/users/admin2
-> exit
SUNSP00144F4641BD login: admin
Password: ********
Daemons ready
Version 2.0.4.0
sc>


ILOM Common Tasks
• Setting up ILOM IP configuration from the ILOM CLI
> SUNSP00144F4641BD login: root
Password: changeme
Daemons ready
Version 2.0.4.0
Warning: password is set to factory default.
-> set /SP/network state=enabled
Set ’state’ to ’enabled’
-> set /SP/network pendingipaddress=10.8.31.163
Set ’pendingipaddress’ to ’10.8.31.163’
-> set /SP/network pendingipdiscovery=static
Set ’pendingipdiscovery’ to ’static’
-> set /SP/network pendingipnetmask=255.255.252.0 Static IP
Set ’pendingipnetmask’ to ’255.255.252.0’ configuration
-> set /SP/network pendingipgateway=10.8.31.248
Set ’pendingipgateway’ to ’10.8.31.248’
-> set /SP/network commitpending=true
Set ’commitpending’ to ’true’
->

-> set/SP/pendingipdiscovery=dhcp
Set ’pendingipdiscovery’ to ’dhcp’
-> set /SP/network commitpending=true
Set ’commitpending’ to ’true’ Dynamic IP
-> (DHCP)
configuration


ILOM IPMI functionality
• IPMI is a widely used management protocol for x86 servers
• ... But its new for SPARC boxes...
• Download the required tools from
> The Solaris Supplemental CD (an older binary package)
> Latest is available at http://ipmitool.sourceforge.net
– Binary packages (usually older versions) are available for Solaris x86 and
SPARC, as well as Linux, Windows, Mac...etc
> There is a Blueprint available from Sun also (mainlyx64 centric, but applies to
SPARC now too)
– http://www.sun.com/blueprints/0107/820-1011.pdf
-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sdr
/MB/V_+1V0_VDD | 1.01 Volts | ok
/MB/V_+3V3_STBY | 3.36 Volts | ok Get the
/MB/V_+3V3_MAIN | 3.36 Volts | ok Sensor Data
/MB/V_+5V0_VCC | 4.97 Volts | ok Repository
/MB/V_+12V0_MAIN | 12.16 Volts | ok
..
..
-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sunoem
sbled set /SYS/LOCATE on Set the System
/SYS/LOCATE | ON Locator LED
-bash-ventus2$ flashing


ILOM Common Tasks
• Logging into the ILOM Web GUI
• Point Web Browser at the IP address
of the ILOM
> Will automatically redirect to a
secure HTTPS port
> Best to disable any Web
Proxies for best performance
> Make sure port 80, 443 are
open on any firewall
• Log in as any defined user, default
Administration user is 'root'


ILOM/ALOM Common Tasks
• A Tour of the ILOM Web GUI • GUI provides ways to set up IP address
configurations for ILOM
• Configure additional ILOM/ALOM users
• Access Sensor Readings
• Control System configurations
• Carry out firmware updates.
• etc....


ALOM 2.0 vs. 1.4 CLI Differences
• Default CLI appearance is ILOM
– User is 'root'
– Must create an 'admin' Administrator user with the ALOM CLI
• Flashupdate in new ALOM expects a TFTP server not an FTP server
• ALOM IP address changes now active only after running ...
– 'setsc netsc_commit true '
– Previously changes were active only after SC reset
– DHCP is enabled by default
• View firmware versions with sc> showhost version
Sun System Firmware 7.0.0_build_13 2007/08/10 23:44

– 'showhost version '
Host flash versions:
Hypervisor 1.5.0.build_13**PROTOTYPE** 2007/08/10 22:54
OBP 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:25

• View FPGA version with
Sun Fire[TM] Huron POST 4.27.0.build_04***PROTOTYPE BUILD***
2007/08/10 13:51

– 'fpga version ' sc> fpga version
FPGA Version: 1.5.0.0
Platform ID
Major revision
: 1
: 5
Minor revision : 0

• Only Admin and Operator roles Lab Debug Version: 0
sc>

are defined Sun Internal and Approved Partners Only 48

ALOM backward Compatibilty CLI
• Uses ALOM-CMT 1.2 for a base feature set
• Includes ALOM-CMT 1.4 features...
> break -d
> Host Watchdog Timer
> Bootmode LDOMs awareness
> Forward SP events to Hot/Solaris syslog via inband
channel


Agenda
• Introduction
> Solaris for T5x20
• ILOM 2.0
• LDOMs for T5x20
• Q & A ...

Logical Domains 1.0.1 Features
Released, aligned with UltraSPARC T2/T5x20
• UltraSPARC T2 Platform Support
> 32 domains at RR, 64 domains following
• I/O domain can reboot while the other domains keep
running
> Dependant domains would wait for IO transactions to complete
• LDoms SNMP MIB
> Modeled after industry standard DMTF CIM (Common Information
Model)
> Running on control domain to enable remote SNMP monitoring
• Domain minimization support
> Minimal/reduced Solaris installation
> Base minimal packages plus optional ones for guest, control, I/O or
service domains

Logical Domains 1.0.1 Requirements
• Solaris 10 08/07 (update 4)
• Systems firmware version
– T5x20/Glendale Firmware 7.0.x = SP FW 2.0.4.0+
– T2000/T1000/T6300/Pelton Firmware 6.5.x
• SUNWldm.v LDOMs Manager Package
– Version 1.0.1
• T5x20 System FPGA
– Version 1.5.0+ (Not a concern for Post-RR systems)


Agenda
• Introduction
> Solaris for T5x20
• ILOM 2.0
• LDOMs for T5x20
• Q & A ...

CRU/FRU List - T5120
• Lots shared with other Jake and Elwood systems
Item Description FRU/CRU
Disk Backplane 4 drive Disk Backplane to support 4 Drives only FRU
Fan Module 1U Dual Fan module, 4 populated in system CRU
Fan Board 1U Fan connector board to accommodate Fan Modules, 2 per system FRU
DVD Assembly DVD Assembly including front USB board FRU
Paddle board 1U Conection between PDB, fan boards and Disk Backplane FRU
PDB 1U Power distribution board, contains master chassis serial number FRU
PSU 1U Power supply (650 Watts) CRU
X8 &XAUI, 1U PCIe Riser PCI Riser assembly for left and centre motherboard slots CRU
x16 1U PCIe Riser PCI Riser for right motherboard slot CRU
Rack mount kit (requires tools) Rack mount kit CRU
Rack Mount Kit (tooless) Rack mount kit, no tools required for installation CRU
CMA Cable management arm CRU
Bus Bars Power Bus Bars between PDB and Motherboard FRU
PDB to MB Ribbon Cable Ribbon cable for control connections between PDB abd Motherboard FRU
Light pipe kit Light pipe assemblies for left and right of disk cage FRU
Motherboard assembly Motherboard assembly, part no varies depending on Core count FRU
DIMMs FBDIMMs (three different part numbers for 1GB, 2GB, 4GB) CRU
XAUI board optical/copper XAUI Adapter card (two different parts for copper/optical) CRU
XFP module XFP GBIC module for use in XAUI cards (three different parts SR, HR, HRMM) CRU
Battery Battery backup for NVRAM of Service Processor CRU
HDD 73GB/146GB SAS Hard Disk drive, different part numbers for capacities CRU
System Configurations PROM System Configuration PROM FRU


CRU/FRU List - T5220
• Lots shared with other Jake and Elwood systems
Item Description FRU/CRU
Disk Backplane 8 drive Disk Backplane to support 8 Drives only FRU
Fan Module 2U Dual Fan module, 3 populated in system CRU
Fan Board 2U Fan connector board to accommodate Fan Modules, 2 per system FRU
DVD Assembly DVD Assembly including front USB board FRU
Paddle board 2U Connection between PDB, fan boards and Disk Backplane FRU
PDB 2U Power distribution board, contains master chassis serial number FRU
PSU 2U Power supply (750 Watts) CRU
X8,X8, XAUI, 2U PCIe Riser PCI Riser assembly for left and centre motherboard slots CRU
X8, x16 2U PCIe Riser PCI Riser for right motherboard slot CRU
Rack mount kit (requires tools) Rack mount kit CRU
Rack Mount Kit (tooless) Rack mount kit, no tools required for installation CRU
CMA Cable management arm CRU
Bus Bars Power Bus Bars between PDB and Motherboard FRU
PDB to MB Ribbon Cable Ribbon cable for control connections between PDB abd Motherboard FRU
Light pipe kit Light pipe assemblies for left and right of disk cage FRU
Motherboard assembly Motherboard assembly, part no varies depending on Core count FRU
DIMMs FBDIMMs (three different part numbers for 1GB, 2GB, 4GB) CRU
XAUI board optical/copper XAUI Adapter card (two different parts for copper/optical) CRU
XFP module XFP GBIC module for use in XAUI cards (three different parts SR, HR, HRMM) CRU
Battery Battery backup for NVRAM of Service Processor CRU
HDD 73GB/146GB SAS Hard Disk drive, different part numbers for capacities CRU
System Configurations PROM System Configuration PROM FRU


Option Parts List

Part Number Option Description Avialable
SECY9PS11Z AC PSU 1RU (650 Watts) RR
SEDY9PS31Z AC PSU 2RU (750 Watts) RR
SECY9PS21Z DC PSU 1RU (660Watts) post-RR
SESY2A1Z 2GB Memory (2x1GB) RR
SESY2B1Z 4GB memory (2x2GB) RR
SESY2C1Z 8GB Memory (2x4GB) RR
SESY3A11Z 73GB, 10KRpm 2.5” SAS Drive RR
SESY3C11Z 146GB, 10KRpm 2.5” SAS Drive RR
- SATA Drives (200, 300,500GB) post-RR
SESY9DV1Z DVD, 8X RW, Slot Load RR
SESY9RK1Z Slide Rail Kit, requires screwdriver (1RU and 2RU) RR
SESY9RK2Z Slide Rail Kit, Tool-less (1RU and 2RU) RR


Option Parts List (cont..)
Part Number Option Description Avialable

SG-PCIE1FC-QF4 Qlogic, 4Gb, PCI-E Single Port (up to 3/6) RR
SG-PCIE2FC-QF4 Qlogic, 4Gb, PCI-E Dual Port (up to 3/6) RR
SG-PCIE1FC-EM4 Emulex, 4Gb, PCI-E Single Port (up to 3/6) RR
SG-PCIE2FC-EM4 Emulex, 4Gb, PCI-E Dual Port (up to 3/6) RR
SG-PCIE2SCSI-U320Z U320 SCSI, PCI-E Dual Port (up to 3/6) RR
SG-PCIE8SAS-E-Z SAS, 8 Port, PCI-E (up to 1) RR
SGXPCIESAS-R-INT-Z SAS, 8 Port, RAID, PCI-E Internal Drive Controller (up to 1) RR+3
SGCPCIESAS-R-EXT-Z SAS, 8 Port, RAID, PCI-E, External Drive Controller (up to 1) RR+3
7280A-2 Northstar Dual port GigE MMF Low profile PCI-E (up to 3/6) RR
7281A-2 Northstar Dual port GigE UTP Low profile PCI-E (up to 3/6) RR
1236A-Z Infiniband HBA, Dual 4x ports, PCI-E (up to 3/6) RR
4447A-X Atlas Quad Port, GigE Copper, PCI-E (up to 3/6) RR
X4240A XVR-300, PCI-E x16/x8 Graphics cards (up to 1) TBD


Agenda
• Introduction
> Solaris for T5x20
• ILOM 2.0
• LDOMs for T5x20
• Q & A ...

Tuning Out of the Box
• Using Solaris 10 08/07...
> Default pages sizes for
– Initdata Segments is 64kB pages
– Private Anon, Heap, Process Stack, Text, Shared Anon segments are
4MB pages
– ISM and DISM use 256MB pages
– New variables introduced to control these sizes if changes are needed
– max_uheap_lpsize,max_ustack_lpsize, max_privmap_lpsize,
max_shm_lpsize, max_uidata_lpsize, max_utext_lpsize,
max_shm_lpsize
> L2 Cache hashing implemented
– helps randomise accesses to L2 caches and ease thrashing
> Stack biasing implemented
– D-Cache is only 4 Way associative... 8 threads
– Running multiple copies of a binary can lead process stacks aligning in
the cache and to hot lines in the cache
– Stack Bias slews the stack variableOnly processes.
Sun Internal and Approved Partners
for 59

Tuning Out of the Box (continued..)
> Processor Groups introduced
– Helps associate CPUs with some shared resources (int/FP pipelines,
Crypto units etc)
– Enables improved multi-level CMT load balancing and affinity policies
> 'bcopy'/'uiomove' Optimizations
– Key kernel operations
– new implementation about 2x faster than previous

• Some optimisations did not make it into S10 8/07
> These will be post RR patches as they become available
> Most fixes are already committed in update 5
> Check patch levels for customers with performance issues
> Also check Firmware versions as this can improve
performance too...

Tuning Recommendations
• Less involved than with T2000
• S10 8/07 is more 'automatic'
– OOB optimised page size selection, thread scheduling, block copy, IP
instances for zones, page coloring etc...
• If tuning is necessary remember...
> Improved pipeline results in Single threaded performance around 20% better
> Floating Point is no longer an issue
> Start with T2000 tuning options
> Use updated trapstat, cpustat,
• LDOMs may be an easier way to scale poorly threaded applications
• Solaris Crypto framework is still being tuned
> Great for big packets, working on improving small packets
• A Blueprint is being prepared (Denis Sheahan)


Tuning Recommendations (cont..)
• Refer to UltraSPARC T1 applications tuning best practices...
> http://www.sun.com/servers/coolthreads/tnb/applications.jsp
• CoolTools are being updated for UltraSPARC T2
> http://www.sun.com/servers/coolthreads/overview/cooltools.jsp
• Sun Studio 12 is UltraSPARC T1 and T2 aware
> Recompiling without code changes should benefit T1/T2 apps
• Processes are being worked on for LDOMS migrations 1.0 to 1.0.1
> LDOMs configs can benefit from a transition to T5x20 platform
> Make sure appropriate LDOMs tuning and sizing is carried out
• Wider Crypto capability, so ensure code is using PKCS#11 framework
> Ensure that the ncp (SSL, RSA, DSA etc) and n2cp (bulk ciphers),
n2rng(random number generator) drivers are used appropriately
• Sun Internal and Approved Partners Only 62

Agenda
• Introduction
> Solaris for T5x20
• ILOM 2.0
• LDOMs for T5x20
• Q & A ...

Tools and Information
• Cool Tools
– http://www.sun.com/servers/coolthreads/overview/cooltools.jsp
• Architecture WhitePaper
– http://www.sun.com/
• SUNWIN Tokens
> Sun Intro : 514416
> Architecture Whitepaper : 512750
> This Technical Presentation: 512749


Reference Information
• T5120 and T5220 (Huron) Engineering TOI
– http://ittdev.east.sun.com/TechTalk2/EngineeringTOIs/Huron_Apr07/
• ILOM/ALOM User Guides
– http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-1160-13.pdf
– http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-7981-10.pdf
• Solaris FMA
– http://fma.eng
– http://lightside.sfbay/n2-fma/docs.php
•


Peter A. Wilson
Senior Technical Marketing Engineer
Sun Microsystems

66

Huron PCI Topology


T5x20 PCI Device Tree
“/pci@0" 0 "px" UltraSPARC T2 PCI bus
“/pci@0/pci@0" 0 "pxb_plx" To First Level PCI-E Switch Chip
“/pci@0/pci@0/pci@1" 1 "pxb_plx" To Second level PCI-E Switch Chip
“/pci@0/pci@0/pci@1/pci@0" 3 "pxb_plx"
“/pci@0/pci@0/pci@1/pci@0/pci@1" 4 "pxb_plx"
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0" 0 "px_pci" TO SouthBridge USB Devices
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0" 0 "ohci"
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,1" 1 "ohci" Standard USB Device manager
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2" 0 "ehci" Enhanced USB Device manger
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/hub@4" 0 "hubd" USB Dual Port HUB
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/hub@4/keyboard@1" 2 "hid" Rear Port 0
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/hub@4/keyboard@2" 0 "hid" Rear Port 1
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@1" 0 "scsa2usb" USB to IDE Converter
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@1/disk@0,0" 0 "sd" DVD ROM Drive
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@1/disk@0,1" 3 "sd"
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@2" 1 "scsa2usb"
“/pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@2/disk@0,0" 2 "sd" Front USB Port 0
“'pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@3" 2 "scsa2usb"
“”pci@0/pci@0/pci@1/pci@0/pci@1/pci@0/usb@0,2/storage@3/disk@0,0" 4 "sd" Front USB Port 1
“/pci@0/pci@0/pci@1/pci@0/pci@2" 5 "pxb_plx" To Dual GbE Controller 0
“/pci@0/pci@0/pci@1/pci@0/pci@2/network@0" 0 "e1000g" NIC 0
"/pci@0/pci@0/pci@1/pci@0/pci@2/network@0,1" 1 "e1000g" NIC 1
"/pci@0/pci@0/pci@1/pci@0/pci@3" 6 "pxb_plx" To Dual GbE Controller 1
"/pci@0/pci@0/pci@1/pci@0/pci@3/network@0" 2 "e1000g" NIC 2
"/pci@0/pci@0/pci@1/pci@0/pci@3/network@0,1" 3 "e1000g" NIC 3
"/pci@0/pci@0/pci@2" 2 "pxb_plx" To SAS Controller
"/pci@0/pci@0/pci@2/scsi@0" 0 "mpt" SAS Controller
"/pci@0/pci@0/pci@2/scsi@0/sd@0,0" 1 "sd" SAS Disk Device
"/pci@0/pci@0/pci@8" 7 "pxb_plx" To Second Level PCI-E Switch Chip
"/pci@0/pci@0/pci@8/pci@0" 9 "pxb_plx"
"/pci@0/pci@0/pci@8/pci@0/pci@1" 10 "pxb_plx" PCI-E Slot 1/2RU
"/pci@0/pci@0/pci@8/pci@0/pci@a" 14 "pxb_plx" PCI-E Slot 0/1RU
"/pci@0/pci@0/pci@9" 8 "pxb_plx" PCI-E Slot 2
"/niu@80" 0 "niumx" Network Interface Unit
"/niu@80/network@0" 0 "nxge" 10Gb_NIC 0
"/niu@80/network@1" 1 "nxge" Sun Internal and Approved Partners Only 10Gb_NIC 1 68

Sun sparc enterprise t5120 and t5220 servers technical presentation

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Sun sparc enterprise t5120 and t5220 servers technical presentation

Semelhante a Sun sparc enterprise t5120 and t5220 servers technical presentation (20)

Mais de xKinAnx

Mais de xKinAnx (20)

Último

Último (20)

Sun sparc enterprise t5120 and t5220 servers technical presentation