1. Distributed
Interactive
Computing
Environment
http://www.arl.hpc.mil/SciVis/dice

2. Challenge : Stovepiped Applications
Platforms
HPC Codes
CFD Code 1
CSM Code 1
CCM Code 2
CCM Code 1
Preprocessing
Runtime Analysis
Tools
Postprocessing

3. Additional Challenges
• Code Setup can be Difficult
• Enormous Amounts of Data
• Heterogeneous Environment
• Security

4. Distributed Interactive Computing Environment
Graphical User
Interface and Runtime
Visualization
Common Distributed
Data Model and Format
High Performance
Computing Resource
User’s Workstation

5. Fully Integrated FMD GUI

6. Fully Integrated ZnsFlow GUI

7. Fully Integrated CTH GUI

8. What is DICE ?
A Distributed Computing Environment Composed of Configurable
Mega-Components
• Heterogeneous DSM *
• Data Organization
• Graphical User Interface
• Visualization
User-Ready Computing Environments Customized for Specific
Technology Areas

9. Network Distributed Global Memory
Clients
VectorPut
VectorGet
Put
Get
0
Barrier
Semaphore
10M
30M
50M
Servers
Process Local
Address Space
System Shared Memory
Disk File
Physical Data Storage

10. Light Data
Heavy Data
• 3D Structured Mesh
• X, Y, Z Values
• 300 x 200 x 300
• Material Volume Fractions
• Number of Materials
• Cell Temperatures
• Material Names
• Total Residual Mass

11. User Application
Scripting and Graphical User Interface Tools
Visual Generators
Plots
Planes
Surfaces
High Performance
Computing Code
DICE Object Directory
Object Oriented Support for Data Model, XML, ...
Hierarchical Data Format ( HDF5 )
Data Format : Number Type, Data Space ...
Network Distributed Global Memory ( NDGM )
with GSSAPI
Shared Memory Facility
TCP/IP Sockets
Operating System
Distributed Resources
Visualization
Toolkit (vtk)
OpenGL
Mesa
File I/O
Process Control

12. DICE Data Model and Format
Convenience Layer
Structured Data
Unstructured Data
Groups
DICE Object Directory
Object Oriented Support for Data Model, XML, ...
Hierarchical Data Format ( HDF5 )
Data Format : Number Type, Data Space ...
FileSystem Serial
and Parallel
Global Access to
Secondary Storage
( Globus - GASS )
Distributed Buffer of Bytes
Network Distributed
Global Memory with
GSSAPI
Other ...

13. Hierarchical Data Format
Version 5
Group
Attribute
Group
Group
Attribute DataSet
• Host Independent Data
Format
• Access via API only
• Physical Storage thru
Drivers
• Includes Data
Compression
• Includes SubSelections
Not
Allowed
Group
DataSet
Group
DataSet
Attribute
Attribute
Attribute
DataSet

14. eXtensible Markup Language
• Similar to HTML Format
• Human Readable
• Pervasive on the WEB
• Strict Hierarchy with No Inheritance
< Parent Name=“Top”>
• Many Parsing Tools Already Exist
< Child Name=“Left”>
Character Data for Left
< /Child >
Parent
<Child Name=“Right” >
Character Data for Right
< /Child >
Child
Child
Character Data
for Left
Character Data
for Right
< /Parent >
Parse
XML
Document Object Model
In-Memory Data Structures
Serialize

15. N
3
5
3
1
4
1
2
N
3
Polyline
4
4
5
3
4
N
2
Polyvertex
4
1
2
2
1
Triangle
Quadrilateral
3
1
2
Polygon
6
5
5
3
4
1
3
1
Tetrahedron 2
Pyramid
8
2
3
4
2
Wedge
1
7
5
6
3
4
1
Hexahedron
2
j
i
j
k
2DSMesh
DICE Object Directory Base Topologies
i
3DSMesh

16. <Topology Type=“Tetrahedra” NumElements=“3”>
Light Data
</Topology>
<Geometry>
<Topology Type=“3DSmesh”
<Vertices>
Dimensions=“10 20 30”>
<Index Location=“XML” Type=“List”>
</Topology>
1000 1001 21001 21000 ……
<Geometry>
</Index>
<Points Type=“XYZ” Location=“HDF” >
<Points Type=“XYZ” Location=“HDF” >
NDGM:File1.h5:/Grid1/Geometry/MyPoints
NDGM:File1.h5:/Grid1/Geometry/MyPoints
<Region Type=“Slab” Location=“XML” >
1 2 10
2 3 20
3 4 30
</Region>
</Points>
</Vertices>
</Geometry>
</Points>
</Geometry>
100 x 200 x 300
HPC
Code
HDF5
NDGM
GASS
MPI-I/O
....

17. Complexity of Model Defined in Light Data
Domain
Domain
Grid
XML
Grid
Attribute
Topology
100 Hexahedra
HPC
Code
Geometry
XYZPoints.h5
HDF5
File
Attribute
NDGM:Pressure.h5
NDGM:Temperature.h5
HDF5
NDGM
HPC Code I/O is Natural and Efficient
GASS

18. XML Contains Parameters for Variable Substitution
Domain
Domain
Grid
XML
Grid
Parameter TimeStep = 0 to 99
Topology
100 Hexahedra
HPC
Code
Geometry
XYZPoints.h5
HDF5
Attribute
NDGM:&TimeStep;/Pressure.h5
HDF5 for TimeStep 0
HDF5 for TimeStep 1
HDF5 for TimeStep 2

19. Parallel / Distributed Entities
Post-Processing Tools
Pre-Processing Tools
HPC Codes
Runtime Tools
Convenience Layer
eXtensible Markup Language ( XML )
Hierarchical Data Format Verision 5 ( HDF5 )
Distributed Data Hub

20. DICE Data Reader for EnSight
Data Reader uses
DICE Object
Directory API
EnSight
DICE Object Directory

21. DICE Visualization using vtk
HDF5
File
HDF5
NDGM
Visualization
“Subnet”
DICE Object
Directory
GASS
Source
Filter
Filter
Filter
Polygons

22. Adding Performance Friendly Functionality
System Programming Language
C C++ Fortran77 Fortran90 etc.
Compiled Object
SWIG
Tcl Wrapper
Perl Wrapper
Python Wrapper
Java Wrapper
}
New Scripting Command
via
Dynamically Loadable Library

23. C++
DiceFloat64Array
MyData, NewData;
DiceInt64Array
MyIndex;
MyData.SetNumberOfElements( 10000 );
Allocate 10,000 Doubles
MyData.Generate(0.0, .9999 );
Set Individual Values
MyIndex.SetNumberOfElements( 100 );
MyIndex.Generate( 0, 99 );
MyIndex *= 2;
Array Operation +, -, *, /
NewData = MyData[ MyIndex ];
Array Indexing
Scripting ( Tcl, Perl, etc. )
DiceHDF H5Object
Object Oriented Scripting Interface
H5Object Open NDGM:FileName.h5:/DataSetName
Open HDF5 File, Query Number Type and Shape
DiceMemory MemoryObject
MemoryObject SetArray [ DiceFloat32Array ]
Request Automatic Conversion to Float32
H5Object Read [ MemoryObject cget -this ]
MemoryObject has Number Type and Shape
set NewData [ MemoryObject GetArray ]
DiceExpr $NewData[ where( $NewData > .5 ) ] = .5
Clip Values to .5

24. DICE Web Interface
Under Construction
DICE on HPC Resource
using Off Screen Rendering
SSL
Kerberos kinit
• Every URL Request is Intercepted to Verify Session Id
• DICE Runs as a User Process
• HTML FORMS result in Tcl Procedure Invocation
• Graphics Updated via Server-Push ( Netscape only ) or Client-Pull
• Intended for Limited Interfaces from Remote ( Slow Connection ) or Graphically Challenged
Client Machines.

25. MPI Finite Element
Visualization
Data Mining
Steering
Coupling
HDF5
NDGM
MPI Finite Volume

26. “Gently Coupling” Codes from Separate Disciplines
Distributed Data Hub
Finite Volume
Structural Mechanics
Global Monitor
Small Scale Analysis
Injection Flow
Analysis
Runtime
Visualization

27. COTS, GOTS and
“Semi-Automatic”
Visualization
DICE and the HPC Environment
X / OpenGL
DICE User
Interface
XML and Http
( SSL + Kerberos )
DICE Server
( User Process )
DICE Object
Directory
XML, HDF5 I/O
One or More
Cooperating HPC
Codes