Source: http://hdfeos.org/workshops/ws04/presentations/Folk/Folk.ppt

1. HDF Update
Mike Folk
HDF-EOS Workshop IV
Sept. 19-21, 2000
University of Illinois at Urbana-Champaign
-1-
HDF

2. Topics
• What is HDF?
• HDF community
• HDF4
– past and future work
• HDF5
–
–
–
–
HDF5-HDF5 interoperability
Activities and growth
Tools and utilities
Focus for 2001
University of Illinois at Urbana-Champaign
-2-
HDF

3. NCSA HDF Mission
To develop, promote, deploy, and support
open and free technologies that facilitate
scientific data storage, exchange, access,
analysis and discovery.
University of Illinois at Urbana-Champaign
-3-
HDF

4. What is HDF?
•
•
•
•
•
•
Format and software for scientific data
Stores images, arrays, tables, etc.
Emphasis on storage and I/O efficiency
Free and commercial software support
Emphasis on standards
Users from many engineering & scientific fields
University of Illinois at Urbana-Champaign
-4-
HDF

5. HDF People
• NCSA
– QA, maintanance & support
– Library development & performance
– Tools development
• EOS & ASCI
– Users, applications and tools
– Performance
• Volunteers from all over
– Users, applications & tools
University of Illinois at Urbana-Champaign
-5-
HDF

6. Who is supporting HDF?
• NASA/ESDIS
– Earth science applications, instrument data
– All aspects of data management
• DOE/ASCI (Accelerated Strategic Computing Init.)
– Simulations on massively parallel machines
– Emphasis on parallel I/O performance, functionality
• NCSA
– Grid, Vis, other R&D, user support
• Others
– Applications, support, some R&D
University of Illinois at Urbana-Champaign
-6-
HDF

7. HDF4
•
•
•
•
User support, tools, maintenance, QA
JHV (Java viewer) - two update releases
HDF4 dumper (hdp) - many improvements
Next HDF4 Release (Oct. 2000)
–
–
–
–
Bug fixes
Chunking for General Raster (GR) images
Unlimited dimensions for SDS
GIF <--> HDF4 conversion tools
University of Illinois at Urbana-Champaign
-7-
HDF

8. HDF4 Archiving
• New File Format and Specification Manual
–
–
–
–
Facilitate long term maintenance of HDF4
Necessary for bug fixing, adding new features
Decreases learning curve for new developers
Available soon in
• PDF, HTML and PS formats
University of Illinois at Urbana-Champaign
-8-
HDF

9. HDF5
University of Illinois at Urbana-Champaign
-9-
HDF

10. The growth of HDF5
• Users in 27 countries
• Users include
–
–
–
–
Government agencies
National labs
Companies
Universities
University of Illinois at Urbana-Champaign
- 10 -
HDF

11. The growth of HDF5
• Scientific fields
– Astronomy, astrophysics, aerospace engineering
– Geophysics, remote sensing, meteorology,
– Oceanography, environmental Science
– Informational Science
– Medical Research (brain, cancer, biotech)
– Product model data
University of Illinois at Urbana-Champaign
- 11 -
HDF

12. Facilitating interoperability
between HDF4 and HDF5
• HDF4 to HDF5 mapping specification
– “Mapping HDF4 Objects to HDF5 Objects”
• http://hdf.ncsa.uiuc.edu/HDF5/papers
– Rules for mapping high level HDF4 objects to HDF5
• How to describe HDF4 objects in HDF5
• How to interpret HDF5 objects as HDF4 objects
• HDF4-to-HDF5 conversion software (later)
• Space Research, Inc. Explorer 1.1
– reads both HDF4 and HDF5
University of Illinois at Urbana-Champaign
- 12 -
HDF

13. HDF5 Activities in 2000
•
•
•
•
•
•
HDF5 1.2.2 library release
Fortran 90 & C++ API
HDF5 Abstract Data Model
XML Document Type Definition (DTD)
HDF5 tools
Support for users and application
developers
• HDF5 Tutorial
• Searchable, printable documentation
University of Illinois at Urbana-Champaign
- 13 -
HDF

14. Tools & Utilities
• NCSA
–
–
–
–
Java wrapper for HDF5
H5View: Java browser/Editor for HDF5
H5gen: XML-to-HDF5 file generator (Java)
H5dump & H5ls
• Others
–
–
–
–
VisAD data adapter for HDF5 (Java toolkit)
HDF Inspector/Explorer
Open Data Explorer (IBM)
Ensight
University of Illinois at Urbana-Champaign
- 15 -
HDF

15. H5View
University of Illinois at Urbana-Champaign
- 16 -
HDF

16. H5View
• Java-based tool for browsing and editing
• Display structure of file
• Display content of objects
• Create and delete objects
• Modify values of objects and attributes
University of Illinois at Urbana-Champaign
- 17 -
HDF

17. HDF5 XML DTD
• A flexible standard language that can
describe an HDF5 file in precise detail
– Datasets, dataspaces, datatypes
– Groups and links, structure of the file
– Values of the attributes and data
•
http://hdf.ncsa.uiuc.edu/HDF5/XML
University of Illinois at Urbana-Champaign
- 18 -
HDF

18. H5Gen: XML HDF5
• Reads an XML description of an HDF5 file
• Generates the corresponding HDF5 file
• Validates XML description vs. HDF5 DTD
University of Illinois at Urbana-Champaign
- 19 -
HDF

19. Some uses of XML
Archive
XML
documentation
of metadata
& structure
XML to HDF5 (H5gen)
HDF5 to XML
Generate, validate,
reconstruct HDF5 files
Description
in XML using
HDF5 DTD
XML
to
HTML
Catalog
records
in XML
View
HDF5 files
using a web
browser
Data
location
services
XML to Java
Java to XML
HDF5
file
- 20 -
Java
viewers,editors,
University of Illinois at Urbana-Champaign
other tools
HDF

20. Focus areas for 2001
•
•
•
•
•
•
Support Terra & Aqua
Get ready for Aura
Support HDF-EOS 3
Enhance HDF5View
Refine XML DTD & design tools around it
HDF5 converters
– HDF5 XML
– HDF4-to-HDF5
– Others (e.g. GIF
HDF5)
University of Illinois at Urbana-Champaign
- 21 -
HDF

21. Focus areas for 2001
•
•
•
•
•
•
•
•
Expand list of applications and users
Facilitate access to other tools & software
Get vendors on board
Performance testing and tuning
Extend API with ease-of-use functions
Clusters and other new environments
Implement a thread-safe version
HDF5 advanced tutorial
University of Illinois at Urbana-Champaign
- 22 -
HDF

22. Thank you!
HDF • HDF website
– http://hdf.ncsa.uiuc.edu/
5 • HDF5 Information Center
– http://hdf.ncsa.uiuc.edu/HDF5/
• HDF Helpdesk
– hdfhelp@ncsa.uiuc.edu
• HDF users mailing list
– hdfnews@ncsa.uiuc.edu
University of Illinois at Urbana-Champaign
- 23 -
HDF