1. HDF-EOS Overview and Status
Larry Klein
Emergent Information Technologies, Inc.
September 19, 2000
larry@eos.hitc.com
2. What is HDF-EOS?
• An extension to HDF, built with HDF objects.
• A standard for data structures for remote
sensing data and in-situ measurements with:
– Tightly coupled geolocation information
– Access based on geolocation
• Incorporates ECS metadata attributes.
• Used for many EOS standard products.
2
3. HDF-EOS File Contents
• Global HDF attributes
– Core Metadata - searchable attributes, copied in
database
– Archive Metadata - non-searchable attributes
• Any number of HDF objects specifying
HDF-EOS objects (Grid, Point, Swath)
– Structural metadata - links geolocation to
science data
• Additional HDF objects.
3
4. HDF-EOS Grid Structure
• Usage - Data which is organized by regular
geographic spacing, specified by projection
parameters.
• Structure
– Any number of 2-D to 8-D data arrays per
structure, one per data type (e.g. temperature)
– Geolocation information contained in projection
formula, coupled by structural metadata.
– Any number of Grid structures per file allowed.
4
6. Grid Data: 1 Km. Digital Elevation Model
South Pole
6
7. HDF-EOS Swath Structure
• Usage - Data which is organized by time,
monotonic geolocation, track parameter.
Spacing can be irregular.
• Structure
– Geolocation information stored explictly in
Geolocation Field (2-D array)
– Data stored in 2-D or 3-D arrays
– Time stored in 1-D or 2-D array, TAI is standard.
– Geolocation/science data connected by structural
7
metadata
9. HDF-EOS Point Structure
• Usage - Data which is specified temporally
and/or spatially, but with no particular
organization.
• Structure
– Tables (Vdatas) used to store science data at a
particular Lat/Long/Height.
– Up to eight levels of data allowed. Structural
metadata specifies relationship between levels.
9
11. HDF-EOS Interface Schematic
User Application
HDF-EOS Interface
Grid
Point
ECS Metadata Interface
Swath
Low Level HDF
File I/O
11
12. User Interface
•
•
•
•
•
•
•
Access - Open, close objects/files
Definition - Set key features of data sets
Basic I/O - Read write data
Index I/O - Link tables (Point only)
Inquiry - Return information about data
Subset - Read data from specific region
Metadata access (SDP toolkit)
12
13. Current Archive Holdings
• GSFC DAAC: MODIS L1, Atmos./Ocean
– 50 TB, 105,000 granules, 300 GB/day
– will be ~1.3 TB / day with addition of MODIS
L2 and EOS Aqua data.
• EDC DAAC: Landsat/ASTER/MODIS Land
– 20 TB, 106,000 granules, 274 GB/day
• Langely DAAC: MISR
– 9 TB, 39,000 granules, 88 GB/day
13
14. Availability
• HDF library - ftp hdf.ncsa.uiuc.edu
• HDF-EOS V2.6 library (HDF4 based)
– http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html
• SDP Toolkit library - same location as HDF-EOS
• Platform Support
– DEC, HP, IBM, Sun, SGI, Windows, Linux (soon)
• Language Support - C, C++, F77, F90
• HDF4-based library will be supported indefinitely
14
15. HDF5 - Based Development
• NCSA has introduced HDF5
– A complete rewrite of HDF4, a new user
interface.
– Simplified data model
– Breaks 2 GB file size limit
– Potential for parallel I/O, multi-threading
15
16. HDF5 - Based Development
HDF-EOS 3.0
• Development overview
– Maintain current HDF-EOS interface to
maximum extent
– Allow all HDF5 features through interface
– Build HDF-EOS (4 to 5) conversion function
– Data production executable can be built with
either or both HDF4 and HDF5 - based library
– Support heterogeneous ECS archive (future)
16
17. Design Schematic
User Application
HDF-EOS Interface
Grid, Point, Swath
ECS Metadata
Interface
Common Functions
Definition, Access, I/O, Inquiry, Subset
*Implemented
if required
HDF4
Interface*
HDF5
Interface
Combined
HDF4/5
Interface*
17
Build
Options
18. Applications
• H5EOSView - EOSView for HDF5
• Java Browser - Browser for both HDF4 and
5 based granules
• HDF-EOS 4 to 5 converter, HDF-EOS 2.x
to HDF-EOS 3.x
18
19. Schedule
• Jan., 2000 - Beta Version of library
– H5EOSView, Sun, SGI, C support
• Oct. 2000 - Beta2 Version
– Add F77, F90, DEC support
– Local Attributes, Profile Interface,
Chunking/Tiling
– HDF-EOS 4 to 5 converter and Java Browser
19
20. Schedule (cont.)
• Jan. 2001 - SCF version
– add HP, IBM, Windows, Linux (maybe a few
months later), C++ support
• DAAC version (ECS Drop 6B) - Summer
2001.
– Archives can contain both HDF4 and HDF5 based granules.
– ECS subsystems don’t crack granules, but
regression testing needed.
– 2 GB size limit is an issue.
20