1. Current Status of HDF-EOS,
Maintenance, Developments,
and Tools
Abe Taaheri, Raytheon IIS
HDF & HDF-EOS Workshop XI
November 2007
Page 1

2. TOOLKIT / HDF-EOS Support
•
•
Maintenance support is funded on EMD Evolution
through May 2008
One year extension until May 2009
•
Current staff
- One full-time developer for maintaining:
 SDP/MTD Toolkit
 HDF-EOS2 & 5
 HDF-EOS plug-in for HDFView
 HDF-EOS2 to GeoTIFF Converter (HEG)
 Other Toolkit/HDF-EOS related software
 Toolkit/HDF-EOS/HEG helpdesk
- One half-time developer for HEG maintenance
Page 2

3. ECS Support of HDF-EOS
• Both HDF4 and HDF5 based flavors of HDF-EOS are part
of ECS baseline.
• OS’s: Solaris (9, 10), Irix6.5 (64, n32), Windows XP,
Linux (including 64-bit Opteron and Itanium),
Mac OS X (MTD Toolkit, HDF-EOS2 & 5)
(Dropped support for Solaris 8, HP, HP11, Dec, IBM for next release)
• adding Mac Intel support for 2007 release and full Cygwin
support for 2008
• Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++
Page 3

4. Downloadable Software and
Applications
• SDP/MTD Toolkit (Science Data Processing Tools)
• HDF-EOS2 (HDF4 based)
• HDF-EOS5 (HDF5 based)
• HE5View* (HDF5 - based browser)
• EOSView* (HDF4 - based browser)
• HDFView, Java-based browser (HDF4, HDF5, HDFEOS 2 and 5 access)
• heconvert (converts HDF4 - based
Grid/Point/Swath structures to HDF-EOS 5
equivalents)
• HEG (GeoTIFF conversion, Spatial subsetting, Field
Subsetting, reprojection, stitching, sub-sampling,
etc.)
* Dropped support for OS updates on July 2006
Page 4

5. Software Releases
Toolkit/HDF-EOS:
• Last release – March 2006
TOOLKIT 5.2.14, HDF-EOS 2.14, HDF-EOS5.1.10
with HDF4.2r1, HDF5-1.6.5-post5, Szip2.0
• Next release – November 2007
TOOLKIT 5.2.15, HDF-EOS 2.15, HDF-EOS5.1.11
with new releases of HDF4, HDF5, and SZIP
HDF-EOS Plug-in for HDFView:
• Last release – May 2006
– Version 2.3
• Next release – Dec. 2007
– HDFView2.4 was released October 31 by The HDF
Group
Page 5

6. Software Releases (Cont.)
HEG (HDF-EOS2 to GeoTIFF Converter):
• Last release – July 2007
– Version 2.8
• Next release – May 2008
– Version 2.9
Page 6

7. SDP Toolkit
•
Science Data Production (SDP) Toolkit is a collection of
tools used by
- Science software developers who produce
code to process instrument data.
- HDF-EOS users who need to
• Access metadata in HDF-EOS files
• Perform Time/Date conversion
• Access Digital Elevation Model Data
• Access Ancillary Data, such as Digital Chart
of the World database (DCW), Olson World
Ecosystem files, DEM data files, etc.
Page 7

8. SDP Toolkit (cont.)
•
Tools provided in (SDP, MTD) Toolkit:
- AA
(Ancillary Data Access)
- CBP
(CELESTIAL Body Position)
- CSC
(Coordinate System Conversion)
- CUC
(Constant and Unit Conversions)
- DEM
(Digital Elevation Model Access)
- EPH
(Ephemeris Data Access)
- GCT
(Geo Coordinate Transformation)
- IO
(Input Output (File I/O))
- MEM
(Memory Management)
- MET
(Metadata Access)
- PC
(Process Control)
- SMF
(Status Message File (Error/Status))
- TD
(Time and Date Conversion)
Page 8

9. What is HDF-EOS ?
• HDF-EOS is an extension to HDF which
standardizes storage of and access to common
Earth Sciences data structures
–By providing a single interface to data structures
common to earth science
–By providing a container for EOS inventory,
archive and product specific metadata
–By establishing a consistent relationship
between geolocation and science data.
Page 9

10. HDF-EOS 2
• HDF4 - based, storage format for EOS standard
products.
• Used operationally by MODIS, MISR, ASTER,
Landsat, AIRS and other EOS instruments
• Support for Grid/Point/Swath structures
• HDF4 and HDF-EOS 2 will be supported by
THG/NASA for the fore-seeable future.
Page 10

11. HDF-EOS 5
• Based on HDF5, a complete rewrite of HDF-EOS2
with a different user interface.
–First released in 2000.
–Format used by EOS Aura instruments
• Designed to ‘resemble’ HDF-EOS 2 to the
maximum extent possible.
–Supports same data structures
–Added prefix ‘HE5_’ to HDF-EOS 2 functions.
Page 11

12. HDFView
•
•
•
•
•
•
HDFView: A Java-based browser from THG for HDF4
and HDF5 – based data.
HDF-EOS plug-in extends HDFView to browse HDFEOS data in both versions.
The extension is a plug-in module and does not
change THG HDFView functionality. All copy, save,
editing, image display, etc. for HDF4 and 5 objects
remains.
Identical look and feel when displaying HDF and
HDF-EOS objects.
Users can access a complete breakdown of HDFEOS objects. e.g. grid information, projection info.,
dimensions, datafields, and attributes are displayed.
Current Support: SUN, SGI, LINUX, Mac, WINDOWS
platforms
Page 12

13. Swath, Grid, and Point Objects
Page 13

14. AMSR-E and OMI Point objects
Page 14

15. AMSR-E Swath
New Features in Next Release Version 2.4
Page 15

16. HDF-EOS to GeoTIFF Converter (HEG)
HEG: Utility that converts EOSDIS data from
HDF-EOS format to common Geographical
Information System (GIS)-compatible formats.
• Two versions available:
– A downloadable desktop version
– Access through NASA archive online storage (Data Pools)
– Implemented also in prototypes for Stand-alone Data Pool,
OGC (Open Geospatial Consortium) for Web Services
Chaining, and DOWS (Deploy OGC Web Services) on the
DAACs
Page 16

17. HEG Functionality
• Conversion:
– HDF-EOS (Swath or Grid ) to a single-band or multi-
•
•
•
•
band GeoTIFF’s
– HDF-EOS Swath to HDF-EOS Grid and Swath
– HDF-EOS Swath or Grid to generic Binary (with
metadata file)
Subsetting (spatial, field, band)
Stitching (mosaicing) + subsetting + Reprojecting
Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN,
Albers)
Subsampling (Subsample stacks)
• Running from Command Line (batch jobs)
or with Java-based GUI
Page 17

18. HEG Functionality
• Metadata preservation/creation.
• Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E
products on TERRA and AQUA (>110 products),
• Allows file selection from users local storage
• Operable on Sun, SGI, Win, Linux, MAC
• Integrated into ECS Data Pool
– Reduces the transfer time of HDF-EOS data sets (if subset is requested)
– Provides the end-user with the exact file required by their application
• User Interface
– Portable, written in Java.
– Not dependent on COTS (eg. IDL).
Page 18

19. Mt. Etna Eruption: MISR L1B
Stitched Blocks
Page 19

20. 2004 Hurricane Francis, MODIS
MOD021KM, Multi-Band GeoTIFF, 3 bands:
EV_1KM_RefSB, Bands 1,2,5
Page 20

21. AST_L1B, 2003 California Fires
Geo. projection, Multi-Band GeoTIFF,
(3 bands: Visible Near Infrared (VNIR))
….. and Subsetted
Two Swaths Stitched …..
Page 21

22. Major Development/Fixes
Next Release: November 2007
 SZIP Insatallation: Default decoder, Choice for encoder
 Choice for configuring HDF4 with/without SZIP
 Choice for disabling HDF4’s internal netCDF
 Choice for installing Toolkit/HDF-EOS in 32 or 64-bit
modes in 64-bit Linux platforms
 Support for new releases of HDF4/HDF5/SZIP
(HDF4.2r2, HDF5-1.6.6, SZIP-2.1)
 Installation fix for a problem caused by limits.h in some
Linux platforms
 Auto Configuration capability (like HDF) added for both
HDF-EOS2 and HDF-EOS5
 Support for Mac Intel platform. Drop of support for
Sun5.8, Dec, HP, HP 11, IBM
 Fixes for Lat/Lon to pixel conversions in DEM tools for
Linux to get results similar to those in Unix platforms Page 22

23. Major Development/Fixes (cont.)
HDF-EOS Plug-in for HDFView:
Current Support:
SUN, SGI, LINUX, WINDOWS,
and MAC platforms
Last Release: Version 2.3
– Display Point Objects
– Display pixel's lat/lon for swath and grid images
– Scientific notation in Table data
Next Release: Version 2.4
– Auto installation of plug-in into HDFView running selfextracting archives.
– Display image for selected pixels in the viewed tables
– Display Lat/Lon for a selected pixel in the data field tables
– Mac OSX fixes for one-button mice
– Better display of Plug-in User’s Guide
– New icons for activating Lat/Lon display for Swath & Grid
Page 23

24. Major Development/Fixes (cont.)
HEG: Last Release (Version 2.8)
– Added Support for:
MOD02SSH, MYD02SSH, MOD14A1.v005, MYD14A1.005
MOD29P1D.V005, MOD08_M3.v005, MOD11_L2.v005
AMSR-E Snow and Ice, AMSR-E Ocean
– MISR Bug fixes & enhancements:
 Added “No-reprojection” for ouput in SOM projection
 Added “Nearest Neighbour” for sampling type
 Fixed problem with missing metadata for HDF-EOS outputs
– Added Albers Projection
– New "Subsetter" functionality for Parameters (aka Fields).
This results in:
• Output to HDF-EOS SWATH from input HDF-EOS SWATH
• No alteration of input data because of projection operation.24
Page

25. HDF-EOS/Toolkit
Future Plans:
•
•
•
•
Auto Configuration for TOOLKIT
Full Support for Cygwin platform
Support for HDF5-1.8
Performance improvement, if possible, for reading HDF
format ephemeris/attitude files in Toolkit
• Performance improvement in HDF-EOS5 by redesigning some
functions
• More functionalities in HDF-EOS Plug-in for HDFView:
Format conversion: HDF-EOS2 -> HDF-EOS5
Improved Help Window
Ability to modify input HDF-EOS2 granules.
Ability to cut/paste objects, modify/delete attributes.
Ability to create new grids/swaths
Page 25

26. Availability
• Access to libraries, applications, Documentation:
TOOLKIT:
http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html
HDFView:
http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/
HDFView_hdfeos_plugin.html
HEG:
http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html
• Email
– Abe_Taaheri@raytheon.com
– Landover_PGSTLKIT@raytheon.com
Page 26

27. Page 27

28. Major Development/Fixes
Last Release
HDF-EOS5:
 Added a routine to return list of aliases for a field
 Used HDF5’s H5T_C_S1 datatype for string type
 Added Fortran wrapper for SWwritedatameta() routine
 Replaced symbol “H5E_NONE_MAJOR” with other symbols
for better description of errors
 Ported to Solaris 9 and Solaris 10
 Fixed POINT object for:
linkage between levels
-
updating levels
writing Forward and Backward Linkage data
char field inquiry
Page 28

29. Major Development/Fixes (cont.)
Last Release
 Fixed subsetting problem in HDF-EOS5 Grid
(correction to conversion of DMS degree to Decimal
degree).
 Set LD_LIBRARY_PATH for SZIP/HDF5 shared libraries
in installation and environment setting scripts
TOOLKIT/HDF-EOS2:
 Ported to Solaris 9 and Solaris 10
 Gdapi.c changes for adding more significant digits after
the decimal point to projection parameters if needed.
 chkeph utility fixes for reading binary ephemeris files
(files with different endiannes than the machine)
 SGI 7.4.2 compiler support
 orbsim fixes for Linux to create files similar to those in
UNIX platforms (Random number generation fixes)
Page 29