ESDIS Status (2002)

ESDIS Status

Richard Ullman
ESDIS Project
Richard.E.Ullman@nasa.gov

05 December, 2002

HDF & HDF-EOS Workhop VI

1

Outline
• ESDIS General Status
• HDF-EOS Plans
• Website http://hdfeos.gsfc.nasa.gov

05 December, 2002 HDF & HDF-EOS Workhop VI
2

ESDIS Science Data Services Today
• EOSDIS provides support for high data volumes from Aqua, Terra,
and Landsat 7, and continues to support QuikSCAT, ACRIMSat,
SAGE-III, JASON and pre-EOS-era data including TRMM, UARS,
TOPEX/Poseidon, RADARSat, and others.
– EOSDIS finalizing preparations to support ICESat and SORCE
– In all, EOSDIS is providing data processing, archival, and/or distribution
for over 15 Earth science satellite missions.

• EOSDIS has set a new benchmark for data management. The total
volume of the science data in our archives totals over 2 Petabytes.
Since 1998, the science data volume managed by the EOSDIS has
increased eight-fold, and continues to grow at a rate of over 2
Terabytes per day
• In FY02, EOSDIS provided more than 16 million data and information
products to over 1.8 million individuals.

3

Science Operations Timeline

Science
Operations
Supported

Heritage
Missions
T/P, UARS,
TOMS, ERBE
SeaWiFS
Radarsat

TRMM

Landsat 7
QuikSCAT

Terra
ACRIMSat

SAGE III
JASON

Aqua
GRACE

2 Petabytes

Archive Volume has doubled each year for the past three years

Archive
Growth

15 Million

Products
Distributed

FY94/95

FY96

FY97

FY98

FY99

FY00

4

FY01

FY02

EOSDIS DAAC Data delivery FY’02
October 1, 2002 through September 30, 2002
• Over 1.8 million distinct users
• 16,013,980 products delivered
• Notes to charts on following slides:
– Distinct users includes users accessing DAAC web pages,
including web-crawlers
– Distinct user type is based on email addresses of users or URLs
– Product delivered is defined as the smallest deliverable unit of data
– Product delivery breakdown is based on email addresses of users
receiving ECS and Non-ECS data
– “FTP Delivery” are to URLs not mapped to specific domains
– “Foreign Other” includes foreign email addresses whose country is
known but domain-type could not be determined

5

16,013,980 Data Products Delivered
October 1, 2001 - September 30, 2002

6

Over 1.8 Million Distinct Users
October 1, 2001 - September 30, 2002

7

ESDIS Status
• After years of development, ECS is operational and
generally recognized as successful.
• Primary ECS development contract is essentially
completed.
• ECS Maintenance and Development (EMD) will
emphasize maintenance more than development.
• Already the majority of ESDIS budget has shifted from
development to operations.
• Still two major areas of new capability (see posters at
AGU for more information):
– Data Pools
– ECHO

8

HDF-EOS Plans
• HDF-EOS 5 development is nearly complete.
• Continue to maintain, port to newer operating systems,
bug-fix.
• Need the advise of this community workshop - we will
discuss this afternoon.
– What tools or capabilities are now needed?
• HDF-EOS 2 and HDF-EOS 5

– When is it the right time to press EOS science teams to migrate to
HDF-EOS 5?
– What steps should NASA take to facilitate?

9

Data Pools Concept:
The Right Data, the Right Way, Right Now!
User-defined
Views, Presentations and Data Access Requests

Data
Items

End Users
Value-Added Providers

Data Producers
Other Data Pools

Geo
PIPE

Other Data Pools

Data Providers

Data Tailoring
Workflow
Management

Data
Services

Dynamic Web and FTP Data Views,
User-specified Data Access
PIPE = Personalized Information Presentation Engine

10

Data Service
Developers

General Capabilities
– The Right Data: Data Location
•
•
•
•
•
•

Groups, themes, bookmarks and views
Navigation and machine-based location
Science views (e.g., science metadata)
Applications views (e.g., OGIS coverage server)
Location aids (e.g., geopolitical overlays)
External location-support services

– The Right Way: Data Tailoring
•
•
•
•

Data reduction, manipulation and reformatting services
Virtual data products
Workflow management and execution monitoring
External tailoring services

– Right Now: Rapid Access
•
•
•
•
•

Low latency data transfers
Secure remote file access
On-the-fly data compression
Automated request routing and load balancing
Near real-time data

11

EOS ClearingHOuse (ECHO)
http://eos.nasa.gov/echo
• ECHO is a metadata clearinghouse
– A single Internet portal for Earth science metadata search
– Index of data provider inventory-level data holdings metadata.

• .ECHO is a data order broker
– Forwards orders for data discovered to the data providers to fill.
– Data providers retain customer fulfillment service

• ECHO is a data service broker
– Registered service are associated with registered datasets
– Four kinds of service association
• Advertised, Context Passing, Brokered, Order Option

• ECHO is an open client API for custom user clients
– The EOS Data Gateway (EDG) is ESDIS’ ECHO client

12

ECHO Data Providers
• Current ECHO Data Providers
– EOSDIS Core System DAACs
(17% thus far)
• EDC Land Processes DAAC
• Goddard DAAC
• NSIDC DAAC

– ORNL DAAC (100%)

• ECHO holds the metadata for
over 3 million granules, and
growing

ECHO Metadata Holdings
4,000,000
3,500,000
3,000,000
2,500,000
ORNL_DAAC
NSIDC_ECS

2,000,000

LP_ECS
GSFC_ECS

1,500,000
1,000,000
500,000
0
Jan-06 Feb- Mar-06 Apr-06 May- Jun-06 Jul-06 Aug06
06
06

Date

13

Sep06

HDF-EOS Tools and Information Web Site
http://hdfeos.gsfc.nasa.gov

05 December, 2002


14

http://hdfeos.gsfc.nasa.gov

15

Website status
• A resource for discovering about hdf-eos in particular.
– This workshop series’ presentation archive
– Links to hdf-eos tools

• Site has been revamped according to comments received at
the last workshop.
– Tools download page now has opportunity for user feedback.
– Workshop presentations are keyword searchable

• New features planned
– Better introductory material.
• Post and organize documentation of HDF-EOS.
• Better navigation to NCSA site for HDF

– Host hdf-eos “web forum”
• Incorporate the eostools@eos.nasa.gov listserv

16

HDF-EOS Profile

Richard Ullman
ESDIS Project

05 December, 2002


17

History
• September 1993, HDF adopted as baseline standard for EOSDIS Core
System standard data product generation, archival, ingest, and
distribution capabilities
• Dec. 94 - ECS Engineering Support Directive to create HDF-EOS
• June 1996, HDF-EOS v1.0 library released
• Upgrades every 6 mo.,
– Current version 2.8 on HDF 4

• HDF5 support (called HDF-EOS 5) beginning November 2000
– Current version 5.1.3 on HDF5

18

HDF-EOS Data Objects
•
•
•
•
•

Point
Swath
Profile (Swath subtype)
Grid
Zonal (HDF-EOS 5 only)

19

HDF-EOS 5
• Based on HDF5, a complete rewrite of HDF4 with a
different interface.
– First released in 2000.

• Designed to ‘resemble’ HDF-EOS 2 to the maximum
extent possible.
–
–
–
–

Support same data structures
Added prefix ‘HE5_’ to HDF-EOS 2 functions.
Doesn’t preclude HDF5 functionality.
Data Type changes, e.g. INT64 -> H5T_NATIVE_LONG

20

HDF-EOS 5 Functionality
•
•
•
•
•
•
•
•

Basic File I/O
Fill Values
Compression
Chunking/Tiling
Swath Interface
Grid Interface
Point Interface
Profile Interface

•
•
•
•
•
•
•

Global (File), Group & Local
Attributes
External Data Files
Subsetting
Unix/Linux Support
Threadsafe Version
FORTRAN, C, C++
General Table Interface
(proposed)

21

Top Level of HDF-EOS 5
Root -- “/”
HDFEOS
INFORMATION

HDFEOS

STRUC. METADATA
ADDITIONAL
SWATH

GRID

POINT

Global (file)
Attributes

The new ADDITIONAL Group has global (file) attributes
The new functionality is added to the EH(utility)
interface.
22

Swath Structure
Global Attribute
<SwathName>:<AttrName>

Group Attribute
<DataFields>:<AttrName>
Local Attribute
<FieldName>:<AttrName>

SwathName

Data Fields

Data Data
Field.1 Field.n

Profile Fields

Profile
Field.1

Profile
Field.n

Geolocation Fields

Longitude Latitude
Time

Each Data Field can have
Attributes and/or
Dimension Scales

CoLatitude

Shaded Objects are implemented
in a fixed way. User doesn’t have
direct access via the interface
Group

Attribute

23

Dataset

HDF-EOS Point
• Intended use
– Discrete points in time and/or
location.
– Table of data linked to table of
geographic information.
– 8-level Hierarchical, each level
may contain indices to the
level below
Latitude

Longitude

Temperature oC

Dew Point
o
C

61.12

-149.48

15.00

5.00

45.31

-122.41

17.00

5.00

38.50

-77.00

24.00

7.00

38.39

-90.15

27.00

11.00

M

M

M

M

24

HDF-EOS Point

Latitude
25.2645
22.3549
23.2564

Longitude
091.2564
-93.4657
-89.2546

Buoy_ID
0126
3564
1256

• Hierarchical links:
–

Every level in a Point data set
must be linked into the
hierarchy.
– Before two levels can be
linked, a link field must exist

Buoy
_ID
0126
0126
3564
1256
1256
0126
3564

Time
01:26
05:56
06:28
08:12
09:58
09:59
10:16

Wave
Height(ft)
2.54
3.58
12.64
7.58
7.76
4.23
10.23

25

Temp
(C)
18.4
18.2
16.4
17.1
17.2
20.1
17.5

HDF-EOS Swath
• Intended use
– Across track scanning
instruments.
– Sounding instruments
– Level 1: Geolocated Sensor
Units
– Level 2: Geophysical
Parameters

Scan Lines

Instrument
Path

Along Track

Instrument
Path

Along Track

26

HDF-EOS Swath
Data Fields

“Brightness
Temperature”

Geolocation Fields
Dimension
Dimension
Name: Scan
Name: Scan
Size: 16
Size: 16
Dimension
Dimension
Name: Track
Name: Track
Size: 42
Size: 42

Dimension
Dimension
Name: Geotrack
Name: Geotrack
Size: 21
Size: 21

“Time”

“Latitude”

“Longitude”
Map1
Map1
DataDimension: “Track”
DataDimension: “Track”
Geodimension: “Geotrack”
Geodimension: “Geotrack”
Offset: 1
Offset: 1
Increment: 2
Increment: 2

27

HDF-EOS Grid
• Intended use
– Variables mapped on uniform
space-time grid scales
– Level 3 - Gridded single
measurement parameters
– Level 4 - Modeled or derived
from multiple measurements

28

HDF-EOS Grid

Altitude
Size 30

Xdim
Size: 2000

Projinfo

Ydim
Size: 800

29

HDF-EOS Grid
•
•
•
•
•
•
•
•

Projections Supported
Geographic
Transverse Mercator
Universal Transverse Mercator
Hotine Oblique Mercator
Space Oblique Mercator
Polar Stereographic
Lambert Azimuthal Equal Area

•
•
•
•
•
•
•
•

Lambert Conformal Conic
Polyconic
Interrupted Goode’s
Homolosine
Integerized Sinusoidal
Compression Methods
Run-Length Encoding
Adaptive Huffman
Gzip

30

Product Levels
• Level 0 - Reconstructed,
unprocessed instrument/payload data at full resolution; any
and all communications
artifacts, e.g., synch. frames,
communications headers,
duplicate data removed.
• Level 1A- Reconstructed,
unprocessed instrument data at
full resolution, time-referenced,
and annotated with ancillary
information, including
radiometric and geometric
calibration coefficients and
georeferencing parameters, e.g.,
platform ephemeris, computed
and appended but not applied to
the Level 0 data.

•

•

•

•

Level 1B - Level 1A data that
have been processed to sensor
units (not all instruments will
have a Level 1B equivalent).
Level 2 - Derived geophysical
variables at the same resolution
and location as the Level 1
source data.
Level 3 - Variables mapped on
uniform space-time grid scales,
usually with some completeness
and consistency.
Level 4 - Model output or
results from analyses of lower
level data, e.g., variables
derived from multiple
measurements.

31

Resources
– HDF-EOS on the web:
• http://hdfeos.gsfc.nasa.gov/
• http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html

– HDF and HDF5 on the web:
• http://hdf.ncsa.uiuc.edu/

– HDF-EOS and HDF via email:
• eostools@eos.nasa.gov
• hdfhelp@ncsa.uiuc.edu

32

ESDIS Status (2002)

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a ESDIS Status (2002)

Semelhante a ESDIS Status (2002) (20)

Mais de The HDF-EOS Tools and Information Center

Mais de The HDF-EOS Tools and Information Center (20)

Último

Último (20)

ESDIS Status (2002)