SlideShare a Scribd company logo
1 of 57
Download to read offline
Introduction to HDF5

HDF and HDF-EOS Workshop XI
November 6-8, 2007

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD

1
Goals
• Introduce HDF5
• Explain how data can be organized and used
in an application
• Provide example code

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
2
For More Information…

All workshop slides will be available from:
http://hdfeos.org/workshops/ws11/workshop_eleven.php

See the Resources handout for where to get software,
Docs, FAQs, etc..

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
3
What is HDF5?
HDF = Hierarchical Data Format
• File format for managing any kind of data
• Software (library and tools) for accessing data in
that format

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
4
HDF5 Features
• Especially suited for large and/or complex
data collections.
• Platform independent
• C, F90, C++ , Java APIs

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
5
Diagram Definitions

= Group

= Dataset

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
6
Example HDF5 file
“/” (root)
“/foo”

3-D array

lat | lon | temp
----|-----|----12 | 23 | 3.1
15 | 24 | 4.2
17 | 21 | 3.6

Table

palette

Raster image
Raster image

11/6/07

2-D array

HDF and HDF-EOS Workshop XI, Landover, MD
7
Viewing an HDF5 File with
HDFView

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD

8
11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
9
Example HDF5 Application
#include<stdio.h>
#include "H5IM.h"
#define WIDTH
57
#define HEIGHT 57
#define RANK 2

/* dataset dimensions */

int main (void) {
hid_t
file;
herr_t
status;
unsigned char data[WIDTH][HEIGHT];
int
i, j, num, val;
FILE *fp;

/* file handle */
/* data to write */

fp = fopen ("storm110.txt", "r");

/* Open ASCII file */

for (i=0; i<WIDTH; i++)
for (j=0; j<HEIGHT; j++) {
num = fscanf (fp, "%d ", &val);
data[i][j] = val;
}

/* Read Values into ‘data’ buffer */

file = H5Fcreate ("storm.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); /* Create file */
status = H5IMmake_image_8bit (file, "Storm_Image", WIDTH, HEIGHT,
(const unsigned char *)data);
status = H5Fclose (file);

11/6/07

/* Create Image */

HDF and HDF-EOS Workshop XI, Landover, MD
10 /* Close file */
HDF5 Data Model

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD

11
HDF5 File
Container for Storing Scientific Data
• Primary Objects
- Datasets
- Groups

• Others Objects
- Attributes
- Property Lists
- Dataspaces
11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
12
HDF5 Dataset
• Data array
- Ordered collection of identically typed data items
distinguished by their indices

• Metadata
- Dataspace: Rank, dimensions; spatial info about dataset
- Datatype: Information to interpret your data
- Storage Properties: How array is organized
- Attributes: User-defined metadata (optional)

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
13
Dataset Components
Metadata

Data

Dataspace
Rank

Dimensions

3

Dim_1 = 4
Dim_2 = 5
Dim_3 = 7

Datatype
IEEE 32-bit float

Attributes

Properties

Time = 32.4

Chunked

Pressure = 987

Compressed

Temp = 56

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
14
HDF5 Dataset: Dataspace
Spatial Information about a dataset
• Rank and dimensions
- Permanent part of dataset definition
• Subset of points, for partial I/O
- Needed only during I/O operations

Rank = 2
Dimensions = 4x6

• Apply to datasets in memory or in
the file
11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
15
HDF5 Dataset: Compound Datatype
3

5

Dimensionality: 5 x 3
int8

Datatype:

int4

int16 2x3x2 array of float32

Each Element

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
16
HDF5 Dataset: Datatype
Information on how to interpret a data element
• Permanent part of the dataset definition
• HDF5 atomic types
- normal integer & float
- user-definable (e.g. 13-bit integer)
- variable length types (e.g. strings)
- pointers - references to objects/dataset regions
- enumeration - names mapped to integers
- array

• HDF5 compound types
- Comparable to C structs
- Members can be atomic or compound types
11/6/07
HDF and HDF-EOS Workshop XI, Landover, MD
17
HDF5 Dataset: Property List
A collection of values that can be passed to HDF5
functions at lower layers of library
• There are property lists that you can use when:
- creating a file
- accessing a file
- creating a dataset
- reading/writing to a dataset.
• To use the HDF5 library defaults: H5Pdefault
11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
18
HDF5 Dataset: Storage Layout Properties
• Contiguous: Dataset stored in continuous array of
bytes (Default)
• Chunked: Dataset stored as fixed sized chunks. Each
chunk is read/written with a single I/O operation.
Required for:
- compression
- unlimited dimension dataset (extendible)

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
19
HDF5 Dataset: Properties
Better subsetting
access time; extend,
compression

Chunked

Improves storage
efficiency,
transmission speed

Compressed

Arrays can be
extended in any
direction

Extendible

External
file
11/6/07

Dataset “Fred”

File B

File A
Metadata for Fred

Data for Fred

Metadata in one file,
raw data in another.

HDF and HDF-EOS Workshop XI, Landover, MD
20
HDF5 Dataset: Attributes
Data of form “name = value” attached to an object
• Scaled­down versions of dataset operations
- Not extendible
- No compression
- No partial I/O

• Optional

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
21
HDF5 Dataset (again)
• Data array
- Ordered collection of identically typed data items
distinguished by their indices

• Metadata
- Dataspace:
- Datatype:
- Properties:
- Attributes:

11/6/07

Rank, dimensions; spatial info about dataset
Information to interpret your data
How array is organized
User-defined metadata (optional)

HDF and HDF-EOS Workshop XI, Landover, MD
22
HDF5 File: Groups
A mechanism for describing collections of
related objects
• Every file starts with a
root group

“/”

• Can have attributes
• Similar to UNIX
directories

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
23
HDF5 objects are identified and
located by their pathnames
/ (root)
/x
/foo
/foo/temp
/foo/bar/temp

11/6/07

foo
temp

“/”

x

bar
temp

HDF and HDF-EOS Workshop XI, Landover, MD
24
HDF5 I/O Library

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD

25
Structure of HDF5 Library

Applications
Object API (C, F90, C++, Java)
Library internals
Virtual file I/O
File or other “storage”
11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
26
Virtual File I/O Layer
Allows HDF5 format address space to map to disk, the
network, memory, or a user-defined device

Virtual file I/O drivers
Stdio

File Family

MPI I/O

Memory

Network

…

“Storage”
File

11/6/07

File
Family

Memory

Network

…

HDF and HDF-EOS Workshop XI, Landover, MD
27
Introduction to
HDF5 API
Programming model for sequential access

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD

28
General API Topics

• General info about HDF5 programming (C )
• Walk through example program

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
29
The General HDF5 API
• Currently has C, Fortran 90, Java, C++ bindings.
• C routines begin with prefix H5X, where X is a single
letter indicating the object on which the operation is
to be performed.
Example APIs:
H5F: File Interface:
H5D: Dataset Interface:
H5S: DataSpace Interface:
H5P: Property List Interface:
H5G: Group Interface:
H5A: Attribute Interface:
11/6/07

H5Fopen
H5Dread
H5Screate_simple
H5Pset_chunk
H5Gcreate
H5Acreate

HDF and HDF-EOS Workshop XI, Landover, MD
30
The General Paradigm
•
•
•
•

11/6/07

Properties of objects are defined (optional)
Objects are opened or created
Objects then accessed
Objects finally closed

HDF and HDF-EOS Workshop XI, Landover, MD
31
Order of Operations

The library imposes an order on the operations by
argument dependencies
Example: A file must be opened before a dataset
because the dataset open call requires a file identifier as
an argument

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
32
HDF5 C Programming Issues
For portability, HDF5 library has its own defined types.
For example:
hid_t:
Object identifiers
hsize_t:
Size used for dimensions
herr_t:
Function return value
For C, include #include hdf5.h at the top of your HDF5
application.

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
33
H5dump Command-line Utility To View HDF5 File

h5dump [--header] [-a ] [-d <names>] [-g <names>]
[-l <names>] [-t <names>] <file>
--header
Display header only; no data is displayed.
-a <names> Display the specified attribute(s).
-d <names> Display the specified dataset(s).
-g <names> Display the specified group(s) and all the members.
-l <names>
Displays the value(s) of the specified soft link(s).
-t <names> Display the specified named datatype(s).
-p
Display properties.

<names> is one or more appropriate object names.

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
34
Example of h5dump Output

HDF5 "dset.h5" {
GROUP "/" {
DATASET "dset" {
DATATYPE { H5T_STD_I32BE }
DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) }
DATA {
“/”
1, 2, 3, 4, 5, 6,
7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18,
‘dset’
19, 20, 21, 22, 23, 24
}
}
}
}

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
35
Example HDF5 Application
1 #include "hdf5.h"
2 #define FILE "dset.h5"
3 int main () {
4 hid_t
file_id, dataset_id, dataspace_id;
5 hsize_t dims[2];
6 herr_t
status;
7 int
i, j, dset_data[4][6];

Steps:
11
Create (or use default) file creation/access
properties
11
Create file w/ above properties
12-15 Create (or use default) dataset
characteristics:
[dataspace, datatype, storage properties]
15
Create dataset using above characteristics
16
Write data to dataset
17-19 Close all interfaces

8
9
10

for (i = 0; i < 4; i++)
for (j = 0; j < 6; j++)
dset_data[i][j] = i * 6 + j + 1;

11

file_id = H5Fcreate (FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

12
13
14

dims[0] = 4;
dims[1] = 6;
dataspace_id = H5Screate_simple (2, dims, NULL);

15

dataset_id = H5Dcreate (file_id, "/dset", H5T_STD_I32BE, dataspace_id,
H5P_DEFAULT);
16 status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
dset_data);
17
18
19

status = H5Sclose (dataspace_id);
status = H5Dclose (dataset_id);
status = H5Fclose (file_id);

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
36
Example Code - Dataspace
Rank

Array of Dimension
Sizes (4x6)
NOT used here.

12
13
14

dims[0] = 4;
dims[1] = 6;
dataspace_id = H5Screate_simple (2, dims, NULL);

15

dataset_id = H5Dcreate (file_id, “/dset", H5T_STD_I32BE, dataspace_id,
H5P_DEFAULT);

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
37
Example Code - Datatype

12
13
14

dims[0] = 4;
dims[1] = 6;
dataspace_id = H5Screate_simple (2, dims, NULL);

15

dataset_id = H5Dcreate (file_id, "/dset", H5T_STD_I32BE, dataspace_id,
H5P_DEFAULT);

Where do you
get the datatype?

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
38
HDF5 Pre-defined Datatype Identifiers
HDF5 opens set of Pre-Defined Datatype identifiers.
For example:
C Type

HDF5 File Type

HDF5 Memory Type

int

H5T_STD_I32BE
H5T_STD_I32LE

H5T_NATIVE_INT

float

H5T_IEEE_F32BE
H5T_IEEE_F32LE

H5T_NATIVE_FLOAT

double

H5T_IEEE_F64BE
H5T_IEEE_F64LE

H5T_NATIVE_DOUBLE

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
39
Pre-Defined File Datatype Identifiers
Examples:
H5T_IEEE_F64LE Eight-byte, little-endian, IEEE floating-point
H5T_VAX_F32
Four-byte VAX floating point
H5T_STD_I32LE Four-byte, little-endian, signed two's
complement integer
H5T_STD_U16BE Two-byte, big-endian, unsigned integer
Architecture*

Programming
Type

NOTE: What you see in the file. Name is the same everywhere and
explicitly defines a datatype.
*STD= “An architecture with a semi-standard type like 2’s complement integer, unsigned integer…”

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
40
Pre-defined Native Datatype Identifiers
Examples of predefined native types in C:
H5T_NATIVE_INT
H5T_NATIVE_FLOAT
H5T_NATIVE_UINT
H5T_NATIVE_LONG
H5T_NATIVE_CHAR

(int)
(float )
(unsigned int)
(long )
(char )

NOTE: Memory types.
Different for each machine.
Used for reading/writing.

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
41
Example Code - H5Dwrite
Dataset Identifier from
H5Dcreate or H5Dopen

Memory Datatype

status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL,
H5S_ALL, H5P_DEFAULT, dset_data);

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
42
Example Code – H5Dwrite

status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
H5P_DEFAULT, dset_data);

Data Transfer
Property List

Memory
Dataspace

File
Dataspace

H5S_ALL selects entire
dataspace

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
43
Memory and File Dataspaces – Why?
Partial I/O: Selected elements from source are mapped
(read/written) to selected elements in destination
- Selections in memory can differ from selection in file:

- Number of selected elements must be the same in source
and destination
- Selection can be slabs, points, or result of set operations
(union, difference ..) on slabs or points
11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
44
Example Code To:
• Create dataset in a group other than root

• Open file and dataset and read data
• Create an attribute for the dataset

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
45
Example HDF5 Application
1 #include "hdf5.h"
2 #define FILE "dset.h5"
3 int main () {
4 hid_t
file_id, dataset_id, dataspace_id;
5 hsize_t dims[2];
6 herr_t
status;
7 int
i, j, dset_data[4][6];
8
9
10

for (i = 0; i < 4; i++)
for (j = 0; j < 6; j++)
dset_data[i][j] = i * 6 + j + 1;

11

file_id = H5Fcreate (FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

12
13
14

dims[0] = 4;
dims[1] = 6;
dataspace_id = H5Screate_simple (2, dims, NULL);

15

dataset_id = H5Dcreate (file_id, "/dset", H5T_STD_I32BE, dataspace_id,
H5P_DEFAULT);
16 status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
dset_data);
17
18
19

status = H5Sclose (dataspace_id);
status = H5Dclose (dataset_id);
status = H5Fclose (file_id);

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
46
How to put Dataset in a Group?

hid_t
…

group_id;

Steps:
a. Create a group
b. Insert the dataset
into the group
c. Close the group

a group_id = H5Gcreate (file_id, "mygroup", 0);
…
b dataset_id = H5Dcreate (group_id, "dset", H5T_STD_I32BE,
dataspace_id, H5P_DEFAULT);
c status = H5Gclose (group_id);

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
47
h5dump Output w/Dataset in a Group
$ h5dump dset.h5
Note that dataset
HDF5 "dset.h5" {
is in the group
GROUP "/" {
“mygroup”
GROUP "mygroup" {
DATASET "dset" {
DATATYPE H5T_STD_I32BE
“/”
DATASPACE SIMPLE { ( 4, 6 ) / ( 4, 6 ) }
DATA {
(0,0): 1, 2, 3, 4, 5, 6,
mygroup
(1,0): 7, 8, 9, 10, 11, 12,
(2,0): 13, 14, 15, 16, 17, 18,
(3,0): 19, 20, 21, 22, 23, 24
}
}
dset
}
}
}

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
48
How to Read an Existing Dataset
file_id = H5Fopen (FILE, H5F_ACC_RDWR, H5P_DEFAULT);
dataset_id = H5Dopen (file_id, "/dset");
status = H5Dread (dataset_id, H5T_NATIVE_INT, H5S_ALL,
H5S_ALL, H5P_DEFAULT, dset_rdata);
status = H5Dclose (dataset_id);
status = H5Fclose (file_id);
Steps:
- Open Existing File
- Open Existing Dataset
- Read Data
- Close dataset, file ids

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
49
How to Create an Attribute in the Dataset?
Steps:
a Create an attribute attached
to already open dataset
b Write data to the attribute
c Close attribute, dataspace

hid_t
aspace_id;
hsize_t dimsa;
int
attr_data[2]= {100, 200};
hid_t
attribute_id;
…
Attach to open dataset
dimsa = 2;
aspace_id = H5Screate_simple(1, &dimsa, NULL);
a

attribute_id = H5Acreate (dataset_id, "Units", H5T_STD_I32BE,

b

aspace_id, H5P_DEFAULT);
status = H5Awrite (attribute_id, H5T_NATIVE_INT, attr_data);

c
c

status = H5Aclose (attribute_id);
status = H5Sclose (aspace_id);

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
50
h5dump Output of Dataset with an Attribute
HDF5 "dset.h5" {
GROUP "/" {
DATASET "dset" {
DATATYPE H5T_STD_I32BE
DATASPACE SIMPLE { ( 4, 6 ) / ( 4, 6 ) }
DATA {
(0,0): 1, 2, 3, 4, 5, 6,
(1,0): 7, 8, 9, 10, 11, 12,
(2,0): 13, 14, 15, 16, 17, 18,
(3,0): 19, 20, 21, 22, 23, 24
}
ATTRIBUTE "Units" {
DATATYPE H5T_STD_I32BE
DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
DATA {
(0): 100, 200
}
}
}
}
}

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
51
HDF5 High Level APIs

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD

52
High Level APIs

• Make HDF5 easier to use
• Encourage standard ways to store objects
• Included with HDF5 library
However:
• Still need HDF5 calls (H5Fopen/H5Fclose…)

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
53
High Level APIs
• HDF5 Lite (H5LT): Functions that simplify the steps
needed to create/read datasets and attributes

• HDF5 Image (H5IM): Functions for creating images
in HDF5.

• HDF5 Table (H5TB): Functions for creating tables
(collections of records) in HDF5.

• Others … (Dimension Scales, Packet Table)
11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
54
High Level Programming Model

file_id = H5Fcreate ( "test.h5", ……. );
/* Call some High Level function */
H5LTsome_function ( file_id, ...extra parameters );
status = H5Fclose ( file_id );

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD
55
HDF5 Hands-On
• Need Secure Shell or putty
• Everyone is logged in as: workshop
• Working directory for each user is specified at
top right of Hands-On page
• Compile Examples with: h5fc, h5cc
• Display Dataset with:
11/6/07

h5dump

HDF and HDF-EOS Workshop XI, Landover, MD
56
Thank you!
This presentation is based upon work supported in part by a
Cooperative Agreement with NASA under NASA NNX06AC83A.
Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the author(s) and do not
necessarily reflect the views of the National Aeronautics and Space
Administration.

11/6/07

HDF and HDF-EOS Workshop XI, Landover, MD

57

More Related Content

What's hot

Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 productsInteroperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 productsThe HDF-EOS Tools and Information Center
 

What's hot (20)

Parallel HDF5 Developments
Parallel HDF5 DevelopmentsParallel HDF5 Developments
Parallel HDF5 Developments
 
Advanced HDF5 Features
Advanced HDF5 FeaturesAdvanced HDF5 Features
Advanced HDF5 Features
 
Introduction to HDF5 Data Model, Programming Model and Library APIs
Introduction to HDF5 Data Model, Programming Model and Library APIsIntroduction to HDF5 Data Model, Programming Model and Library APIs
Introduction to HDF5 Data Model, Programming Model and Library APIs
 
NetCDF and HDF5
NetCDF and HDF5NetCDF and HDF5
NetCDF and HDF5
 
Performance Tuning in HDF5
Performance Tuning in HDF5 Performance Tuning in HDF5
Performance Tuning in HDF5
 
HDF Tools Tutorial
HDF Tools TutorialHDF Tools Tutorial
HDF Tools Tutorial
 
H5Coro: The Cloud-Optimized Read-Only Library
H5Coro: The Cloud-Optimized Read-Only LibraryH5Coro: The Cloud-Optimized Read-Only Library
H5Coro: The Cloud-Optimized Read-Only Library
 
Using HDF5 and Python: The H5py module
Using HDF5 and Python: The H5py moduleUsing HDF5 and Python: The H5py module
Using HDF5 and Python: The H5py module
 
HDF Tools Tutorial
HDF Tools TutorialHDF Tools Tutorial
HDF Tools Tutorial
 
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewHDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
 
Data Interoperability
Data InteroperabilityData Interoperability
Data Interoperability
 
HDF-EOS 2/5 to netCDF Converter
HDF-EOS 2/5 to netCDF ConverterHDF-EOS 2/5 to netCDF Converter
HDF-EOS 2/5 to netCDF Converter
 
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 productsInteroperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
Interoperability with netCDF-4 - Experience with NPP and HDF-EOS5 products
 
HDF Group Support for NPP/NPOESS/JPSS
HDF Group Support for NPP/NPOESS/JPSSHDF Group Support for NPP/NPOESS/JPSS
HDF Group Support for NPP/NPOESS/JPSS
 
Easy Access of NASA HDF data via OPeNDAP
Easy Access of NASA HDF data via OPeNDAPEasy Access of NASA HDF data via OPeNDAP
Easy Access of NASA HDF data via OPeNDAP
 
Easy Remote Access Via OPeNDAP
Easy Remote Access Via OPeNDAPEasy Remote Access Via OPeNDAP
Easy Remote Access Via OPeNDAP
 
Big data overview by Edgars
Big data overview by EdgarsBig data overview by Edgars
Big data overview by Edgars
 
HDF4 and HDF5 Performance Preliminary Results
HDF4 and HDF5 Performance Preliminary ResultsHDF4 and HDF5 Performance Preliminary Results
HDF4 and HDF5 Performance Preliminary Results
 
HDF5 Life cycle of data
HDF5 Life cycle of dataHDF5 Life cycle of data
HDF5 Life cycle of data
 
May 2013 HUG: HCatalog/Hive Data Out
May 2013 HUG: HCatalog/Hive Data OutMay 2013 HUG: HCatalog/Hive Data Out
May 2013 HUG: HCatalog/Hive Data Out
 

Similar to Introduction to HDF5

Similar to Introduction to HDF5 (20)

Introduction to HDF5
Introduction to HDF5Introduction to HDF5
Introduction to HDF5
 
Hdf5 intro
Hdf5 introHdf5 intro
Hdf5 intro
 
Introduction to HDF5
Introduction to HDF5Introduction to HDF5
Introduction to HDF5
 
Integrating HDF5 with SRB
Integrating HDF5 with SRBIntegrating HDF5 with SRB
Integrating HDF5 with SRB
 
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout MapsEnsuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
 
Introduction to HDF5 Data and Programming Models
Introduction to HDF5 Data and Programming ModelsIntroduction to HDF5 Data and Programming Models
Introduction to HDF5 Data and Programming Models
 
HDF5 iRODS
HDF5 iRODSHDF5 iRODS
HDF5 iRODS
 
Parallel HDF5 Introductory Tutorial
Parallel HDF5 Introductory TutorialParallel HDF5 Introductory Tutorial
Parallel HDF5 Introductory Tutorial
 
HDF-EOS Subsetting: HEW and other tools
HDF-EOS Subsetting: HEW and other toolsHDF-EOS Subsetting: HEW and other tools
HDF-EOS Subsetting: HEW and other tools
 
HDFView and HDF Java Products
HDFView and HDF Java ProductsHDFView and HDF Java Products
HDFView and HDF Java Products
 
HDF5 and Storage Resource Broker
HDF5 and Storage Resource BrokerHDF5 and Storage Resource Broker
HDF5 and Storage Resource Broker
 
HDF5 Advanced Topics
HDF5 Advanced TopicsHDF5 Advanced Topics
HDF5 Advanced Topics
 
HDF-Java Products
HDF-Java ProductsHDF-Java Products
HDF-Java Products
 
Advanced HDF5 Features
Advanced HDF5 FeaturesAdvanced HDF5 Features
Advanced HDF5 Features
 
HDF Update for DAAC Managers (2017-02-27)
HDF Update for DAAC Managers (2017-02-27)HDF Update for DAAC Managers (2017-02-27)
HDF Update for DAAC Managers (2017-02-27)
 
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 dataUsage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data
 
HDF OPeNDAP project update and demo
HDF OPeNDAP project update and demoHDF OPeNDAP project update and demo
HDF OPeNDAP project update and demo
 
HDF5 Advanced Topics
HDF5 Advanced TopicsHDF5 Advanced Topics
HDF5 Advanced Topics
 
HDF Update
HDF UpdateHDF Update
HDF Update
 
HDF5 Tools Updates
HDF5 Tools UpdatesHDF5 Tools Updates
HDF5 Tools Updates
 

More from The HDF-EOS Tools and Information Center

STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
 

More from The HDF-EOS Tools and Information Center (20)

Cloud-Optimized HDF5 Files
Cloud-Optimized HDF5 FilesCloud-Optimized HDF5 Files
Cloud-Optimized HDF5 Files
 
Accessing HDF5 data in the cloud with HSDS
Accessing HDF5 data in the cloud with HSDSAccessing HDF5 data in the cloud with HSDS
Accessing HDF5 data in the cloud with HSDS
 
The State of HDF
The State of HDFThe State of HDF
The State of HDF
 
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance FeaturesHighly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
 
Creating Cloud-Optimized HDF5 Files
Creating Cloud-Optimized HDF5 FilesCreating Cloud-Optimized HDF5 Files
Creating Cloud-Optimized HDF5 Files
 
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance DiscussionHDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
 
Hyrax: Serving Data from S3
Hyrax: Serving Data from S3Hyrax: Serving Data from S3
Hyrax: Serving Data from S3
 
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLABAccessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
 
HDF - Current status and Future Directions
HDF - Current status and Future DirectionsHDF - Current status and Future Directions
HDF - Current status and Future Directions
 
HDFEOS.org User Analsys, Updates, and Future
HDFEOS.org User Analsys, Updates, and FutureHDFEOS.org User Analsys, Updates, and Future
HDFEOS.org User Analsys, Updates, and Future
 
HDF - Current status and Future Directions
HDF - Current status and Future Directions HDF - Current status and Future Directions
HDF - Current status and Future Directions
 
MATLAB Modernization on HDF5 1.10
MATLAB Modernization on HDF5 1.10MATLAB Modernization on HDF5 1.10
MATLAB Modernization on HDF5 1.10
 
HDF for the Cloud - Serverless HDF
HDF for the Cloud - Serverless HDFHDF for the Cloud - Serverless HDF
HDF for the Cloud - Serverless HDF
 
HDF5 <-> Zarr
HDF5 <-> ZarrHDF5 <-> Zarr
HDF5 <-> Zarr
 
HDF for the Cloud - New HDF Server Features
HDF for the Cloud - New HDF Server FeaturesHDF for the Cloud - New HDF Server Features
HDF for the Cloud - New HDF Server Features
 
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
 
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
 
HDF5 and Ecosystem: What Is New?
HDF5 and Ecosystem: What Is New?HDF5 and Ecosystem: What Is New?
HDF5 and Ecosystem: What Is New?
 
HDF5 Roadmap 2019-2020
HDF5 Roadmap 2019-2020HDF5 Roadmap 2019-2020
HDF5 Roadmap 2019-2020
 
Leveraging the Cloud for HDF Software Testing
Leveraging the Cloud for HDF Software TestingLeveraging the Cloud for HDF Software Testing
Leveraging the Cloud for HDF Software Testing
 

Recently uploaded

COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 

Recently uploaded (20)

COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 

Introduction to HDF5

  • 1. Introduction to HDF5 HDF and HDF-EOS Workshop XI November 6-8, 2007 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 1
  • 2. Goals • Introduce HDF5 • Explain how data can be organized and used in an application • Provide example code 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 2
  • 3. For More Information… All workshop slides will be available from: http://hdfeos.org/workshops/ws11/workshop_eleven.php See the Resources handout for where to get software, Docs, FAQs, etc.. 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 3
  • 4. What is HDF5? HDF = Hierarchical Data Format • File format for managing any kind of data • Software (library and tools) for accessing data in that format 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 4
  • 5. HDF5 Features • Especially suited for large and/or complex data collections. • Platform independent • C, F90, C++ , Java APIs 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 5
  • 6. Diagram Definitions = Group = Dataset 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 6
  • 7. Example HDF5 file “/” (root) “/foo” 3-D array lat | lon | temp ----|-----|----12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 Table palette Raster image Raster image 11/6/07 2-D array HDF and HDF-EOS Workshop XI, Landover, MD 7
  • 8. Viewing an HDF5 File with HDFView 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 8
  • 9. 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 9
  • 10. Example HDF5 Application #include<stdio.h> #include "H5IM.h" #define WIDTH 57 #define HEIGHT 57 #define RANK 2 /* dataset dimensions */ int main (void) { hid_t file; herr_t status; unsigned char data[WIDTH][HEIGHT]; int i, j, num, val; FILE *fp; /* file handle */ /* data to write */ fp = fopen ("storm110.txt", "r"); /* Open ASCII file */ for (i=0; i<WIDTH; i++) for (j=0; j<HEIGHT; j++) { num = fscanf (fp, "%d ", &val); data[i][j] = val; } /* Read Values into ‘data’ buffer */ file = H5Fcreate ("storm.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); /* Create file */ status = H5IMmake_image_8bit (file, "Storm_Image", WIDTH, HEIGHT, (const unsigned char *)data); status = H5Fclose (file); 11/6/07 /* Create Image */ HDF and HDF-EOS Workshop XI, Landover, MD 10 /* Close file */
  • 11. HDF5 Data Model 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 11
  • 12. HDF5 File Container for Storing Scientific Data • Primary Objects - Datasets - Groups • Others Objects - Attributes - Property Lists - Dataspaces 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 12
  • 13. HDF5 Dataset • Data array - Ordered collection of identically typed data items distinguished by their indices • Metadata - Dataspace: Rank, dimensions; spatial info about dataset - Datatype: Information to interpret your data - Storage Properties: How array is organized - Attributes: User-defined metadata (optional) 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 13
  • 14. Dataset Components Metadata Data Dataspace Rank Dimensions 3 Dim_1 = 4 Dim_2 = 5 Dim_3 = 7 Datatype IEEE 32-bit float Attributes Properties Time = 32.4 Chunked Pressure = 987 Compressed Temp = 56 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 14
  • 15. HDF5 Dataset: Dataspace Spatial Information about a dataset • Rank and dimensions - Permanent part of dataset definition • Subset of points, for partial I/O - Needed only during I/O operations Rank = 2 Dimensions = 4x6 • Apply to datasets in memory or in the file 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 15
  • 16. HDF5 Dataset: Compound Datatype 3 5 Dimensionality: 5 x 3 int8 Datatype: int4 int16 2x3x2 array of float32 Each Element 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 16
  • 17. HDF5 Dataset: Datatype Information on how to interpret a data element • Permanent part of the dataset definition • HDF5 atomic types - normal integer & float - user-definable (e.g. 13-bit integer) - variable length types (e.g. strings) - pointers - references to objects/dataset regions - enumeration - names mapped to integers - array • HDF5 compound types - Comparable to C structs - Members can be atomic or compound types 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 17
  • 18. HDF5 Dataset: Property List A collection of values that can be passed to HDF5 functions at lower layers of library • There are property lists that you can use when: - creating a file - accessing a file - creating a dataset - reading/writing to a dataset. • To use the HDF5 library defaults: H5Pdefault 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 18
  • 19. HDF5 Dataset: Storage Layout Properties • Contiguous: Dataset stored in continuous array of bytes (Default) • Chunked: Dataset stored as fixed sized chunks. Each chunk is read/written with a single I/O operation. Required for: - compression - unlimited dimension dataset (extendible) 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 19
  • 20. HDF5 Dataset: Properties Better subsetting access time; extend, compression Chunked Improves storage efficiency, transmission speed Compressed Arrays can be extended in any direction Extendible External file 11/6/07 Dataset “Fred” File B File A Metadata for Fred Data for Fred Metadata in one file, raw data in another. HDF and HDF-EOS Workshop XI, Landover, MD 20
  • 21. HDF5 Dataset: Attributes Data of form “name = value” attached to an object • Scaled­down versions of dataset operations - Not extendible - No compression - No partial I/O • Optional 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 21
  • 22. HDF5 Dataset (again) • Data array - Ordered collection of identically typed data items distinguished by their indices • Metadata - Dataspace: - Datatype: - Properties: - Attributes: 11/6/07 Rank, dimensions; spatial info about dataset Information to interpret your data How array is organized User-defined metadata (optional) HDF and HDF-EOS Workshop XI, Landover, MD 22
  • 23. HDF5 File: Groups A mechanism for describing collections of related objects • Every file starts with a root group “/” • Can have attributes • Similar to UNIX directories 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 23
  • 24. HDF5 objects are identified and located by their pathnames / (root) /x /foo /foo/temp /foo/bar/temp 11/6/07 foo temp “/” x bar temp HDF and HDF-EOS Workshop XI, Landover, MD 24
  • 25. HDF5 I/O Library 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 25
  • 26. Structure of HDF5 Library Applications Object API (C, F90, C++, Java) Library internals Virtual file I/O File or other “storage” 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 26
  • 27. Virtual File I/O Layer Allows HDF5 format address space to map to disk, the network, memory, or a user-defined device Virtual file I/O drivers Stdio File Family MPI I/O Memory Network … “Storage” File 11/6/07 File Family Memory Network … HDF and HDF-EOS Workshop XI, Landover, MD 27
  • 28. Introduction to HDF5 API Programming model for sequential access 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 28
  • 29. General API Topics • General info about HDF5 programming (C ) • Walk through example program 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 29
  • 30. The General HDF5 API • Currently has C, Fortran 90, Java, C++ bindings. • C routines begin with prefix H5X, where X is a single letter indicating the object on which the operation is to be performed. Example APIs: H5F: File Interface: H5D: Dataset Interface: H5S: DataSpace Interface: H5P: Property List Interface: H5G: Group Interface: H5A: Attribute Interface: 11/6/07 H5Fopen H5Dread H5Screate_simple H5Pset_chunk H5Gcreate H5Acreate HDF and HDF-EOS Workshop XI, Landover, MD 30
  • 31. The General Paradigm • • • • 11/6/07 Properties of objects are defined (optional) Objects are opened or created Objects then accessed Objects finally closed HDF and HDF-EOS Workshop XI, Landover, MD 31
  • 32. Order of Operations The library imposes an order on the operations by argument dependencies Example: A file must be opened before a dataset because the dataset open call requires a file identifier as an argument 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 32
  • 33. HDF5 C Programming Issues For portability, HDF5 library has its own defined types. For example: hid_t: Object identifiers hsize_t: Size used for dimensions herr_t: Function return value For C, include #include hdf5.h at the top of your HDF5 application. 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 33
  • 34. H5dump Command-line Utility To View HDF5 File h5dump [--header] [-a ] [-d <names>] [-g <names>] [-l <names>] [-t <names>] <file> --header Display header only; no data is displayed. -a <names> Display the specified attribute(s). -d <names> Display the specified dataset(s). -g <names> Display the specified group(s) and all the members. -l <names> Displays the value(s) of the specified soft link(s). -t <names> Display the specified named datatype(s). -p Display properties. <names> is one or more appropriate object names. 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 34
  • 35. Example of h5dump Output HDF5 "dset.h5" { GROUP "/" { DATASET "dset" { DATATYPE { H5T_STD_I32BE } DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) } DATA { “/” 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, ‘dset’ 19, 20, 21, 22, 23, 24 } } } } 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 35
  • 36. Example HDF5 Application 1 #include "hdf5.h" 2 #define FILE "dset.h5" 3 int main () { 4 hid_t file_id, dataset_id, dataspace_id; 5 hsize_t dims[2]; 6 herr_t status; 7 int i, j, dset_data[4][6]; Steps: 11 Create (or use default) file creation/access properties 11 Create file w/ above properties 12-15 Create (or use default) dataset characteristics: [dataspace, datatype, storage properties] 15 Create dataset using above characteristics 16 Write data to dataset 17-19 Close all interfaces 8 9 10 for (i = 0; i < 4; i++) for (j = 0; j < 6; j++) dset_data[i][j] = i * 6 + j + 1; 11 file_id = H5Fcreate (FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); 12 13 14 dims[0] = 4; dims[1] = 6; dataspace_id = H5Screate_simple (2, dims, NULL); 15 dataset_id = H5Dcreate (file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT); 16 status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); 17 18 19 status = H5Sclose (dataspace_id); status = H5Dclose (dataset_id); status = H5Fclose (file_id); 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 36
  • 37. Example Code - Dataspace Rank Array of Dimension Sizes (4x6) NOT used here. 12 13 14 dims[0] = 4; dims[1] = 6; dataspace_id = H5Screate_simple (2, dims, NULL); 15 dataset_id = H5Dcreate (file_id, “/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT); 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 37
  • 38. Example Code - Datatype 12 13 14 dims[0] = 4; dims[1] = 6; dataspace_id = H5Screate_simple (2, dims, NULL); 15 dataset_id = H5Dcreate (file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT); Where do you get the datatype? 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 38
  • 39. HDF5 Pre-defined Datatype Identifiers HDF5 opens set of Pre-Defined Datatype identifiers. For example: C Type HDF5 File Type HDF5 Memory Type int H5T_STD_I32BE H5T_STD_I32LE H5T_NATIVE_INT float H5T_IEEE_F32BE H5T_IEEE_F32LE H5T_NATIVE_FLOAT double H5T_IEEE_F64BE H5T_IEEE_F64LE H5T_NATIVE_DOUBLE 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 39
  • 40. Pre-Defined File Datatype Identifiers Examples: H5T_IEEE_F64LE Eight-byte, little-endian, IEEE floating-point H5T_VAX_F32 Four-byte VAX floating point H5T_STD_I32LE Four-byte, little-endian, signed two's complement integer H5T_STD_U16BE Two-byte, big-endian, unsigned integer Architecture* Programming Type NOTE: What you see in the file. Name is the same everywhere and explicitly defines a datatype. *STD= “An architecture with a semi-standard type like 2’s complement integer, unsigned integer…” 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 40
  • 41. Pre-defined Native Datatype Identifiers Examples of predefined native types in C: H5T_NATIVE_INT H5T_NATIVE_FLOAT H5T_NATIVE_UINT H5T_NATIVE_LONG H5T_NATIVE_CHAR (int) (float ) (unsigned int) (long ) (char ) NOTE: Memory types. Different for each machine. Used for reading/writing. 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 41
  • 42. Example Code - H5Dwrite Dataset Identifier from H5Dcreate or H5Dopen Memory Datatype status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 42
  • 43. Example Code – H5Dwrite status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); Data Transfer Property List Memory Dataspace File Dataspace H5S_ALL selects entire dataspace 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 43
  • 44. Memory and File Dataspaces – Why? Partial I/O: Selected elements from source are mapped (read/written) to selected elements in destination - Selections in memory can differ from selection in file: - Number of selected elements must be the same in source and destination - Selection can be slabs, points, or result of set operations (union, difference ..) on slabs or points 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 44
  • 45. Example Code To: • Create dataset in a group other than root • Open file and dataset and read data • Create an attribute for the dataset 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 45
  • 46. Example HDF5 Application 1 #include "hdf5.h" 2 #define FILE "dset.h5" 3 int main () { 4 hid_t file_id, dataset_id, dataspace_id; 5 hsize_t dims[2]; 6 herr_t status; 7 int i, j, dset_data[4][6]; 8 9 10 for (i = 0; i < 4; i++) for (j = 0; j < 6; j++) dset_data[i][j] = i * 6 + j + 1; 11 file_id = H5Fcreate (FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); 12 13 14 dims[0] = 4; dims[1] = 6; dataspace_id = H5Screate_simple (2, dims, NULL); 15 dataset_id = H5Dcreate (file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT); 16 status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); 17 18 19 status = H5Sclose (dataspace_id); status = H5Dclose (dataset_id); status = H5Fclose (file_id); 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 46
  • 47. How to put Dataset in a Group? hid_t … group_id; Steps: a. Create a group b. Insert the dataset into the group c. Close the group a group_id = H5Gcreate (file_id, "mygroup", 0); … b dataset_id = H5Dcreate (group_id, "dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT); c status = H5Gclose (group_id); 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 47
  • 48. h5dump Output w/Dataset in a Group $ h5dump dset.h5 Note that dataset HDF5 "dset.h5" { is in the group GROUP "/" { “mygroup” GROUP "mygroup" { DATASET "dset" { DATATYPE H5T_STD_I32BE “/” DATASPACE SIMPLE { ( 4, 6 ) / ( 4, 6 ) } DATA { (0,0): 1, 2, 3, 4, 5, 6, mygroup (1,0): 7, 8, 9, 10, 11, 12, (2,0): 13, 14, 15, 16, 17, 18, (3,0): 19, 20, 21, 22, 23, 24 } } dset } } } 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 48
  • 49. How to Read an Existing Dataset file_id = H5Fopen (FILE, H5F_ACC_RDWR, H5P_DEFAULT); dataset_id = H5Dopen (file_id, "/dset"); status = H5Dread (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_rdata); status = H5Dclose (dataset_id); status = H5Fclose (file_id); Steps: - Open Existing File - Open Existing Dataset - Read Data - Close dataset, file ids 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 49
  • 50. How to Create an Attribute in the Dataset? Steps: a Create an attribute attached to already open dataset b Write data to the attribute c Close attribute, dataspace hid_t aspace_id; hsize_t dimsa; int attr_data[2]= {100, 200}; hid_t attribute_id; … Attach to open dataset dimsa = 2; aspace_id = H5Screate_simple(1, &dimsa, NULL); a attribute_id = H5Acreate (dataset_id, "Units", H5T_STD_I32BE, b aspace_id, H5P_DEFAULT); status = H5Awrite (attribute_id, H5T_NATIVE_INT, attr_data); c c status = H5Aclose (attribute_id); status = H5Sclose (aspace_id); 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 50
  • 51. h5dump Output of Dataset with an Attribute HDF5 "dset.h5" { GROUP "/" { DATASET "dset" { DATATYPE H5T_STD_I32BE DATASPACE SIMPLE { ( 4, 6 ) / ( 4, 6 ) } DATA { (0,0): 1, 2, 3, 4, 5, 6, (1,0): 7, 8, 9, 10, 11, 12, (2,0): 13, 14, 15, 16, 17, 18, (3,0): 19, 20, 21, 22, 23, 24 } ATTRIBUTE "Units" { DATATYPE H5T_STD_I32BE DATASPACE SIMPLE { ( 2 ) / ( 2 ) } DATA { (0): 100, 200 } } } } } 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 51
  • 52. HDF5 High Level APIs 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 52
  • 53. High Level APIs • Make HDF5 easier to use • Encourage standard ways to store objects • Included with HDF5 library However: • Still need HDF5 calls (H5Fopen/H5Fclose…) 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 53
  • 54. High Level APIs • HDF5 Lite (H5LT): Functions that simplify the steps needed to create/read datasets and attributes • HDF5 Image (H5IM): Functions for creating images in HDF5. • HDF5 Table (H5TB): Functions for creating tables (collections of records) in HDF5. • Others … (Dimension Scales, Packet Table) 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 54
  • 55. High Level Programming Model file_id = H5Fcreate ( "test.h5", ……. ); /* Call some High Level function */ H5LTsome_function ( file_id, ...extra parameters ); status = H5Fclose ( file_id ); 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 55
  • 56. HDF5 Hands-On • Need Secure Shell or putty • Everyone is logged in as: workshop • Working directory for each user is specified at top right of Hands-On page • Compile Examples with: h5fc, h5cc • Display Dataset with: 11/6/07 h5dump HDF and HDF-EOS Workshop XI, Landover, MD 56
  • 57. Thank you! This presentation is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration. 11/6/07 HDF and HDF-EOS Workshop XI, Landover, MD 57

Editor's Notes

  1. Format and software for scientific data. HDF5 is a different format from earlier versions of HDF, as is the library. Stores images, multidimensional arrays, tables, etc. That is, you can construct all of these different kinds structures and store them in HDF5. You can also mix and match them in HDF5 files according to your needs. Emphasis on storage and I/O efficiency Both the library and the format are designed to address this. Free and commercial software support As far as HDF5 goes, this is just a goal now. There is commercial support for HDF4, but little if any for HDF5 at this time. We are working with vendors to change this. Emphasis on standards You can store data in HDF5 in a variety of ways, so we try to work with users to encourage them to organize HDF5 files in standard ways. Users from many engineering and scientific fields
  2. Format and software for scientific data. HDF5 is a different format from earlier versions of HDF, as is the library. Stores images, multidimensional arrays, tables, etc. That is, you can construct all of these different kinds structures and store them in HDF5. You can also mix and match them in HDF5 files according to your needs. Emphasis on storage and I/O efficiency Both the library and the format are designed to address this. Free and commercial software support As far as HDF5 goes, this is just a goal now. There is commercial support for HDF4, but little if any for HDF5 at this time. We are working with vendors to change this. Emphasis on standards You can store data in HDF5 in a variety of ways, so we try to work with users to encourage them to organize HDF5 files in standard ways. Users from many engineering and scientific fields
  3. This shows that you can mix objects of different types according to your needs. Typically, there will be metadata stored with objects to indicate what type of object they are. Like HDF4, HDF5 has a grouping structure. The main difference is that every HDF5 file starts with a root group, whereas HDF4 doesn’t need any groups at all.
  4. Here is an example of a basic HDF5 object. Notice that each element in the 3D array is a record with four values in it.
  5. Like HDF4, HDF5 has a grouping structure. The main difference is that every HDF5 file starts with a root group, whereas HDF4 doesn’t need any groups at all.
  6. Like HDF4, HDF5 has a grouping structure. The main difference is that every HDF5 file starts with a root group, whereas HDF4 doesn’t need any groups at all.