The HDF-EOS Tools and Information Center, profile picture
Uploaded byThe HDF-EOS Tools and Information Center
PPT, PDF1,058 views

Introduction to HDF5

The document is an introduction to HDF5, outlining its history, structure, and use cases, particularly in managing large and complex data sets. It covers HDF5's data model, programming interface, and tools for accessing and manipulating data, highlighting its efficiency and versatility across various computing environments. Additionally, it provides examples of operations supported by the API and resources for accessing further information about HDF5.

Downloaded 24 times
Introduction to HDF5 HDF & HDF-EOS Workshop XII October 15, 2008 10/15/08 HDF & HDF-EOS Workshop XII 1 1
Topics Covered - Introduce HDF5 - Describe HDF5 Data and Programming Models - Walk Through Example Code 10/15/08 HDF & HDF-EOS Workshop XII 2 2
For More Information … All workshop slides will be available from: http://hdfeos.org/workshops/ws12/workshop_twelve.php 10/15/08 HDF & HDF-EOS Workshop XII 3
What is HDF5? HDF = Hierarchical Data Format • Data model, library and file format for managing data • Tools for accessing data in the HDF5 format 10/15/08 HDF & HDF-EOS Workshop XII 4
Brief History of HDF 1987 At NCSA (University of Illinois), a task force formed to create an architecture-independent format and library: AEHOO (All Encompassing Hierarchical Object Oriented format) Became HDF Early NASA adopted HDF for Earth Observing System project 1990’s 1996 DOE’s ASC (Advanced Simulation and Computing) Project began collaborating with the HDF group (NCSA) to create “Big HDF” (Increase in computing power of DOE systems at LLNL, LANL and Sandia National labs, required bigger, more complex data files). “Big HDF” became HDF5. 1998 HDF5 was released with support from National Labs, NASA, NCSA 2006 The HDF Group spun off from University of Illinois as non-profit corporation 10/15/08 HDF & HDF-EOS Workshop XII 5
Why HDF5? In one sentence ... 10/15/08 HDF & HDF-EOS Workshop XII 6 6
Answering big questions … Matter and the universe Life and nature August 24, 2001 August 24, 2002 Total Column Ozone (Dobson) 60 385 610 Weather and climate 10/15/08 HDF & HDF-EOS Workshop XII 7 7
… involves big data … 10/15/08 HDF & HDF-EOS Workshop XII 8 8
… varied data … LCI Tutorial 10/15/08 Thanks to Mark HDF & HDF-EOS Workshop XII 9 Miller, LLNL 9
… and complex relationships … SNP Score Contig Summaries Discrepancies Contig Qualities Coverage Depth Trace Reads Aligned bases Read quality Contig Percent match 10/15/08 HDF & HDF-EOS Workshop XII 10 10
… on big computers … … and small computers … 10/15/08 HDF & HDF-EOS Workshop XII 11 11
How do we… • Describe our data? • Read it? Store it? Find it? Share it? Mine it? • Move it into, out of, and between computers and repositories? • Achieve storage and I/O efficiency? • Give applications and tools easy access our data? 10/15/08 HDF & HDF-EOS Workshop XII 12 12
Solution: HDF5! • Can store all kinds of data in a variety of ways • Runs on most systems • Lots of tools to access data • Emphasis on standards (HDF-EOS, CGNS) • Library and format emphasis on I/O efficiency and storage 10/15/08 HDF & HDF-EOS Workshop XII 13
Structure of HDF5 Library Applications Object API (C, F90, C++, Java) Library internals Virtual file I/O File or other “storage” 10/15/08 HDF & HDF-EOS Workshop XII 14
HDF Tools - HDFView and Java Products - Command-line utilities (h5dump, h5ls, h5cc, h5diff, h5repack) 10/15/08 HDF & HDF-EOS Workshop XII 15 15
HDF5 Applications & Domains Examples: Thermonuclear simulations Product modeling Data mining tools Visualization tools Climate models Simulation, visualization, remote sensing… HDF-EOS Virtual File Layer (I/O Drivers) Stdio CGNS HDF5 Data Model & API Split Files MPI I/O Storage HDF5 format 10/15/08 File ASC Custom ? Split metadata File on parallel and raw data files file system User-defined device HDF & HDF-EOS Workshop XII 16 Communities
Lots of Layers in HDF5! “Ogres are like onions.” Shrek  HDF5 Monster?? Just like Shrek, once you get to know HDF5 you will really like it!! 10/15/08 HDF & HDF-EOS Workshop XII 17
The HDF5 Format 10/15/08 HDF & HDF-EOS Workshop XII 18 18
An HDF5 file is a container… …into which you can put your data objects. 10/15/08 lat | lon | temp ----|-----|----12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 te let pa HDF & HDF-EOS Workshop XII 19 19
HDF5 Structures for Organizing Objects “/” (root) “foo” 3-D array lat | lon | temp ----|-----|----12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 palette Table Raster image Raster image 10/15/08 2-D array HDF & HDF-EOS Workshop XII 20 20
HDF5 Data Model Primary Objects • Groups • Datasets Additional ways to organize and annotate data • Attributes • Storage and access properties Everything else is built from these parts. 10/15/08 HDF & HDF-EOS Workshop XII 21 21
HDF5 Dataset Metadata Data Dataspace Rank Dimensions 3 Dim_1 = 4 Dim_2 = 5 Dim_3 = 7 Datatype Integer Storage Info Attributes Time = 32.4 Chunked Pressure = 987 Compressed Temp = 56 10/15/08 HDF & HDF-EOS Workshop XII 22 22
Dataspaces Two roles: • Dataspace contains spatial info about a dataset stored in a file • Rank and dimensions • Permanent part of dataset definition Rank = 2 Dimensions = 4x6 • Partial I/0: Dataspace describes application’s data buffer and data elements participating in I/O Rank = 1 Dimension = 10 10/15/08 HDF & HDF-EOS Workshop XII 23 23
Write – from memory to disk memory 10/15/08 disk HDF & HDF-EOS Workshop XII 24 24
Partial I/O Move just part of a dataset memory disk (a) Slab from a 2D array to the corner of a smaller 2D array Elements in each must be same. 10/15/08 disk memory (b) Regular series of blocks from a 2D array to a contiguous sequence at a certain offset in a 1D array HDF & HDF-EOS Workshop XII 25 25
Datatypes (array elements) • Datatype – how to interpret a data element • Permanent part of the dataset definition • Two classes: atomic and compound 10/15/08 HDF & HDF-EOS Workshop XII 26 26
Datatypes • HDF5 atomic types include: integer & float user-definable (e.g., 13-bit integer) variable length types (e.g., strings) references to objects/dataset regions enumeration - names mapped to integers • HDF5 compound types Comparable to C structs (“records”) Members can be atomic or compound types 10/15/08 HDF & HDF-EOS Workshop XII 27 27
HDF5 dataset: array of records 3 5 Dimensionality: 5 x 3 int8 int4 int16 2x3x2 array of float32 Datatype: Record 10/15/08 HDF & HDF-EOS Workshop XII 28 28
Properties • Properties are characteristics of HDF5 objects that can be modified • Default properties handle most needs • By changing properties can take advantage of the more powerful features in HDF5 10/15/08 HDF & HDF-EOS Workshop XII 29
Special Storage Properties Better subsetting access time; extensible chunked Improves storage efficiency, transmission speed compressed Arrays can be extended in any direction extensible File B split file Dataset “Fred” File A Metadata for Fred 10/15/08 Metadata in one file, raw data in another Data for Fred HDF & HDF-EOS Workshop XII 30 30
Attributes (optional) • Attribute – data of the form “name = value”, attached to an object • Operations similar to dataset operations, but … Not extensible No compression or partial I/O • Can be overwritten, deleted, added during the “life” of a dataset 10/15/08 HDF & HDF-EOS Workshop XII 31 31
HDF5 Dataset (again) Metadata Data Dataspace Rank Dimensions 3 Dim_1 = 4 Dim_2 = 5 Dim_3 = 7 Datatype Integer Storage info Attributes Time = 32.4 Chunked Pressure = 987 Compressed Temp = 56 10/15/08 HDF & HDF-EOS Workshop XII 32 32
Groups • A mechanism for organizing collections • Every file starts with a root group • Similar to UNIX directories A • Can have attributes k 10/15/08 “/” B l m HDF & HDF-EOS Workshop XII 33 33 C
Path to HDF5 Object in a File / (root) /x /foo /foo/temp /foo/bar/temp 10/15/08 foo temp “/” x bar temp HDF & HDF-EOS Workshop XII 34 34
Shared Objects “/” A P C B R P /A/P /B/R /C/P 10/15/08 HDF & HDF-EOS Workshop XII 35 35
Questions So Far? 10/15/08 HDF & HDF-EOS Workshop XII 36
Useful Tools For New Users h5dump: Tool to “dump” or display contents of HDF5 files h5cc, h5c++, h5fc: Scripts to compile applications HDFView: Java browser to view HDF4 and HDF5 files 10/15/08 HDF & HDF-EOS Workshop XII 37
H5dump Command-line Utility To View HDF5 File h5dump [--header] [-a ] [-d <names>] [-g <names>] [-l <names>] [-t <names>] [-p] <file> --header Display header only; no data is displayed. -a <names> Display the specified attribute(s). -d <names> Display the specified dataset(s). -g <names> Display the specified group(s) and all the members. -l <names> Displays the value(s) of the specified soft link(s). -t <names> Display the specified named datatype(s). -p Display properties. <names> is one or more appropriate object names. 10/15/08 HDF & HDF-EOS Workshop XII 38
Example of h5dump Output HDF5 "dset.h5" { GROUP "/" { DATASET "dset" { DATATYPE { H5T_STD_I32BE } DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) } DATA { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24 “/” } } } } 10/15/08 HDF & HDF-EOS Workshop XII 39 ‘dset’
HDF5 Compile Scripts • h5cc – HDF5 C compiler command • h5fc – HDF5 F90 compiler command • h5c++ – HDF5 C++ compiler command To compile: % h5cc h5prog.c % h5fc h5prog.f90 10/15/08 HDF & HDF-EOS Workshop XII 40 40
Compile option: -show -show: displays the compiler commands and options without executing them % h5cc –show Sample_c.c gcc -I/home/packages/hdf5_1.6.6/Linux_2.6/include -UH5_DEBUG_API -DNDEBUG -I/home/packages/szip/static/encoder/Linux2.6-gcc/include -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_POSIX_SOURCE -D_BSD_SOURCE -std=c99 -Wno-long-long -O -fomit-frame-pointer -finline-functions -c Sample_c.c gcc -std=c99 -Wno-long-long -O -fomit-frame-pointer -finline-functions -L/home/packages/szip/static/encoder/Linux2.6-gcc/lib Sample_c.o -L/home/packages/hdf5_1.6.6/Linux_2.6/lib /home/packages/hdf5_1.6.6/Linux_2.6/lib/libhdf5_hl.a /home/packages/hdf5_1.6.6/Linux_2.6/lib/libhdf5.a -lsz -lz -lm -Wl,-rpath -Wl,/home/packages/hdf5_1.6.6/Linux_2.6/lib 10/15/08 HDF & HDF-EOS Workshop XII 41 41
Browsing HDF5 Files with HDFView 10/15/08 HDF & HDF-EOS Workshop XII 42
HDFView Structure of File 10/15/08 Contents of Dataset HDF & HDF-EOS Workshop XII 43
HDFView File Menu 10/15/08 HDF & HDF-EOS Workshop XII 44
10/15/08 HDF & HDF-EOS Workshop XII 45
Simple HDF5 File in HDFView Right-click and select “Open” with mouse Right-click and select “Show Properties” with mouse 10/15/08 HDF & HDF-EOS Workshop XII 46
Simple HDF5 File in HDFView 10/15/08 HDF & HDF-EOS Workshop XII 47
HDF-EOS5 File in HDFView 10/15/08 HDF & HDF-EOS Workshop XII 48
Right-click and select “Open As” with mouse 10/15/08 HDF & HDF-EOS Workshop XII 49
What you can’t see with slides: -Picture displayed instantly -File size is 906,229,176 10/15/08 HDF & HDF-EOS Workshop XII 50
Introduction to HDF5 Programming Model and APIs 10/15/08 HDF & HDF-EOS Workshop XII 51 51
Operations Supported by the API • Create objects (groups, datasets, attributes, complex data types, …) • Assign storage and I/O properties to objects • Perform complex subsetting during read/write • Use variety of I/O “devices” (parallel, remote, etc.) • Transform data during I/O • Make inquiries on file and object structure, content, properties 10/15/08 HDF & HDF-EOS Workshop XII 52 52
General Programming Paradigm • Properties of object are optionally defined  Creation properties  Access property lists • Object is opened or created • Object is accessed, possibly many times • Object is closed 10/15/08 HDF & HDF-EOS Workshop XII 53 53
Order of Operations • An order is imposed on operations by argument dependencies For Example: A file must be opened before a dataset -becausethe dataset open call requires a file handle as an argument. • Objects can be closed in any order. 10/15/08 HDF & HDF-EOS Workshop XII 54 54
The General HDF5 API • Currently C, Fortran 90, Java, and C++ bindings. • C routines begin with prefix H5? ? is a character corresponding to the type of object the function acts on Example Functions: H5D : Dataset interface H5F : File interface e.g., H5Dread e.g., H5Fopen H5S : dataSpace interface e.g., H5Sclose 10/15/08 HDF & HDF-EOS Workshop XII 55 55
HDF5 Defined Types For portability, the HDF5 library has its own defined types: hid_t: hsize_t: hssize_t: object identifiers (native integer) size used for dimensions (unsigned long or unsigned long long) for specifying coordinates and sometimes for dimensions (signed long or signed long long) herr_t: function return value hvl_t: variable length datatype For C, include hdf5.h in your HDF5 application. 10/15/08 HDF & HDF-EOS Workshop XII 56 56
The HDF5 API • For flexibility, the API is extensive  300+ functions Victronix Swiss Army Cybertool 34 • This can be daunting… but there is hope  A few functions can do a lot  Start simple  Build up knowledge as more features are needed 10/15/08 HDF & HDF-EOS Workshop XII 57 57
Basic Functions H5Fcreate (H5Fopen) H5Screate_simple H5Dcreate (H5Dopen) H5Dread, H5Dwrite H5Dclose H5Sclose H5Fclose 10/15/08 create (open) File create dataSpace create (open) Dataset access Dataset close Dataset close dataSpace close File HDF & HDF-EOS Workshop XII 58
Other Common Functions DataSpaces: H5Sselect_hyperslab (Partial I/O) H5Sselect_elements (Partial I/O) Groups: H5Gcreate, H5Gopen, H5Gclose Attributes: H5Acreate, H5Aopen_name, H5Aclose, H5Aread, H5Awrite Property lists: H5Pcreate, H5Pclose H5Pset_chunk, H5Pset_deflate 10/15/08 HDF & HDF-EOS Workshop XII 59
High Level APIs • Included along with the HDF5 library • Simplify steps for creating, writing, and reading objects • Do not entirely ‘wrap’ HDF5 library 10/15/08 HDF & HDF-EOS Workshop XII 60
Example HDF5 Code 10/15/08 HDF & HDF-EOS Workshop XII 61
Steps to Create a File 1. Decide on special properties the file should have • • • Creation properties, like size of user block Access properties, such as metadata cache size Use default properties (H5P_DEFAULT) 2. Create property lists, if necessary 3. Create the file 4. Close the file and the property lists, as needed 10/15/08 HDF & HDF-EOS Workshop XII 62 62
Code: Create a File hid_t herr_t file_id; status; file_id = H5Fcreate ("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); status = H5Fclose (file_id); “/” (root) Note: Return codes not checked for errors in code samples. 10/15/08 HDF & HDF-EOS Workshop XII 63 63
Dataset Components Metadata Data Dataspace Rank Dimensions 3 Dim_1 = 4 Dim_2 = 5 Dim_3 = 7 Datatype Integer Storage info Attributes Time = 32.4 Chunked Pressure = 987 Compressed Temp = 56 10/15/08 HDF & HDF-EOS Workshop XII 64 64
Steps to Create a Dataset 1. Define dataset characteristics • • • Dataspace - 4x6 Datatype – integer Properties if needed, or use H5P_DEFAULT 2. Decide where to put it • Obtain location ID: - Group ID puts it in a Group - File ID puts it in Root Group “/” (root) A 3. Create dataset in file 4. Close everything 10/15/08 HDF & HDF-EOS Workshop XII 65 65
HDF5 Pre-defined Datatype Identifiers HDF5 defines* set of Datatype Identifiers per HDF5 session. For example: C Type HDF5 File Type HDF5 Memory Type int H5T_STD_I32BE H5T_STD_I32LE H5T_NATIVE_INT float H5T_IEEE_F32BE H5T_IEEE_F32LE H5T_NATIVE_FLOAT double H5T_IEEE_F64BE H5T_IEEE_F64LE H5T_NATIVE_DOUBLE * Value of datatype is NOT fixed 10/15/08 HDF & HDF-EOS Workshop XII 66
Pre-defined File Datatype Identifiers Examples: H5T_IEEE_F64LE Eight-byte, little-endian, IEEE floating-point H5T_STD_I32LE Four-byte, little-endian, signed two's complement integer Architecture* Programming Type NOTE: What you see in the file. Name is the same everywhere and explicitly defines a datatype. *STD= “An architecture with a semi-standard type like 2’s complement integer, unsigned integer…” 10/15/08 HDF & HDF-EOS Workshop XII 67
Pre-defined Native Datatypes Examples of predefined native types in C: H5T_NATIVE_INT H5T_NATIVE_FLOAT H5T_NATIVE_UINT H5T_NATIVE_LONG H5T_NATIVE_CHAR (int) (float ) (unsigned int) (long ) (char ) NOTE: Memory types. Different for each machine. Used for reading/writing. 10/15/08 HDF & HDF-EOS Workshop XII 68
Dataset Creation Property List Dataset creation property list: information on how to organize data in storage. Chunked Chunked & compressed H5P_DEFAULT: contiguous 10/15/08 HDF & HDF-EOS Workshop XII 69 69
Code: Create a Dataset 1 2 3 hid_t hsize_t herr_t file_id, dataset_id, dataspace_id; dims[2]; status; 4 file_id = H5Fcreate (”file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); Create a 5 dataspace= 4; dims[0] 6 7 rank dims[1] = 6; dataspace_id = H5Screate_simple (2, dims, NULL); Create a dataset 8 current dims pathname datatype dataset_id = H5Dcreate(file_id,”A",H5T_STD_I32BE, dataspace_id, H5P_DEFAULT); dataspace Terminate access to dataset, dataspace, file 9 status = H5Dclose (dataset_id); 10 status = H5Sclose (dataspace_id); 11 status = H5Fclose (file_id); 10/15/08 property list (default) HDF & HDF-EOS Workshop XII 70 70
Example Code - H5Dwrite Dataset Identifier from H5Dcreate or H5Dopen Memory Datatype status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); 10/15/08 HDF & HDF-EOS Workshop XII 71
Example Code – H5Dwrite status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); Data Transfer Property List (MPI I/O, Transformations, …) Memory Dataspace File Dataspace H5S_ALL selects entire dataspace 10/15/08 HDF & HDF-EOS Workshop XII 72
Partial I/O Memory Dataspace H5S_ALL File Dataspace (disk) H5S_ALL Get a Dataspace: H5Screate_simple H5Dget_space Modify Dataspace: H5Sselect_hyperslab H5Sselect_elements 10/15/08 HDF & HDF-EOS Workshop XII 73
Example Code – H5Dread status = H5Dread (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_rdata); 10/15/08 HDF & HDF-EOS Workshop XII 74
High Level APIs: HDF5 Lite (H5LT) #include "H5LT.h" … file_id = H5Fcreate (“file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); status = H5LTmake_dataset (file_id,“A", 2, dims, H5T_STD_I32BE, data); status = H5Fclose (file_id); 10/15/08 HDF & HDF-EOS Workshop XII 75
High Level APIs • • • • • 10/15/08 HDF5 Lite HDF5 Image HDF5 Table HDF5 Dimension Scales HDF5 Packet Table HDF & HDF-EOS Workshop XII 76
Example: Create a Group “/” (root) A B 4x6 array of integers file.h5 10/15/08 HDF & HDF-EOS Workshop XII 77 77
Steps to Create a Group 1. Decide where to put it – “root group” • Obtain location ID 2. Decide name – “B” 3. Create group in file 4. (Eventually) close the group. 10/15/08 HDF & HDF-EOS Workshop XII 78 78
Code: Create a Group hid_t file_id, group_id; ... /* Open “file.h5” */ file_id = H5Fopen (“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT); /* Create group "/B" in file. */ group_id = H5Gcreate (file_id,"B",0); Size hint for number of bytes to store names of objects. 0=default /* Close group and file. */ status = H5Gclose (group_id); status = H5Fclose (file_id); 10/15/08 HDF & HDF-EOS Workshop XII 79 79
Thank you! This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 10/15/08 HDF & HDF-EOS Workshop XII 80

More Related Content

PPTX
Information Intermediaries
byDave Reynolds
 
PPTX
Subject Headings Authority File for Filipiniana materials / Annelyn C. Kim an...
byCILIP MDG
 
PPT
Technical processing section
bymrafique
 
PPT
Planning and Implementing a Digital Library Project
byJenn Riley
 
PDF
Archival Acquisition (LIS 170)
byRoy Santos Necesario
 
PDF
Sap fiori push notifications
byAnkur Garg
 
PPT
Fundamentals of RDA: Resource Description & Access
byGLA: Technical Services Interest Group (TSIG)
 
PPTX
Building a library disaster preparedness plan
byPhilippine Association of Academic/Research Librarians
 
Information Intermediaries
byDave Reynolds
 
Subject Headings Authority File for Filipiniana materials / Annelyn C. Kim an...
byCILIP MDG
 
Technical processing section
bymrafique
 
Planning and Implementing a Digital Library Project
byJenn Riley
 
Archival Acquisition (LIS 170)
byRoy Santos Necesario
 
Sap fiori push notifications
byAnkur Garg
 
Fundamentals of RDA: Resource Description & Access
byGLA: Technical Services Interest Group (TSIG)
 
Building a library disaster preparedness plan
byPhilippine Association of Academic/Research Librarians
 

Viewers also liked

PPTX
ICESat-2 Metadata and Status
byThe HDF-EOS Tools and Information Center
 
PPT
Migrating from HDF5 1.6 to 1.8
byThe HDF-EOS Tools and Information Center
 
PPTX
HDF Cloud Services
byThe HDF-EOS Tools and Information Center
 
PPTX
Utilizing HDF4 File Content Maps for the Cloud Computing
byThe HDF-EOS Tools and Information Center
 
PPTX
Scientific Computing and Visualization using HDF
byThe HDF-EOS Tools and Information Center
 
PPTX
SPD and KEA: HDF5 based file formats for Earth Observation
byThe HDF-EOS Tools and Information Center
 
PPTX
NEON HDF5
byThe HDF-EOS Tools and Information Center
 
PPT
Using visualization tools to access HDF data via OPeNDAP
byThe HDF-EOS Tools and Information Center
 
PPT
Shifting the Burden from the User to the Data Provider
byThe HDF-EOS Tools and Information Center
 
PPTX
HDF5 OPeNDAP project update and demo
byThe HDF-EOS Tools and Information Center
 
PPTX
Support for NPP/NPOESS by The HDF Group
byThe HDF-EOS Tools and Information Center
 
PPT
HDF Update
byThe HDF-EOS Tools and Information Center
 
PPT
Proposal for adding Named Dimensions to HDF5 Arrays
byThe HDF-EOS Tools and Information Center
 
PPTX
HDF Update 2016
byThe HDF-EOS Tools and Information Center
 
PPTX
Pilot Project for HDF5 Metadata Structures for SWOT
byThe HDF-EOS Tools and Information Center
 
PDF
HDF and HDF-EOS Experiences and Applications
byThe HDF-EOS Tools and Information Center
 
PPT
Profile of HDF-EOS5 Files
byThe HDF-EOS Tools and Information Center
 
PDF
ENVI/IDL for HDF
byThe HDF-EOS Tools and Information Center
 
PPT
EOSDIS Status
byThe HDF-EOS Tools and Information Center
 
PDF
Breakthrough Listen
byThe HDF-EOS Tools and Information Center
 
ICESat-2 Metadata and Status
byThe HDF-EOS Tools and Information Center
 
Migrating from HDF5 1.6 to 1.8
byThe HDF-EOS Tools and Information Center
 
HDF Cloud Services
byThe HDF-EOS Tools and Information Center
 
Utilizing HDF4 File Content Maps for the Cloud Computing
byThe HDF-EOS Tools and Information Center
 
Scientific Computing and Visualization using HDF
byThe HDF-EOS Tools and Information Center
 
SPD and KEA: HDF5 based file formats for Earth Observation
byThe HDF-EOS Tools and Information Center
 
NEON HDF5
byThe HDF-EOS Tools and Information Center
 
Using visualization tools to access HDF data via OPeNDAP
byThe HDF-EOS Tools and Information Center
 
Shifting the Burden from the User to the Data Provider
byThe HDF-EOS Tools and Information Center
 
HDF5 OPeNDAP project update and demo
byThe HDF-EOS Tools and Information Center
 
Support for NPP/NPOESS by The HDF Group
byThe HDF-EOS Tools and Information Center
 
HDF Update
byThe HDF-EOS Tools and Information Center
 
Proposal for adding Named Dimensions to HDF5 Arrays
byThe HDF-EOS Tools and Information Center
 
HDF Update 2016
byThe HDF-EOS Tools and Information Center
 
Pilot Project for HDF5 Metadata Structures for SWOT
byThe HDF-EOS Tools and Information Center
 
HDF and HDF-EOS Experiences and Applications
byThe HDF-EOS Tools and Information Center
 
Profile of HDF-EOS5 Files
byThe HDF-EOS Tools and Information Center
 
ENVI/IDL for HDF
byThe HDF-EOS Tools and Information Center
 
EOSDIS Status
byThe HDF-EOS Tools and Information Center
 
Breakthrough Listen
byThe HDF-EOS Tools and Information Center
 

Similar to Introduction to HDF5

PPTX
Introduction to HDF5 Data and Programming Models
byThe HDF-EOS Tools and Information Center
 
PPTX
Advanced HDF5 Features
byThe HDF-EOS Tools and Information Center
 
PPT
HDF5 Advanced Topics - Datatypes and Partial I/O
byThe HDF-EOS Tools and Information Center
 
PPT
Introduction to HDF5
byThe HDF-EOS Tools and Information Center
 
PPT
Introduction to HDF5
byThe HDF-EOS Tools and Information Center
 
PPT
HDF5 Tools
byThe HDF-EOS Tools and Information Center
 
PPT
Introduction to HDF5
byThe HDF-EOS Tools and Information Center
 
PPT
Introduction to HDF5 Data Model, Programming Model and Library APIs
byThe HDF-EOS Tools and Information Center
 
PPT
HDF5 Advanced Topics
byThe HDF-EOS Tools and Information Center
 
PDF
Introduction to HDF5 Data Model, Programming Model and Library APIs
byThe HDF-EOS Tools and Information Center
 
PPT
HDF5 Tools Update
byThe HDF-EOS Tools and Information Center
 
PPTX
HDF Tools Tutorial
byThe HDF-EOS Tools and Information Center
 
PPT
HDF5 Tools Updates
byThe HDF-EOS Tools and Information Center
 
PDF
HDF5 Documentation
byThe HDF-EOS Tools and Information Center
 
PPT
Hdf5 intro
bySmith Kim
 
PPT
Advanced HDF5 Features
byThe HDF-EOS Tools and Information Center
 
PPTX
Advanced HDF5 Features
byThe HDF-EOS Tools and Information Center
 
PDF
Introduction to HDF5 Data Model, Programming Model and Library APIs
byThe HDF-EOS Tools and Information Center
 
PPT
What will be new in HDF5?
byThe HDF-EOS Tools and Information Center
 
PPT
HDF5 Tools
byThe HDF-EOS Tools and Information Center
 
Introduction to HDF5 Data and Programming Models
byThe HDF-EOS Tools and Information Center
 
Advanced HDF5 Features
byThe HDF-EOS Tools and Information Center
 
HDF5 Advanced Topics - Datatypes and Partial I/O
byThe HDF-EOS Tools and Information Center
 
Introduction to HDF5
byThe HDF-EOS Tools and Information Center
 
Introduction to HDF5
byThe HDF-EOS Tools and Information Center
 
HDF5 Tools
byThe HDF-EOS Tools and Information Center
 
Introduction to HDF5
byThe HDF-EOS Tools and Information Center
 
Introduction to HDF5 Data Model, Programming Model and Library APIs
byThe HDF-EOS Tools and Information Center
 
HDF5 Advanced Topics
byThe HDF-EOS Tools and Information Center
 
Introduction to HDF5 Data Model, Programming Model and Library APIs
byThe HDF-EOS Tools and Information Center
 
HDF5 Tools Update
byThe HDF-EOS Tools and Information Center
 
HDF Tools Tutorial
byThe HDF-EOS Tools and Information Center
 
HDF5 Tools Updates
byThe HDF-EOS Tools and Information Center
 
HDF5 Documentation
byThe HDF-EOS Tools and Information Center
 
Hdf5 intro
bySmith Kim
 
Advanced HDF5 Features
byThe HDF-EOS Tools and Information Center
 
Advanced HDF5 Features
byThe HDF-EOS Tools and Information Center
 
Introduction to HDF5 Data Model, Programming Model and Library APIs
byThe HDF-EOS Tools and Information Center
 
What will be new in HDF5?
byThe HDF-EOS Tools and Information Center
 
HDF5 Tools
byThe HDF-EOS Tools and Information Center
 

More from The HDF-EOS Tools and Information Center

PPTX
Highly Scalable Data Service (HSDS) Performance Features
byThe HDF-EOS Tools and Information Center
 
PDF
Accessing HDF5 data in the cloud with HSDS
byThe HDF-EOS Tools and Information Center
 
PDF
Creating Cloud-Optimized HDF5 Files
byThe HDF-EOS Tools and Information Center
 
PDF
Cloud-Optimized HDF5 Files
byThe HDF-EOS Tools and Information Center
 
PDF
H5Coro: The Cloud-Optimized Read-Only Library
byThe HDF-EOS Tools and Information Center
 
PPTX
HDF - Current status and Future Directions
byThe HDF-EOS Tools and Information Center
 
PPTX
Upcoming New HDF5 Features: Multi-threading, sparse data storage, and encrypt...
byThe HDF-EOS Tools and Information Center
 
PPTX
The State of HDF5 / Dana Robinson / The HDF Group
byThe HDF-EOS Tools and Information Center
 
PPTX
The State of HDF
byThe HDF-EOS Tools and Information Center
 
PPTX
MATLAB Modernization on HDF5 1.10
byThe HDF-EOS Tools and Information Center
 
PPSX
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
byThe HDF-EOS Tools and Information Center
 
PDF
Cloud Optimized HDF5 for the ICESat-2 mission
byThe HDF-EOS Tools and Information Center
 
PDF
Cloud-Optimized HDF5 Files - Current Status
byThe HDF-EOS Tools and Information Center
 
PDF
Using a Hierarchical Data Format v5 file as Zarr v3 Shard
byThe HDF-EOS Tools and Information Center
 
PPTX
Hyrax: Serving Data from S3
byThe HDF-EOS Tools and Information Center
 
PDF
HDF5 2.0: Cloud Optimized from the Start
byThe HDF-EOS Tools and Information Center
 
PPTX
HDF5 OPeNDAP Handler Updates, and Performance Discussion
byThe HDF-EOS Tools and Information Center
 
PDF
HDF - Current status and Future Directions
byThe HDF-EOS Tools and Information Center
 
PPSX
HDFEOS.org User Analsys, Updates, and Future
byThe HDF-EOS Tools and Information Center
 
PPTX
Access HDF Data in the Cloud via OPeNDAP Web Service
byThe HDF-EOS Tools and Information Center
 
Highly Scalable Data Service (HSDS) Performance Features
byThe HDF-EOS Tools and Information Center
 
Accessing HDF5 data in the cloud with HSDS
byThe HDF-EOS Tools and Information Center
 
Creating Cloud-Optimized HDF5 Files
byThe HDF-EOS Tools and Information Center
 
Cloud-Optimized HDF5 Files
byThe HDF-EOS Tools and Information Center
 
H5Coro: The Cloud-Optimized Read-Only Library
byThe HDF-EOS Tools and Information Center
 
HDF - Current status and Future Directions
byThe HDF-EOS Tools and Information Center
 
Upcoming New HDF5 Features: Multi-threading, sparse data storage, and encrypt...
byThe HDF-EOS Tools and Information Center
 
The State of HDF5 / Dana Robinson / The HDF Group
byThe HDF-EOS Tools and Information Center
 
The State of HDF
byThe HDF-EOS Tools and Information Center
 
MATLAB Modernization on HDF5 1.10
byThe HDF-EOS Tools and Information Center
 
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
byThe HDF-EOS Tools and Information Center
 
Cloud Optimized HDF5 for the ICESat-2 mission
byThe HDF-EOS Tools and Information Center
 
Cloud-Optimized HDF5 Files - Current Status
byThe HDF-EOS Tools and Information Center
 
Using a Hierarchical Data Format v5 file as Zarr v3 Shard
byThe HDF-EOS Tools and Information Center
 
Hyrax: Serving Data from S3
byThe HDF-EOS Tools and Information Center
 
HDF5 2.0: Cloud Optimized from the Start
byThe HDF-EOS Tools and Information Center
 
HDF5 OPeNDAP Handler Updates, and Performance Discussion
byThe HDF-EOS Tools and Information Center
 
HDF - Current status and Future Directions
byThe HDF-EOS Tools and Information Center
 
HDFEOS.org User Analsys, Updates, and Future
byThe HDF-EOS Tools and Information Center
 
Access HDF Data in the Cloud via OPeNDAP Web Service
byThe HDF-EOS Tools and Information Center
 

Recently uploaded

PDF
ODSC West 2025 OpenMetadata Workshop.pdf
byOpenMetadata
 
PPTX
Integration Everywhere - Supercharge Your AI Agents with Custom Connectors.pptx
byEric Shupps
 
PPTX
UiPath Data Fabric From Data Silos to Unified Flows.pptx
bysuhanisingh58689
 
PDF
Igalia and WebKit: Status update and plans (2025)
byIgalia
 
PPTX
Stop Worrying, Start Wiring: Azure Serverless for the AI Era
byMuralidharan Deenathayalan
 
PDF
Devfest Harare 2025 Slides [GDG Harare] - Combined Slide Deck
byGoogle Developer Group - Harare
 
PDF
Session 8 Specialized AI Associate Series: Fundamentals of Model Training
byDianaGray10
 
PDF
Integrare AI e Automazione con UiPath Agent Builder
byUiPathCommunity
 
PDF
n8n - Next Generation Workflow Automation with Open Source
byDaisuke Masuda
 
PDF
API Days Australia - API Management in the AI Era
byNilesh Gule
 
PDF
Linux Day Prato 2025: NixOS spiegato ammiocuggino - NixOS for Human Beings
byPeter Bittner
 
PPTX
Artificial Intelligence lecture slides.pptx
bymaimelabernard6
 
PPTX
Unitree H2 Humanoid Robot: Redefining Robotics with Advanced Dexterity and In...
byDronevex
 
PDF
Computer Virus and their classifications ppt
byignitedminds222
 
PDF
How does a computer vision development company help retailers decode consumer...
byNextbrain Technologies
 
DOCX
Formula 3 - MACO using general limit.docx
byMarkus Janssen
 
PPTX
Building Smarter Integrations with MuleSoft_ MCP Server and Cursor in Action ...
byKunal Gupta
 
PDF
Dev Dives: Unlocking Enterprise Data with UiPath Data Fabric
byUiPathCommunity
 
PPTX
Assignment Review and Accessibility Audit Findings.pptx
byJulia Undeutsch
 
PDF
UiPath Community Day UNI - Nuevos productos de UiPath.pdf
byfelixluen
 
ODSC West 2025 OpenMetadata Workshop.pdf
byOpenMetadata
 
Integration Everywhere - Supercharge Your AI Agents with Custom Connectors.pptx
byEric Shupps
 
UiPath Data Fabric From Data Silos to Unified Flows.pptx
bysuhanisingh58689
 
Igalia and WebKit: Status update and plans (2025)
byIgalia
 
Stop Worrying, Start Wiring: Azure Serverless for the AI Era
byMuralidharan Deenathayalan
 
Devfest Harare 2025 Slides [GDG Harare] - Combined Slide Deck
byGoogle Developer Group - Harare
 
Session 8 Specialized AI Associate Series: Fundamentals of Model Training
byDianaGray10
 
Integrare AI e Automazione con UiPath Agent Builder
byUiPathCommunity
 
n8n - Next Generation Workflow Automation with Open Source
byDaisuke Masuda
 
API Days Australia - API Management in the AI Era
byNilesh Gule
 
Linux Day Prato 2025: NixOS spiegato ammiocuggino - NixOS for Human Beings
byPeter Bittner
 
Artificial Intelligence lecture slides.pptx
bymaimelabernard6
 
Unitree H2 Humanoid Robot: Redefining Robotics with Advanced Dexterity and In...
byDronevex
 
Computer Virus and their classifications ppt
byignitedminds222
 
How does a computer vision development company help retailers decode consumer...
byNextbrain Technologies
 
Formula 3 - MACO using general limit.docx
byMarkus Janssen
 
Building Smarter Integrations with MuleSoft_ MCP Server and Cursor in Action ...
byKunal Gupta
 
Dev Dives: Unlocking Enterprise Data with UiPath Data Fabric
byUiPathCommunity
 
Assignment Review and Accessibility Audit Findings.pptx
byJulia Undeutsch
 
UiPath Community Day UNI - Nuevos productos de UiPath.pdf
byfelixluen
 
In this document
Powered by AI

Overview of the HDF5 workshop, its objectives, and resources available for further information.

Explanation of HDF5 as a hierarchical data format, its data model, library, and file format.

HDF5's role in addressing complex data questions and facilitating management of large data sets.

Highlight of HDF5's capabilities such as supporting diverse data types and ensuring efficient I/O operations.

Overview of the HDF5 library structure and essential tools available for data manipulation.

Broad range of HDF5 applications across various domains, including simulations and climate modeling.

Description of HDF5 file structure and organization of data objects.

Primary objects in the HDF5 data model, including groups and datasets, and explanation of datatypes. Insight into HDF5 datasets, their metadata, and how data is organized and stored.

Discussion on customizable properties of HDF5 objects that enhance data management.

Attributes in HDF5 datasets and their functionalities and constraints.

Understanding the organization of data into groups in HDF5 and navigating the file structure.

Introduction to essential tools for accessing and manipulating HDF5 files, including h5dump.

Usage of HDFView for browsing and interacting with HDF5 files, including file navigation features.

Overview of HDF5 programming models and APIs, including operations supported and example usage. Step-by-step guidance on creating datasets and groups within HDF5, showcasing coding examples.

Closing remarks on the workshop and acknowledgment of supporting organizations.

Introduction to HDF5

  • 1.
    Introduction to HDF5 HDF& HDF-EOS Workshop XII October 15, 2008 10/15/08 HDF & HDF-EOS Workshop XII 1 1
  • 2.
    Topics Covered - IntroduceHDF5 - Describe HDF5 Data and Programming Models - Walk Through Example Code 10/15/08 HDF & HDF-EOS Workshop XII 2 2
  • 3.
    For More Information… All workshop slides will be available from: http://hdfeos.org/workshops/ws12/workshop_twelve.php 10/15/08 HDF & HDF-EOS Workshop XII 3
  • 4.
    What is HDF5? HDF= Hierarchical Data Format • Data model, library and file format for managing data • Tools for accessing data in the HDF5 format 10/15/08 HDF & HDF-EOS Workshop XII 4
  • 5.
    Brief History ofHDF 1987 At NCSA (University of Illinois), a task force formed to create an architecture-independent format and library: AEHOO (All Encompassing Hierarchical Object Oriented format) Became HDF Early NASA adopted HDF for Earth Observing System project 1990’s 1996 DOE’s ASC (Advanced Simulation and Computing) Project began collaborating with the HDF group (NCSA) to create “Big HDF” (Increase in computing power of DOE systems at LLNL, LANL and Sandia National labs, required bigger, more complex data files). “Big HDF” became HDF5. 1998 HDF5 was released with support from National Labs, NASA, NCSA 2006 The HDF Group spun off from University of Illinois as non-profit corporation 10/15/08 HDF & HDF-EOS Workshop XII 5
  • 6.
    Why HDF5? In onesentence ... 10/15/08 HDF & HDF-EOS Workshop XII 6 6
  • 7.
    Answering big questions… Matter and the universe Life and nature August 24, 2001 August 24, 2002 Total Column Ozone (Dobson) 60 385 610 Weather and climate 10/15/08 HDF & HDF-EOS Workshop XII 7 7
  • 8.
    … involves bigdata … 10/15/08 HDF & HDF-EOS Workshop XII 8 8
  • 9.
    … varied data… LCI Tutorial 10/15/08 Thanks to Mark HDF & HDF-EOS Workshop XII 9 Miller, LLNL 9
  • 10.
    … and complexrelationships … SNP Score Contig Summaries Discrepancies Contig Qualities Coverage Depth Trace Reads Aligned bases Read quality Contig Percent match 10/15/08 HDF & HDF-EOS Workshop XII 10 10
  • 11.
    … on bigcomputers … … and small computers … 10/15/08 HDF & HDF-EOS Workshop XII 11 11
  • 12.
    How do we… •Describe our data? • Read it? Store it? Find it? Share it? Mine it? • Move it into, out of, and between computers and repositories? • Achieve storage and I/O efficiency? • Give applications and tools easy access our data? 10/15/08 HDF & HDF-EOS Workshop XII 12 12
  • 13.
    Solution: HDF5! • Canstore all kinds of data in a variety of ways • Runs on most systems • Lots of tools to access data • Emphasis on standards (HDF-EOS, CGNS) • Library and format emphasis on I/O efficiency and storage 10/15/08 HDF & HDF-EOS Workshop XII 13
  • 14.
    Structure of HDF5Library Applications Object API (C, F90, C++, Java) Library internals Virtual file I/O File or other “storage” 10/15/08 HDF & HDF-EOS Workshop XII 14
  • 15.
    HDF Tools - HDFViewand Java Products - Command-line utilities (h5dump, h5ls, h5cc, h5diff, h5repack) 10/15/08 HDF & HDF-EOS Workshop XII 15 15
  • 16.
    HDF5 Applications &Domains Examples: Thermonuclear simulations Product modeling Data mining tools Visualization tools Climate models Simulation, visualization, remote sensing… HDF-EOS Virtual File Layer (I/O Drivers) Stdio CGNS HDF5 Data Model & API Split Files MPI I/O Storage HDF5 format 10/15/08 File ASC Custom ? Split metadata File on parallel and raw data files file system User-defined device HDF & HDF-EOS Workshop XII 16 Communities
  • 17.
    Lots of Layersin HDF5! “Ogres are like onions.” Shrek  HDF5 Monster?? Just like Shrek, once you get to know HDF5 you will really like it!! 10/15/08 HDF & HDF-EOS Workshop XII 17
  • 18.
    The HDF5 Format 10/15/08 HDF& HDF-EOS Workshop XII 18 18
  • 19.
    An HDF5 fileis a container… …into which you can put your data objects. 10/15/08 lat | lon | temp ----|-----|----12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 te let pa HDF & HDF-EOS Workshop XII 19 19
  • 20.
    HDF5 Structures forOrganizing Objects “/” (root) “foo” 3-D array lat | lon | temp ----|-----|----12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 palette Table Raster image Raster image 10/15/08 2-D array HDF & HDF-EOS Workshop XII 20 20
  • 21.
    HDF5 Data Model PrimaryObjects • Groups • Datasets Additional ways to organize and annotate data • Attributes • Storage and access properties Everything else is built from these parts. 10/15/08 HDF & HDF-EOS Workshop XII 21 21
  • 22.
    HDF5 Dataset Metadata Data Dataspace Rank Dimensions 3 Dim_1= 4 Dim_2 = 5 Dim_3 = 7 Datatype Integer Storage Info Attributes Time = 32.4 Chunked Pressure = 987 Compressed Temp = 56 10/15/08 HDF & HDF-EOS Workshop XII 22 22
  • 23.
    Dataspaces Two roles: • Dataspacecontains spatial info about a dataset stored in a file • Rank and dimensions • Permanent part of dataset definition Rank = 2 Dimensions = 4x6 • Partial I/0: Dataspace describes application’s data buffer and data elements participating in I/O Rank = 1 Dimension = 10 10/15/08 HDF & HDF-EOS Workshop XII 23 23
  • 24.
    Write – frommemory to disk memory 10/15/08 disk HDF & HDF-EOS Workshop XII 24 24
  • 25.
    Partial I/O Move justpart of a dataset memory disk (a) Slab from a 2D array to the corner of a smaller 2D array Elements in each must be same. 10/15/08 disk memory (b) Regular series of blocks from a 2D array to a contiguous sequence at a certain offset in a 1D array HDF & HDF-EOS Workshop XII 25 25
  • 26.
    Datatypes (array elements) •Datatype – how to interpret a data element • Permanent part of the dataset definition • Two classes: atomic and compound 10/15/08 HDF & HDF-EOS Workshop XII 26 26
  • 27.
    Datatypes • HDF5 atomictypes include: integer & float user-definable (e.g., 13-bit integer) variable length types (e.g., strings) references to objects/dataset regions enumeration - names mapped to integers • HDF5 compound types Comparable to C structs (“records”) Members can be atomic or compound types 10/15/08 HDF & HDF-EOS Workshop XII 27 27
  • 28.
    HDF5 dataset: arrayof records 3 5 Dimensionality: 5 x 3 int8 int4 int16 2x3x2 array of float32 Datatype: Record 10/15/08 HDF & HDF-EOS Workshop XII 28 28
  • 29.
    Properties • Properties arecharacteristics of HDF5 objects that can be modified • Default properties handle most needs • By changing properties can take advantage of the more powerful features in HDF5 10/15/08 HDF & HDF-EOS Workshop XII 29
  • 30.
    Special Storage Properties Bettersubsetting access time; extensible chunked Improves storage efficiency, transmission speed compressed Arrays can be extended in any direction extensible File B split file Dataset “Fred” File A Metadata for Fred 10/15/08 Metadata in one file, raw data in another Data for Fred HDF & HDF-EOS Workshop XII 30 30
  • 31.
    Attributes (optional) • Attribute– data of the form “name = value”, attached to an object • Operations similar to dataset operations, but … Not extensible No compression or partial I/O • Can be overwritten, deleted, added during the “life” of a dataset 10/15/08 HDF & HDF-EOS Workshop XII 31 31
  • 32.
    HDF5 Dataset (again) Metadata Data Dataspace RankDimensions 3 Dim_1 = 4 Dim_2 = 5 Dim_3 = 7 Datatype Integer Storage info Attributes Time = 32.4 Chunked Pressure = 987 Compressed Temp = 56 10/15/08 HDF & HDF-EOS Workshop XII 32 32
  • 33.
    Groups • A mechanismfor organizing collections • Every file starts with a root group • Similar to UNIX directories A • Can have attributes k 10/15/08 “/” B l m HDF & HDF-EOS Workshop XII 33 33 C
  • 34.
    Path to HDF5Object in a File / (root) /x /foo /foo/temp /foo/bar/temp 10/15/08 foo temp “/” x bar temp HDF & HDF-EOS Workshop XII 34 34
  • 35.
  • 36.
    Questions So Far? 10/15/08 HDF& HDF-EOS Workshop XII 36
  • 37.
    Useful Tools ForNew Users h5dump: Tool to “dump” or display contents of HDF5 files h5cc, h5c++, h5fc: Scripts to compile applications HDFView: Java browser to view HDF4 and HDF5 files 10/15/08 HDF & HDF-EOS Workshop XII 37
  • 38.
    H5dump Command-line UtilityTo View HDF5 File h5dump [--header] [-a ] [-d <names>] [-g <names>] [-l <names>] [-t <names>] [-p] <file> --header Display header only; no data is displayed. -a <names> Display the specified attribute(s). -d <names> Display the specified dataset(s). -g <names> Display the specified group(s) and all the members. -l <names> Displays the value(s) of the specified soft link(s). -t <names> Display the specified named datatype(s). -p Display properties. <names> is one or more appropriate object names. 10/15/08 HDF & HDF-EOS Workshop XII 38
  • 39.
    Example of h5dumpOutput HDF5 "dset.h5" { GROUP "/" { DATASET "dset" { DATATYPE { H5T_STD_I32BE } DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) } DATA { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24 “/” } } } } 10/15/08 HDF & HDF-EOS Workshop XII 39 ‘dset’
  • 40.
    HDF5 Compile Scripts •h5cc – HDF5 C compiler command • h5fc – HDF5 F90 compiler command • h5c++ – HDF5 C++ compiler command To compile: % h5cc h5prog.c % h5fc h5prog.f90 10/15/08 HDF & HDF-EOS Workshop XII 40 40
  • 41.
    Compile option: -show -show:displays the compiler commands and options without executing them % h5cc –show Sample_c.c gcc -I/home/packages/hdf5_1.6.6/Linux_2.6/include -UH5_DEBUG_API -DNDEBUG -I/home/packages/szip/static/encoder/Linux2.6-gcc/include -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_POSIX_SOURCE -D_BSD_SOURCE -std=c99 -Wno-long-long -O -fomit-frame-pointer -finline-functions -c Sample_c.c gcc -std=c99 -Wno-long-long -O -fomit-frame-pointer -finline-functions -L/home/packages/szip/static/encoder/Linux2.6-gcc/lib Sample_c.o -L/home/packages/hdf5_1.6.6/Linux_2.6/lib /home/packages/hdf5_1.6.6/Linux_2.6/lib/libhdf5_hl.a /home/packages/hdf5_1.6.6/Linux_2.6/lib/libhdf5.a -lsz -lz -lm -Wl,-rpath -Wl,/home/packages/hdf5_1.6.6/Linux_2.6/lib 10/15/08 HDF & HDF-EOS Workshop XII 41 41
  • 42.
    Browsing HDF5 Fileswith HDFView 10/15/08 HDF & HDF-EOS Workshop XII 42
  • 43.
    HDFView Structure of File 10/15/08 Contents ofDataset HDF & HDF-EOS Workshop XII 43
  • 44.
    HDFView File Menu 10/15/08 HDF& HDF-EOS Workshop XII 44
  • 45.
    10/15/08 HDF & HDF-EOSWorkshop XII 45
  • 46.
    Simple HDF5 Filein HDFView Right-click and select “Open” with mouse Right-click and select “Show Properties” with mouse 10/15/08 HDF & HDF-EOS Workshop XII 46
  • 47.
    Simple HDF5 Filein HDFView 10/15/08 HDF & HDF-EOS Workshop XII 47
  • 48.
    HDF-EOS5 File inHDFView 10/15/08 HDF & HDF-EOS Workshop XII 48
  • 49.
    Right-click and select “OpenAs” with mouse 10/15/08 HDF & HDF-EOS Workshop XII 49
  • 50.
    What you can’tsee with slides: -Picture displayed instantly -File size is 906,229,176 10/15/08 HDF & HDF-EOS Workshop XII 50
  • 51.
    Introduction to HDF5 ProgrammingModel and APIs 10/15/08 HDF & HDF-EOS Workshop XII 51 51
  • 52.
    Operations Supported bythe API • Create objects (groups, datasets, attributes, complex data types, …) • Assign storage and I/O properties to objects • Perform complex subsetting during read/write • Use variety of I/O “devices” (parallel, remote, etc.) • Transform data during I/O • Make inquiries on file and object structure, content, properties 10/15/08 HDF & HDF-EOS Workshop XII 52 52
  • 53.
    General Programming Paradigm •Properties of object are optionally defined  Creation properties  Access property lists • Object is opened or created • Object is accessed, possibly many times • Object is closed 10/15/08 HDF & HDF-EOS Workshop XII 53 53
  • 54.
    Order of Operations •An order is imposed on operations by argument dependencies For Example: A file must be opened before a dataset -becausethe dataset open call requires a file handle as an argument. • Objects can be closed in any order. 10/15/08 HDF & HDF-EOS Workshop XII 54 54
  • 55.
    The General HDF5API • Currently C, Fortran 90, Java, and C++ bindings. • C routines begin with prefix H5? ? is a character corresponding to the type of object the function acts on Example Functions: H5D : Dataset interface H5F : File interface e.g., H5Dread e.g., H5Fopen H5S : dataSpace interface e.g., H5Sclose 10/15/08 HDF & HDF-EOS Workshop XII 55 55
  • 56.
    HDF5 Defined Types Forportability, the HDF5 library has its own defined types: hid_t: hsize_t: hssize_t: object identifiers (native integer) size used for dimensions (unsigned long or unsigned long long) for specifying coordinates and sometimes for dimensions (signed long or signed long long) herr_t: function return value hvl_t: variable length datatype For C, include hdf5.h in your HDF5 application. 10/15/08 HDF & HDF-EOS Workshop XII 56 56
  • 57.
    The HDF5 API •For flexibility, the API is extensive  300+ functions Victronix Swiss Army Cybertool 34 • This can be daunting… but there is hope  A few functions can do a lot  Start simple  Build up knowledge as more features are needed 10/15/08 HDF & HDF-EOS Workshop XII 57 57
  • 58.
    Basic Functions H5Fcreate (H5Fopen) H5Screate_simple H5Dcreate(H5Dopen) H5Dread, H5Dwrite H5Dclose H5Sclose H5Fclose 10/15/08 create (open) File create dataSpace create (open) Dataset access Dataset close Dataset close dataSpace close File HDF & HDF-EOS Workshop XII 58
  • 59.
    Other Common Functions DataSpaces: H5Sselect_hyperslab(Partial I/O) H5Sselect_elements (Partial I/O) Groups: H5Gcreate, H5Gopen, H5Gclose Attributes: H5Acreate, H5Aopen_name, H5Aclose, H5Aread, H5Awrite Property lists: H5Pcreate, H5Pclose H5Pset_chunk, H5Pset_deflate 10/15/08 HDF & HDF-EOS Workshop XII 59
  • 60.
    High Level APIs •Included along with the HDF5 library • Simplify steps for creating, writing, and reading objects • Do not entirely ‘wrap’ HDF5 library 10/15/08 HDF & HDF-EOS Workshop XII 60
  • 61.
    Example HDF5 Code 10/15/08 HDF& HDF-EOS Workshop XII 61
  • 62.
    Steps to Createa File 1. Decide on special properties the file should have • • • Creation properties, like size of user block Access properties, such as metadata cache size Use default properties (H5P_DEFAULT) 2. Create property lists, if necessary 3. Create the file 4. Close the file and the property lists, as needed 10/15/08 HDF & HDF-EOS Workshop XII 62 62
  • 63.
    Code: Create aFile hid_t herr_t file_id; status; file_id = H5Fcreate ("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); status = H5Fclose (file_id); “/” (root) Note: Return codes not checked for errors in code samples. 10/15/08 HDF & HDF-EOS Workshop XII 63 63
  • 64.
    Dataset Components Metadata Data Dataspace Rank Dimensions 3 Dim_1= 4 Dim_2 = 5 Dim_3 = 7 Datatype Integer Storage info Attributes Time = 32.4 Chunked Pressure = 987 Compressed Temp = 56 10/15/08 HDF & HDF-EOS Workshop XII 64 64
  • 65.
    Steps to Createa Dataset 1. Define dataset characteristics • • • Dataspace - 4x6 Datatype – integer Properties if needed, or use H5P_DEFAULT 2. Decide where to put it • Obtain location ID: - Group ID puts it in a Group - File ID puts it in Root Group “/” (root) A 3. Create dataset in file 4. Close everything 10/15/08 HDF & HDF-EOS Workshop XII 65 65
  • 66.
    HDF5 Pre-defined DatatypeIdentifiers HDF5 defines* set of Datatype Identifiers per HDF5 session. For example: C Type HDF5 File Type HDF5 Memory Type int H5T_STD_I32BE H5T_STD_I32LE H5T_NATIVE_INT float H5T_IEEE_F32BE H5T_IEEE_F32LE H5T_NATIVE_FLOAT double H5T_IEEE_F64BE H5T_IEEE_F64LE H5T_NATIVE_DOUBLE * Value of datatype is NOT fixed 10/15/08 HDF & HDF-EOS Workshop XII 66
  • 67.
    Pre-defined File DatatypeIdentifiers Examples: H5T_IEEE_F64LE Eight-byte, little-endian, IEEE floating-point H5T_STD_I32LE Four-byte, little-endian, signed two's complement integer Architecture* Programming Type NOTE: What you see in the file. Name is the same everywhere and explicitly defines a datatype. *STD= “An architecture with a semi-standard type like 2’s complement integer, unsigned integer…” 10/15/08 HDF & HDF-EOS Workshop XII 67
  • 68.
    Pre-defined Native Datatypes Examplesof predefined native types in C: H5T_NATIVE_INT H5T_NATIVE_FLOAT H5T_NATIVE_UINT H5T_NATIVE_LONG H5T_NATIVE_CHAR (int) (float ) (unsigned int) (long ) (char ) NOTE: Memory types. Different for each machine. Used for reading/writing. 10/15/08 HDF & HDF-EOS Workshop XII 68
  • 69.
    Dataset Creation PropertyList Dataset creation property list: information on how to organize data in storage. Chunked Chunked & compressed H5P_DEFAULT: contiguous 10/15/08 HDF & HDF-EOS Workshop XII 69 69
  • 70.
    Code: Create aDataset 1 2 3 hid_t hsize_t herr_t file_id, dataset_id, dataspace_id; dims[2]; status; 4 file_id = H5Fcreate (”file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); Create a 5 dataspace= 4; dims[0] 6 7 rank dims[1] = 6; dataspace_id = H5Screate_simple (2, dims, NULL); Create a dataset 8 current dims pathname datatype dataset_id = H5Dcreate(file_id,”A",H5T_STD_I32BE, dataspace_id, H5P_DEFAULT); dataspace Terminate access to dataset, dataspace, file 9 status = H5Dclose (dataset_id); 10 status = H5Sclose (dataspace_id); 11 status = H5Fclose (file_id); 10/15/08 property list (default) HDF & HDF-EOS Workshop XII 70 70
  • 71.
    Example Code -H5Dwrite Dataset Identifier from H5Dcreate or H5Dopen Memory Datatype status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); 10/15/08 HDF & HDF-EOS Workshop XII 71
  • 72.
    Example Code –H5Dwrite status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); Data Transfer Property List (MPI I/O, Transformations, …) Memory Dataspace File Dataspace H5S_ALL selects entire dataspace 10/15/08 HDF & HDF-EOS Workshop XII 72
  • 73.
    Partial I/O Memory Dataspace H5S_ALL FileDataspace (disk) H5S_ALL Get a Dataspace: H5Screate_simple H5Dget_space Modify Dataspace: H5Sselect_hyperslab H5Sselect_elements 10/15/08 HDF & HDF-EOS Workshop XII 73
  • 74.
    Example Code –H5Dread status = H5Dread (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_rdata); 10/15/08 HDF & HDF-EOS Workshop XII 74
  • 75.
    High Level APIs:HDF5 Lite (H5LT) #include "H5LT.h" … file_id = H5Fcreate (“file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); status = H5LTmake_dataset (file_id,“A", 2, dims, H5T_STD_I32BE, data); status = H5Fclose (file_id); 10/15/08 HDF & HDF-EOS Workshop XII 75
  • 76.
    High Level APIs • • • • • 10/15/08 HDF5Lite HDF5 Image HDF5 Table HDF5 Dimension Scales HDF5 Packet Table HDF & HDF-EOS Workshop XII 76
  • 77.
    Example: Create aGroup “/” (root) A B 4x6 array of integers file.h5 10/15/08 HDF & HDF-EOS Workshop XII 77 77
  • 78.
    Steps to Createa Group 1. Decide where to put it – “root group” • Obtain location ID 2. Decide name – “B” 3. Create group in file 4. (Eventually) close the group. 10/15/08 HDF & HDF-EOS Workshop XII 78 78
  • 79.
    Code: Create aGroup hid_t file_id, group_id; ... /* Open “file.h5” */ file_id = H5Fopen (“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT); /* Create group "/B" in file. */ group_id = H5Gcreate (file_id,"B",0); Size hint for number of bytes to store names of objects. 0=default /* Close group and file. */ status = H5Gclose (group_id); status = H5Fclose (file_id); 10/15/08 HDF & HDF-EOS Workshop XII 79 79
  • 80.
    Thank you! This workwas supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 10/15/08 HDF & HDF-EOS Workshop XII 80

Editor's Notes

  • #14 The CFD General Notation System (CGNS) provides a general, portable, and extensible standard for the storage and retrieval of computational fluid dynamics (CFD) analysis data. It consists of a collection of conventions, and free and open software implementing those conventions. It is self-descriptive, machine-independent, well-documented, and administered by an international steering committee. The CGNS implementation of SIDS, so-called MLL, was originally built using a file format called ADF (Advanced Data Format).  This format was based on a common file format system previously in use at McDonnell Douglas.  The ADF has worked extremely well, requiring little repair, upgrade, or maintenance over the last decade. However, ADF does not have parallel I/O or data compression capabilities, and does not have the support and tools that the storage format HDF5 offers.  HDF5, supported by The HDF Group, has rapidly grown to become a world-wide format standard for storing scientific data.  HDF5 has parallel capability as well as a broader support base than ADF. 
  • #21 This shows that you can mix objects of different types according to your needs. Typically, there will be metadata stored with objects to indicate what type of object they are. Like HDF4, HDF5 has a grouping structure. The main difference is that every HDF5 file starts with a root group, whereas HDF4 doesn’t need any groups at all.
  • #23 Data Array is an ordered collection of identically typed data items distinguished by their indices Metadata: Dataspace – Rank, dimensions; spatial info about dataset Datatype – Information on how to interpret your data Storage Properties – How array is organized Attributes – User-defined metadata (optional)
  • #29 Here is an example of a basic HDF5 object. Notice that each element in the 3D array is a record with four values in it.
  • #33 Data Array is an ordered collection of identically typed data items distinguished by their indices Metadata: Dataspace – Rank, dimensions; spatial info about dataset Datatype – Information on how to interpret your data Storage Properties – How array is organized Attributes – User-defined metadata (optional)
  • #35 Like HDF4, HDF5 has a grouping structure. The main difference is that every HDF5 file starts with a root group, whereas HDF4 doesn’t need any groups at all.
  • #66 To create this file, we would start by creating the file itself. When you create a file, the root group gets created with it. So every file has at least that one group.