Introduction to the
HDF5 Packet Table API

Originally published as "NCSA Technical Report: HDF5 Packet Table API"
A Joint Project between Boeing and University of Illinois
February 2005
Mike Folk, Quincey Koziol, James Laird, Elena Pourmal, NCSA, Univ. of Illinois
John Wegener, George Lewandowski, Joe Turner, Rodney Davis, Boeing Flight
Test Instrumentation

Introduction

The proliferation of sensors and other instruments introduces enormous challenges to data management. Even for a single event, incoming synchronized time-sequenced data can have many sources, and the number of incoming data streams, as well as the types of data, can be large. In Boeing's flight test data applications, for instance, data arrives from test aircraft, voice communications, video, ground, satellite tracking, and other sources. This data must be gathered, integrated, processed, visualized, and archived. Similar scenarios exist for many different applications, such as environmental monitoring, vehicle testing, and medicine.

The collection and storing of these kinds of data historically have been reduced to unique in-house implementations. There is surprisingly little sharing of these infrastructure technologies even within an application area, let alone across application domains, resulting in frequent and costly re-invention of the same technologies.

HDF5 provides, in a single package, many of the capabilities that otherwise have to be developed from scratch. HDF5 can store virtually any kind of scientific or engineering data and to mix any number of objects of different types in a single container. HDF5 can support different access patterns, simplified data integration, datatype translation, fast I/O, and visualization and analysis software.

Using the HDF5 Packet Table

The following example describes the typical use of the HDF5 Packet Table API.

Writing homogeneous, fixed-sized packets of instrument data. An experimenter wants to store measurements from a fixed number of sensors. All measurements taken at the same time are collected and transmitted in a "packet", including a time stamp. The packet contents are to be saved as a record in a table. (Figure 1.) To do this, the experimenter can create an HDF5 datatype representing a fixed-size "packet" of data in an HDF5 file. This can be any atomic HDF5 datatype, or it could be a compound datatype. Each packet corresponds to a fixed-length horizontal entry, or "record," in a table.

Figure 1. Fixed length packet example.

HDF5 Packet library for flight test data

Boeing's Flight Test Instrumentations Group and the HDF5 development group at the University of Illinois have developed a library that is particularly suited for "packet" data, data that arrives in streams of packets from instruments at potentially very high speeds.

A key to a general-purpose packet I/O library is that, at a certain level, all packet data falls into simple structural categories that can be handled by fixed-length packets. In this scenario, every record to be stored is of the same length.

The Packet Table implementation includes a C API, as well as prototype C++ wrapper objects, which are described in the Packet Table Reference Manual.

HDF5 "Table" vs. "Packet" APIs. The HDF5 packet API is similar to but separate from the HDF5 "Table" API, whose prefix is H5TB. The H5TB Table is designed to be as high-level as possible, adding attributes and abstracting away much of HDF5's internal workings. In contrast, the Packet Table API, whose prefix is H5PT, is a lower-level approach, designed to maximize throughput (as is needed in real-time applications). The Packet Table thus provides bare-bones table functionality with essentially no overhead.

Programming with the Packet library

Packet Tables have states, and must be opened and closed like HDF5 objects. Thus the Packet Table programming model is similar to that for working with other types of HDF5 objects:

An HDF5 file is created or opened.
The location of the object of interest is obtained.
The Packet Table is created or opened using the Packet Table API.
Operations are carried out on the Packet Table.
The Packet Table is closed
Other relevant objects are closed as necessary.

The Packet Table API covers steps 3-5. See the HDF5 documentation for information on how to do the other steps.

The following operations can be performed on Packet Tables:

Create and open Packet Table
1. Create packet table with fixed-size record
Close Packet Table
Write and read Packet Table
1. Append packet(s)
2. Position pointer for read operation
3. Read packet(s) starting at specified position in the table
Query properties of Packet Table
1. Number of packets in the Packet Table
2. If an HDF5 object is a Packet Table

Packet Table APIs implement those operations. Full descriptions of Packet Table APIs can be found on the H5PT: HDF5 Packet Table page of the HDF5 Reference Manual.

Example

The example below demonstrates the use of the C language version of the Packet Table API to create a packet table with fixed-length records and to perform writes and reads. (C sample program) (C++ sample program)

#include "H5PT.h"
#include <stdlib.h>

/*-------------------------------------------------------------------------
 * Packet Table Fixed-Length Example
 *
 * Example program that creates a packet table and performs
 * writes and reads.
 *
 *-------------------------------------------------------------------------
 */

int main(void)
{
 hid_t          fid;        /* File identifier */
 hid_t          ptable;     /* Packet table identifier */

 herr_t         err;        /* Function return status */
 hsize_t        count;      /* Number of records in the table */

 int            x;          /* Loop variable */

    /* Buffers to hold data */
 int writeBuffer[5];
 int readBuffer[5];

   /* Initialize buffers */
 for(x=0; x<5; x++)
 {
     writeBuffer[x]=x;
     readBuffer[x] = -1;
 }

    /* Create a file using default properties */
 fid=H5Fcreate("packet_table_FLexample.h5",H5F_ACC_TRUNC,H5P_DEFAULT,H5P_DEFAULT);

    /* Create a fixed-length packet table within the file */
    /* This table's "packets" will be simple integers. */
 ptable = H5PTcreate_fl(fid, "Packet Test Dataset", H5T_NATIVE_INT, 1);
 if(ptable == H5I_INVALID_HID)
     goto out;

    /* Write one packet to the packet table */
 err = H5PTappend(ptable, 1, &(writeBuffer[0]) );
 if(err < 0)
     goto out;

    /* Write several packets to the packet table */
 err = H5PTappend(ptable, 4, &(writeBuffer[1]) );
 if(err < 0)
     goto out;

    /* Get the number of packets in the packet table.  This should be five. */
 err = H5PTget_num_packets(ptable, &count);
 if(err < 0)
     goto out;

 printf("Number of packets in packet table after five appends: %d\n", count);

    /* Initialize packet table's "current record" */
 err = H5PTcreate_index(ptable);
 if(err < 0)
     goto out;

    /* Iterate through packets, read each one back */
 for(x=0; x<5; x++)
 {
    err = H5PTget_next(ptable, 1, &(readBuffer[x]) );
    if(err < 0)
        goto out;

    printf("Packet %d's value is %d\n", x, readBuffer[x]);
 }
 
    /* Close the packet table */
 err = H5PTclose(ptable);
 if(err < 0)
     goto out;

    /* Close the file */
 H5Fclose(fid);

 return 0;

 out: /* An error has occurred.  Clean up and exit. */
    H5PTclose(ptable);
    H5Fclose(fid);
    return -1;
}

The HDF Group Help Desk: