• Virtual Dataset Overview
• Virtual Dataset User and Resource Documents
• HDF5 Library APIs
• Expected Updates and Additional Documentation
• Tools
• Virtual Dataset Design

With a growing amount of data in HDF5, the need has emerged to access data stored across HDF5 files using standard HDF5 objects, such as groups and datasets, without rewriting or rearranging the data.

While the ability to build hierarchical structures across existing HDF5 files has been available in HDF5 for quite some time through the mounting and external link features, the ability has not existed to present data stored in several HDF5 datasets and files as a single HDF5 dataset and to access the data via HDF5 APIs without rewriting and rearranging the data.

To address this, The HDF Group has implemented a new feature called the HDF5 Virtual Dataset (VDS).

The feature is a logical next step in the development of HDF5 that enables HDF5 users to access and work with data stored in a collection of HDF5 files using well-known tools and existing HDF5 applications and higher-level libraries such as h5py, MATLAB, and IDL without changing the way the data is collected and stored.

The following examples illustrate situations that will benefit from use of virtual datasets:

• Synchrotron centers such as DLS and DESY will be generating and storing terabytes of experimental data per day in HDF5 files. Because of the nature of the experiments and hardware constraints, the data representing, for example, an X-ray image will be stored across different HDF5 datasets in multiple HDF5 files. With VDS, the whole image may be be accessed by an application without any specific knowledge of where data for each part of the image is stored.
• Climatologists who study and analyze climate variations (temporal changes at a given location) will be able to use the VDS feature to describe and access “data rods” – data of interest stored in a series of HDF5 files organized by time stamps – without rewriting the data into new HDF5 file. The “data rods” will be accessible as a regular HDF5 dataset via their applications without any special knowledge “coded” into the applications.

• Update: “RFC: HDF5 Virtual Dataset” (see below).
  The current document reflects the design, strategies, and general approach employed in the VDS feature, but the API implementation had to be modified from the specification. The update will correct this divergence.
   
• VDS User Guide material
   
• Presentation materials describing the VDS feature

This document contains useful illustrations that provide an intuitive understanding of virtual datasets.

Note: The current version reflects the design, strategies, and general approach employed in the VDS feature, but the API implementation had to be modified from the specification. An expected update will correct this divergence.