Help us improve by taking our short survey: https://www.hdfgroup.org/website-survey/

| HDF5 Last Updated on 2025-11-11 The HDF5 Field Guide

- Main Page
- Getting started
- User Guide
- Reference Manual
- Cookbook
- Technical Notes
- RFCs
- Specifications
- Glossary
- Full-Text Search
- About

Loading...
Searching...
No Matches
Introduction to the Virtual Dataset - VDS

Navigate back: Main / Technical Notes

# Introduction to VDS

The HDF5 Virtual Dataset (VDS) feature enables users to access data in a collection of HDF5 files as a single HDF5 dataset and to use the HDF5 APIs to work with that dataset.

For example, your data may be collected into four files:

You can map the datasets in the four files into a single VDS that can be accessed just like any other dataset:

The mapping between a VDS and the HDF5 source datasets is persistent and transparent to an application. If a source file is missing the fill value will be displayed.

See the Virtual (VDS) Documentation for complete details regarding the VDS feature.

The VDS feature was implemented using hyperslab selection (H5Sselect_hyperslab). See the tutorial on Reading From or Writing to a Subset of a Dataset for more information on selecting hyperslabs.

## Programming Model

To create a Virtual Dataset you simply follow the HDF5 programming model and add a few additional API calls to map the source code datasets to the VDS.

Following are the steps for creating a Virtual Dataset:

- Create the source datasets that will comprise the VDS
- Create the VDS: ‐ Define a datatype and dataspace (can be unlimited)
- Define the dataset creation property list (including fill value)
- (Repeat for each source dataset) Map elements from the source dataset to elements of the VDS
- Select elements in the source dataset (source selection)
- Select elements in the virtual dataset (destination selection)
- Map destination selections to source selections (see Functions for Working with a VDS)
- Call H5Dcreate using the properties defined above
- Access the VDS as a regular HDF5 dataset
- Close the VDS when finished

#### Functions for Working with a VDS

The H5Pset_virtual API sets the mapping between virtual and source datasets. This is a dataset creation property list. Using this API will change the layout of the dataset to H5D_VIRTUAL. As with specifying any dataset creation property list, an instance of the property list is created, modified, passed into the dataset creation call and then closed:

dcpl = H5Pcreate (H5P_DATASET_CREATE);
src_space = H5screate_simple ...
status = H5Sselect_hyperslab (space, ...
status = H5Pset_virtual (dcpl, space, SRC_FILE[i], SRC_DATASET[i], src_space);
dset = H5Dcreate2 (file, DATASET, H5T_NATIVE_INT, space, H5P_DEFAULT, dcpl, H5P_DEFAULT);
status = H5Pclose (dcpl);
H5P_DEFAULT
#define H5P_DEFAULT
Definition H5Ppublic.h:220
H5P_DATASET_CREATE
#define H5P_DATASET_CREATE
Definition H5Ppublic.h:60
H5Pset_virtual
herr_t H5Pset_virtual(hid_t dcpl_id, hid_t vspace_id, const char *src_file_name, const char *src_dset_name, hid_t src_space_id)
Sets the mapping between virtual and source datasets.
H5Dcreate2
hid_t H5Dcreate2(hid_t loc_id, const char *name, hid_t type_id, hid_t space_id, hid_t lcpl_id, hid_t dcpl_id, hid_t dapl_id)
Creates a new dataset and links it into the file.
H5Sselect_hyperslab
herr_t H5Sselect_hyperslab(hid_t space_id, H5S_seloper_t op, const hsize_t start[], const hsize_t stride[], const hsize_t count[], const hsize_t block[])
Selects a hyperslab region to add to the current selected region.
H5T_NATIVE_INT
#define H5T_NATIVE_INT
Definition H5Tpublic.h:863
H5Pclose
herr_t H5Pclose(hid_t plist_id)
Terminates access to a property list.
H5Pcreate
hid_t H5Pcreate(hid_t cls_id)
Creates a new property list as an instance of a property list class.

There are several other APIs introduced with Virtual Datasets, including query functions. For details see the complete list of HDF5 library APIs that support Virtual Datasets.

#### Limitations

This feature was introduced in HDF5-1.10.

The number of source datasets is unlimited. However, there is a limit on the size of each source dataset.

## Programming Examples

Example 1 This example creates three HDF5 files, each with a one-dimensional dataset of 6 elements. The datasets in these files are the source datasets that are then used to create a 4 x 6 Virtual Dataset with a fill value of -1. The first three rows of the VDS are mapped to the data from the three source datasets as shown below:

In this example the three source datasets are mapped to the VDS with this code:

>
src_space = H5Screate_simple (RANK1, dims, NULL);
for (i = 0; i &lt; 3; i++) {
start[0] = (hsize_t)i;
// Select i-th row in the virtual dataset; selection in the source datasets is the same.
status = H5Sselect_hyperslab (space, H5S_SELECT_SET, start, NULL, count, block);
status = H5Pset_virtual (dcpl, space, SRC_FILE[i], SRC_DATASET[i], src_space);
}
endcode>
After the VDS is created and closed, it is reopened. The property list is then queried to determine the
layout of the dataset and its mappings, and the data in the VDS is read and printed.
This example is in the HDF5 source code and can be obtained from here:
<h4>C Example</h4>
For details on compiling an HDF5 application: [ Compiling HDF5 Applications ]
<h4>Example 2</h4>
This example shows how to use a C-style printf statement for specifying multiple source datasets as one virtual
dataset. Only one mapping is required. In other words only one #H5Pset_virtual call is needed to map multiple datasets.
It creates a 2-dimensional unlimited VDS. Then it re-opens the file, makes queries, and reads the virtual dataset.
The source datasets are specified as A-0, A-1, A-2, and A-3. These are mapped to the virtual dataset with one call:
\code
status = H5Pset_virtual (dcpl, vspace, SRCFILE, "A-%b", src_space);
H5S_SELECT_SET
@ H5S_SELECT_SET
Definition H5Spublic.h:93
hsize_t
uint64_t hsize_t
Definition H5public.h:304
H5Screate_simple
hid_t H5Screate_simple(int rank, const hsize_t dims[], const hsize_t maxdims[])
Creates a new simple dataspace and opens it for access.

The b indicates that the block count of the selection in the dimension should be used.

#### C Example

For details on compiling an HDF5 application: [ Compiling HDF5 Applications ]

Using h5dump with a VDS The h5dump utility can be used to view a VDS. The h5dump output for a VDS looks exactly like that for any other dataset. If h5dump cannot find a source dataset then the fill value will be displayed.

You can determine that a dataset is a VDS by looking at its properties with

h5dump -p

It will display each source dataset mapping, beginning with Mapping 0. Below is an excerpt of the output of

h5dump -p

on the vds.h5 file created in Example 1.You can see that the entire source file a.h5 is mapped to the first row of the VDS dataset.

Navigate back: Main / Technical Notes