Help us improve by taking our short survey: https://www.hdfgroup.org/website-survey/

| HDF5 Last Updated on 2025-11-11 The HDF5 Field Guide

- Main Page
- Getting started
- User Guide
- Reference Manual
- Cookbook
- Technical Notes
- RFCs
- Specifications
- Glossary
- Full-Text Search
- About

Loading...
Searching...
No Matches
HDF5 Parallel Compression

Navigate back: Main / Technical Notes

# Introduction

When an HDF5 dataset is created, the application can specify optional data filters to be applied to the dataset (as long as the dataset uses a chunked data layout). These filters may perform compression, shuffling, checksumming/error detection and more on the dataset data. The filters are added to a filter pipeline for the dataset and are automatically applied to the data during dataset writes and reads.

Prior to the HDF5 1.10.2 release, a parallel HDF5 application could read datasets with filters applied to them, but could not write to those datasets in parallel. The datasets would have to first be written in a serial HDF5 application or from a single MPI rank in a parallel HDF5 application. This restriction was in place because:

- Updating the data in filtered datasets requires management of file metadata, such as the dataset's chunk index and file space for data chunks, which must be done collectively in order for MPI ranks to have a consistent view of the file. At the time, HDF5 lacked the collective coordination of this metadata management.
- When multiple MPI ranks are writing independently to the same chunk in a dataset (even if their selected portions of the chunk don't overlap), the whole chunk has to be read, unfiltered, modified, re-filtered and then written back to disk. This read-modify-write style of operation would cause conflicts among the MPI ranks and lead to an inconsistent view of the file.

Introduced in the HDF5 1.10.2 release, the parallel compression feature allows an HDF5 application to write in parallel to datasets with filters applied to them, as long as collective I/O is used. The feature introduces new internal infrastructure that coordinates the collective management of the file metadata between MPI ranks during dataset writes. It also accounts for multiple MPI ranks writing to a chunk by assigning ownership to one of the MPI ranks, at which point the other MPI ranks send their modifications to the owning MPI rank.

The parallel compression feature is always enabled when HDF5 is built with parallel enabled, but the feature may be disabled if the necessary MPI-3 routines are not available. Therefore, HDF5 conditionally defines the macro `H5_HAVE_PARALLEL_FILTERED_WRITES` which an application can check for to see if the feature is available.

# Examples

Using the parallel compression feature is very similar to using compression in serial HDF5, except that dataset writes must be collective:

hid_t dxpl_id = H5Pcreate(H5P_DATASET_XFER);
H5Pset_dxpl_mpio(dxpl_id, H5FD_MPIO_COLLECTIVE);
H5Dwrite(..., dxpl_id, ...);
H5FD_MPIO_COLLECTIVE
@ H5FD_MPIO_COLLECTIVE
Definition H5FDmpi.h:38
hid_t
int64_t hid_t
Definition H5Ipublic.h:60
H5P_DATASET_XFER
#define H5P_DATASET_XFER
Definition H5Ppublic.h:68
H5Pset_dxpl_mpio
herr_t H5Pset_dxpl_mpio(hid_t dxpl_id, H5FD_mpio_xfer_t xfer_mode)
Sets data transfer mode.
H5Dwrite
herr_t H5Dwrite(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id, hid_t dxpl_id, const void *buf)
Writes raw data from a buffer to a dataset.
H5Pcreate
hid_t H5Pcreate(hid_t cls_id)
Creates a new property list as an instance of a property list class.

The following are two simple examples of using the parallel compression feature:

ph5_filtered_writes.c

ph5_filtered_writes_no_sel.c

The former contains simple examples of using the parallel compression feature to write to compressed datasets, while the latter contains an example of how to write to compressed datasets when one or MPI ranks don't have any data to write to a dataset. Remember that the feature requires these writes to use collective I/O, so the MPI ranks which have nothing to contribute must still participate in the collective write call.

# Multi-dataset I/O support

The parallel compression feature is supported when using the multi-dataset I/O API routines (H5Dwrite_multi/H5Dread_multi), but the following should be kept in mind:

- Parallel writes to filtered datasets must still be collective, even when using the multi-dataset I/O API routines
- When the multi-dataset I/O API routines are passed a mixture of filtered and unfiltered datasets, the library currently has to perform I/O on them separately in two phases. Since there is some slight complexity involved in this, it may be best (depending on the number of datasets, number of selected chunks, number of filtered vs. unfiltered datasets, etc.) to make two individual multi-dataset I/O calls, one for the filtered datasets and one for the unfiltered datasets. When performing writes to the datasets, this would also allow independent write access to the unfiltered datasets if desired, while still performing collective writes to the filtered datasets.

# Incremental file space allocation support

HDF5's file space allocation time function. H5Pset_alloc_time, is a dataset creation property that can have significant effects on application performance, especially if the application uses parallel HDF5. In a serial HDF5 application, the default file space allocation time for chunked datasets is incremental. This means that allocation of space in the HDF5 file for data chunks is deferred until data is first written to those chunks. In parallel HDF5, the file space allocation time was previously always forced to early, which allocates space in the file for all of a dataset's data chunks at creation time (or during the first open of a dataset if it was created serially). This would ensure that all the necessary file space was allocated so MPI ranks could perform independent I/O operations on a dataset without needing further coordination of file metadata as described previously.

While this strategy has worked in the past, it has some noticeable drawbacks. For one, the larger the chunked dataset being created, the more noticeable overhead there will be during dataset creation as all of the data chunks are being allocated in the HDF5 file. Further, these data chunks will, by default, be filled, using H5Pset_fill_value, with HDF5's default fill data value, leading to extraordinary dataset creation overhead and resulting in pre-filling large portions of a dataset that the application might have been planning to overwrite anyway. Even worse, there will be more initial overhead from compressing that fill data before writing it out, only to have it read back in, unfiltered and modified the first time a chunk is written to. In the past, it was typically suggested that parallel HDF5 applications should use H5Pset_fill_time with a value of H5D_FILL_TIME_NEVER in order to disable writing of the fill value to dataset chunks, but this isn't ideal if the application actually wishes to make use of fill values.

With improvements made to the parallel compression feature for the HDF5 1.14.0 release, incremental file space allocation is now the default for datasets created in parallel only if they have filters applied to them. Early file space allocation is still supported for these datasets if desired and is still forced for datasets created in parallel that do not have filters applied to them. This change should significantly reduce the overhead of creating filtered datasets in parallel HDF5 applications and should be helpful to applications that wish to use a fill value for these datasets. It should also help significantly reduce the size of the HDF5 file, as file space for the data chunks is allocated as needed rather than all at once.

# Performance Considerations

Since getting good performance out of HDF5's parallel compression feature involves several factors, the following is a list of performance considerations (generally from most to least important) and best practices to take into account when trying to get the optimal performance out of the parallel compression feature.

## Begin with a good chunking strategy

Starting with a good Chunking in HDF5 strategy will generally have the largest impact on overall application performance. The different chunking parameters can be difficult to fine-tune, but it is essential to start with a well-performing chunking layout before adding compression and parallel I/O into the mix. Compression itself adds overhead and may have side effects that necessitate further adjustment of the chunking parameters and HDF5 application settings. Consider that the chosen chunk size becomes a very important factor when compression is involved, as data chunks have to be completely read and re-written to perform partial writes to the chunk.

Improving I/O Performance When Working with HDF5 Compressed Datasets is a useful reference for more information on getting good performance when using a chunked dataset layout.

## Avoid chunk sharing

Since the parallel compression feature has to assign ownership of data chunks to a single MPI rank in order to avoid the previously described read-modify-write issue, an HDF5 application may need to take care when determining how a dataset will be divided up among the MPI ranks writing to it. Each dataset data chunk that is written to by more than 1 MPI rank will incur extra MPI overhead as one of the ranks takes ownership and the other ranks send it their data and information about where in the chunk that data belongs. While not always possible to do, an HDF5 application will get the best performance out of parallel compression if it can avoid writing in a way that causes more than 1 MPI rank to write to any given data chunk in a dataset.

## Collective metadata operations

The parallel compression feature typically works with a significant amount of metadata related to the management of the data chunks in datasets. In initial performance results gathered from various HPC machines, it was found that the parallel compression feature did not scale well at around 8k MPI ranks and beyond. On further investigation, it became obvious that the bottleneck was due to heavy filesystem pressure from the metadata management for dataset data chunks as they changed size (as a result of data compression) and moved around in the HDF5 file.

Enabling collective metadata operations in the HDF5 application (as in the below snippet) showed significant improvement in performance and scalability and is generally always recommended unless application performance shows negative benefits by doing so.

...
hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS);
H5Pset_fapl_mpio(fapl_id, MPI_COMM_WORLD, MPI_INFO_NULL);
H5Pset_all_coll_metadata_ops(fapl_id, 1);
H5Pset_coll_metadata_write(fapl_id, 1);
hid_t file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id);
...
H5F_ACC_TRUNC
#define H5F_ACC_TRUNC
Definition H5Fpublic.h:30
H5P_FILE_ACCESS
#define H5P_FILE_ACCESS
Definition H5Ppublic.h:56
H5P_DEFAULT
#define H5P_DEFAULT
Definition H5Ppublic.h:220
H5Pset_coll_metadata_write
herr_t H5Pset_coll_metadata_write(hid_t plist_id, bool is_collective)
Sets metadata write mode to be collective or independent (default)
H5Pset_fapl_mpio
herr_t H5Pset_fapl_mpio(hid_t fapl_id, MPI_Comm comm, MPI_Info info)
Stores MPI IO communicator information to the file access property list.
H5Pset_all_coll_metadata_ops
herr_t H5Pset_all_coll_metadata_ops(hid_t plist_id, bool is_collective)
Sets metadata I/O mode for read operations to be collective or independent (default)
H5Fcreate
hid_t H5Fcreate(const char *filename, unsigned flags, hid_t fcpl_id, hid_t fapl_id)
Creates an HDF5 file.

## Align chunks in the file

The natural layout of an HDF5 file may cause dataset data chunks to end up at addresses in the file that do not align well with the underlying file system, possibly leading to poor performance. As an example, Lustre performance is generally good when writes are aligned with the chosen stripe size. The HDF5 application can use H5Pset_alignment to have a bit more control over where objects in the HDF5 file end up. However, do note that setting the alignment of objects generally wastes space in the file and has the potential to dramatically increase its resulting size, so caution should be used when choosing the alignment parameters.

H5Pset_alignment has two parameters that control the alignment of objects in the HDF5 file, the "threshold" value and the alignment value. The threshold value specifies that any object greater than or equal in size to that value will be aligned in the file at addresses which are multiples of the chosen alignment value. While the value 0 can be specified for the threshold to make every object in the file be aligned according to the alignment value, this isn't generally recommended, as it will likely waste an excessive amount of space in the file.

In the example below, the chosen dataset chunk size is provided for the threshold value and 1MiB is specified for the alignment value. Assuming that 1MiB is an optimal alignment value (e.g., assuming that it matches well with the Lustre stripe size), this should cause dataset data chunks to be well-aligned and generally give good write performance.

hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS);
H5Pset_fapl_mpio(fapl_id, MPI_COMM_WORLD, MPI_INFO_NULL);
/* Assuming Lustre stripe size is 1MiB, align data chunks
in the file to address multiples of 1MiB. */
H5Pset_alignment(fapl_id, dataset_chunk_size, 1048576);
hid_t file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id);
H5Pset_alignment
herr_t H5Pset_alignment(hid_t fapl_id, hsize_t threshold, hsize_t alignment)
Sets alignment properties of a file access property list.

## File free space managers

As data chunks in a dataset get written to and compressed, they can change in size and be relocated in the HDF5 file. Since parallel compression usually involves many data chunks in a file, this can create significant amounts of free space in the file over its lifetime and eventually cause performance issues.

An HDF5 application can use H5Pset_file_space_strategy with a value of H5F_FSPACE_STRATEGY_PAGE to enable the paged aggregation feature, which can accumulate metadata and raw data for dataset data chunks into well-aligned, configurably sized pages for better performance. However, note that using the paged aggregation feature will cause any setting from H5Pset_alignment to be ignored. While an application should be able to get comparable performance effects by setting the size of these pages, using H5Pset_file_space_page_size, to be equal to the value that would have been set for H5Pset_alignment, this may not necessarily be the case and should be studied.

Note that H5Pset_file_space_strategy has a persist parameter. This determines whether or not the file free space manager should include extra metadata in the HDF5 file about free space sections in the file. If this parameter is false, any free space in the HDF5 file will become unusable once the HDF5 file is closed. For parallel compression, it's generally recommended that persist be set to true, as this will keep better track of file free space for data chunks between accesses to the HDF5 file.

hid_t fcpl_id = H5Pcreate(H5P_FILE_CREATE);
/* Use persistent free space manager with paged aggregation */
H5Pset_file_space_strategy(fcpl_id, H5F_FSPACE_STRATEGY_PAGE, 1, 1);
/* Assuming Lustre stripe size is 1MiB, set page size to that */
H5Pset_file_space_page_size(fcpl_id, 1048576);
...
hid_t file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC, fcpl_id, fapl_id);
H5F_FSPACE_STRATEGY_PAGE
@ H5F_FSPACE_STRATEGY_PAGE
Definition H5Fpublic.h:193
H5P_FILE_CREATE
#define H5P_FILE_CREATE
Definition H5Ppublic.h:52
H5Pset_file_space_page_size
herr_t H5Pset_file_space_page_size(hid_t plist_id, hsize_t fsp_size)
Sets the file space page size for a file creation property list.
H5Pset_file_space_strategy
herr_t H5Pset_file_space_strategy(hid_t plist_id, H5F_fspace_strategy_t strategy, bool persist, hsize_t threshold)
Sets the file space handling strategy and persisting free-space values for a file creation property l...

## Low-level collective vs. independent I/O

While the parallel compression feature requires that the HDF5 application set and maintain collective I/O at the application interface level (via H5Pset_dxpl_mpio), it does not require that the actual MPI I/O that occurs at the lowest layers of HDF5 be collective; independent I/O may perform better depending on the application I/O patterns and parallel file system performance, among other factors. The application may use H5Pset_dxpl_mpio_collective_opt to control this setting and see which I/O method provides the best performance.

hid_t dxpl_id = H5Pcreate(H5P_DATASET_XFER);
H5Pset_dxpl_mpio(dxpl_id, H5FD_MPIO_COLLECTIVE);
H5Pset_dxpl_mpio_collective_opt(dxpl_id, H5FD_MPIO_INDIVIDUAL_IO); /* Try independent I/O */
H5Dwrite(..., dxpl_id, ...);
H5FD_MPIO_INDIVIDUAL_IO
@ H5FD_MPIO_INDIVIDUAL_IO
Definition H5FDmpi.h:51
H5Pset_dxpl_mpio_collective_opt
herr_t H5Pset_dxpl_mpio_collective_opt(hid_t dxpl_id, H5FD_mpio_collective_opt_t opt_mode)
Sets low-level data transfer mode.

## Runtime HDF5 Library version

An HDF5 application can use the H5Pset_libver_bounds routine to set the upper and lower bounds on library versions to use when creating HDF5 objects. For parallel compression specifically, setting the library version to the latest available version can allow access to better/more efficient chunk indexing types and data encoding methods. For example:

...
hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS);
H5Pset_libver_bounds(fapl_id, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);
hid_t file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id);
...
H5F_LIBVER_LATEST
@ H5F_LIBVER_LATEST
Definition H5Fpublic.h:182
H5Pset_libver_bounds
herr_t H5Pset_libver_bounds(hid_t plist_id, H5F_libver_t low, H5F_libver_t high)
Controls the range of library release versions used when creating objects in a file.

Navigate back: Main / Technical Notes