The HDF Group logo The HDF Group Documentation
Other HDF Links

HDF5 1.10 Documentation

New Features in HDF5 1.10
Additional New APIs

Collective Metadata I/O

Fine-tuning the Metadata Cache

File Space Management

Page Buffering

Partial Edge Chunks



Virtual Datasets (VDS)

Collective Metadata I/O Documentation
The purpose of this page is to briefly describe the new HDF5 Collective Metadata I/O feature and provide a gateway to available documentation. The page includes the following sections:
Collective Metadata I/O Overview

Calls for HDF5 metadata can result in many small reads and writes. On metadata reads, collective metadata I/O can improve performance by allowing the library to perform optimizations when reading the metadata, by having one rank read the data and broadcasting it to all other ranks.

Collective metadata I/O improves metadata write performance through the construction of an MPI derived datatype that is then written collectively in a single call.

Collective Metadata I/O User and Resource Documents
HDF5 Collective Metadata I/O User Document
(This document is not yet available.)

Until an HDF5 Collective Metadata I/O User Document becomes available, users may find the following resources helpful:

RFC: Collective Metadata Reads (PDF)
RFC: Collective Metadata Writes (PDF)
Taken together, these papers discuss the motivation, design, implementation, and API for HDF5’s Collective Metadata I/O feature.

HDF5 Library APIs
New Collective Metadata I/O Functions
H5Pset_coll_metadata_write    Establishes I/O mode property setting, collective or independent, for metadata writes.
H5Pget_coll_metadata_write  Retrieves I/O mode property setting for metadata writes.
H5Pset_all_coll_metadata_ops    Establishes I/O mode, collective or independent, for metadata read operations.
H5Pget_all_coll_metadata_ops  Retrieves I/O mode for metadata read operations.

Additional API Reference
Functions with No Access Property List Parameter that May Generate Metadata Reads
This page lists the functions associated with HDF5 operations that can issue metadata reads from the metadata cache, but that currently provide no access property list parameter by which to set a collective requirement individually for those read operations.

The need for this document is discussed in H5Pset_all_coll_metadata_ops.


The HDF Group Help Desk: The HDF Group Help Desk
Last modified: February 2017

  Copyright by The HDF Group.
All rights reserved.