Help us improve by taking our short survey: https://www.hdfgroup.org/website-survey/

| HDF5 Last Updated on 2025-11-11 The HDF5 Field Guide

- Main Page
- Getting started
- User Guide
- Reference Manual
- Cookbook
- Technical Notes
- RFCs
- Specifications
- Glossary
- Full-Text Search
- About

Loading...
Searching...
No Matches
Collective Metadata I/O

Navigate back: Main / Technical Notes

The purpose of this page is to briefly describe the new HDF5 Collective Metadata I/O feature and provide a gateway to available documentation. The page includes the following sections:

- Collective Metadata I/O Overview
- Collective Metadata I/O User and Resource Documents
- HDF5 Library APIs

# Collective Metadata I/O Overview

Calls for HDF5 metadata can result in many small reads and writes. On metadata reads, collective metadata I/O can improve performance by allowing the library to perform optimizations when reading the metadata, by having one rank read the data and broadcasting it to all other ranks.

Collective metadata I/O improves metadata write performance through the construction of an MPI derived datatype that is then written collectively in a single call.

# Collective Metadata I/O User and Resource Documents

HDF5 Collective Metadata I/O User Document (This document is not yet available.)

Until an HDF5 Collective Metadata I/O User Document becomes available, users may find the following resources helpful:

- Collective Metadata Writes
- Enabling Collective Metadata Reads

Taken together, these papers discuss the motivation, design, implementation, and API for HDF5’s Collective Metadata I/O feature.

# HDF5 Library APIs

## New Collective Metadata I/O Functions

API | Description
--- | ---
H5Pset_coll_metadata_write | Establishes I/O mode property setting, collective or independent, for metadata writes
H5Pget_coll_metadata_write | Retrieves I/O mode property setting for metadata writes
H5Pset_all_coll_metadata_ops | Establishes I/O mode, collective or independent, for metadata read operations
H5Pget_all_coll_metadata_ops | Retrieves I/O mode for metadata read operations

## Additional API Reference

### Functions with No Access Property List Parameter that May Generate Metadata Reads

Currently there are several operations in HDF5 that can issue metadata reads from the metadata cache, but that take no property list. It is therefore not possible set a collective requirement individually for those operations. The only solution with the HDF5 1.10.0 release is to set the collective requirement globally on H5Fopen or H5Fcreate for all metadata operations to be collective.

The following is a list of those functions in the HDF5 library. This list is integral to the discussion in the H5Pset_all_coll_metadata_ops entry:

API
Attributes
H5Awrite
H5Aread
H5Arename
H5Aiterate2
H5Adelete
H5Aexists
Datasets
H5Dget_space_status
H5Dget_storage_size
H5Dset_extent
H5Ddebug
H5Dclose
H5Dget_create_plist
H5Dget_space (when dataset is a virtual dataset)
Groups
H5Gget_create_plist
H5Gget_info
H5Gclose
Links
H5Literate
H5Lvisit
References
H5Rcreate
H5Rdereference2 (when reference is an object reference)
H5Rget_region
H5Rget_obj_type2
H5Rget_name
Objects
H5Ocopy
H5Oopen_by_addr
H5Oincr_refcount
H5Odecr_refcount
H5Oget_info
H5Oset_comment
H5Ovisit
Files
H5Fis_hdf5
H5Fflush
H5Fclose
H5Fget_file_image
H5Freopen
H5Fget_freespace
H5Fget_info2
H5Fget_free_sections
H5Fmount
H5Funmount
Identifiers
H5Iget_name
Datatypes
H5Tget_create_plist
H5Tclose
Filters
H5Zunregister

In addition, most deprecated functions fall into this category.

The HDF Group may address the above limitation in a future major release, but no decision has been made at this time. Such a change might, for example, include adding new versions of some or all the above functions with an extra property list parameter to allow an individual setting for the collective calling requirement.

Navigate back: Main / Technical Notes