The HDF Group logo The HDF Group Documentation
Other HDF Links

HDF5 1.10 Documentation

New Features in HDF5 1.10
Additional New APIs

Collective Metadata I/O

Fine-tuning the Metadata Cache

File Space Management

Page Buffering

Partial Edge Chunks



Virtual Datasets (VDS)

Single-Writer/Multiple-Reader (SWMR) Documentation
The purpose of this page is to list and briefly describe the documentation available to those who will use the Single-Writer/Multiple-Reader (SWMR) feature found in the HDF5 Library. The following are the sections on this page:
SWMR User’s Guide
The HDF5 Single-Writer/Multiple-Reader User’s Guide describes HDF5’s SWMR feature and describes how to use the feature. The feature allows concurrent access of HDF5 files by writer and reader processes.

HDF5 Library APIs
The APIs listed below will be incorporated into the HDF5 Reference Manual when the SWMR feature is fully released.
H5Fstart_swmr_write Enables SWMR writing mode for a file.
H5DOappend Appends data to a dataset along a specified dimension.
H5Pget_append_flush Retrieves the values of the append property that is set up in the dataset access property list.
H5Pset_append_flush Sets two actions to perform when the size of a dataset’s dimension being appended reaches a specified boundary.
H5Pget_object_flush_cb Retrieves the object flush property values from the file access property list.
H5Pset_object_flush_cb Sets a callback function to invoke when an object flush occurs in the file.
H5Odisable_mdc_flushes Prevents metadata entries for an HDF5 object from being flushed from the metadata cache to storage.
H5Oenable_mdc_flushes Returns the cache entries associated with an HDF5 object to the default metadata flush and eviction algorithm.
H5Oare_mdc_flushes_disabled Determines if an HDF5 object (dataset, group, committed datatype) has had flushes of metadata entries disabled.
The tools listed below will be incorporated into the HDF5 Reference Manual when they are fully released.
h5watch Allows users to output new records appended to a dataset as it grows. In other words, this tool can be used to watch data that is added to a dataset. The functionality is similar to the Unix user command tail with the follow option, which outputs appended data as the file grows.
h5format_convert Allows users to convert the indexing type of a chunked dataset made with a 1.10.x version of the HDF5 Library to the 1.8.x version 1 B-tree indexing type.
h5clear Allows users to clear the file consistency flags field in the superblock of a file.

SWMR Design Documents
The SWMR design documents, listed below, describe how the feature works and why design choices were made.
Metadata Cache Issues under SWMR and their Solutions The HDF5 Library contains a cache that provides more efficient access to HDF5 file metadata. Although helpful in most instances, this cache can cause problems when the single-writer/multiple-reader (SWMR) data access pattern is used to access an HDF5 file. Additionally, there are even deeper I/O issues that can affect HDF5 file metadata operations. This document describes the cache-oriented problems that the HDF5 Library must overcome in order to implement the SWMR feature as well as our solutions.
Improvements for SWMR File Access and Dataset Append This document describes changes to the HDF5 Library that improve the SWMR (Single-Writer/Multiple-Reader) file access model and provide better support for dataset append operations.
HDF5 Single-Writer/Multiple-Reader Feature Design and Semantics This document describes the design and semantics of HDF5’s single-writer/multiple-reader (SWMR) feature. This feature allows multiple reader processes to inspect a file that is concurrently being written to by a single writer process without requiring any inter-process communication.
File Locking under SWMR Semantics, Programming Model, and Implementation This document describes the design and semantics of how file locking is used to help prevent inappropriate concurrent file access. This locking is only intended to help the user conform to SWMR semantics and is not a fundamental part of SWMR synchronization.

The HDF Group Help Desk: The HDF Group Help Desk
Last modified: February 2017

  Copyright by The HDF Group and the Board
of Trustees of the University of Illinois.
All rights reserved.