Page Buffering Documentation

The HDF Group

Other HDF Links

HDF5 1.10 Documentation
New Features in HDF5 1.10
Additional New APIs
Collective Metadata I/O
Fine-tuning the Metadata Cache
File Space Management
Page Buffering
Partial Edge Chunks
Reference
SWMR
Virtual Datasets (VDS)

Page Buffering Documentation

This page describes the new HDF5 Page Buffering option and provides links to available reference manual entries. The page includes the following sections:

Overview of the Page Buffering Feature

Summary

Page buffering adds a layer immediately above the Virtual File Driver I/O layer within the library, which acts as a page cache for metadata and raw data accesses that are smaller than the page size for file space allocation. For most applications the HDF5 library generates very small amounts of metadata and raw data accesses in HDF5 datasets, even though the total amount of metadata in an HDF5 file varies between applications. These small and random I/O accesses on parallel file systems result in poor performance for applications.

The page buffering feature, in conjunction with paged aggregation, can be used to reduce the number of small accesses in HDF5. Together these features give an application control over minimizing the granularity and alignment of HDF5 I/O requests. See File Space Management for information on paged aggregation.

Background Information

See the RFC on this feature for complete details.

HDF5 Library APIs

Following are the APIs introduced for this feature.

New Functions for Page Buffering

H5Pget_file_space_strategy Retrieves the File Space Strategy for a file creation property list. See File Space Management for more details.

H5Pset_file_space_strategy H5F_FSPACE_STRATEGY_PAGE sets the file space handling strategy to use Page Buffering. See File Space Management for more details.

H5Pget_file_space_page_size Retrieves the file space page size.

H5Pset_file_space_page_size Sets the file space page size when file space allocation is H5F_FSPACE_STRATEGY_PAGE.

H5Pget_page_buffer_size Retrieves the maximum size for the page buffer and the minimum percentage for metadata and raw data pages.

H5Pset_page_buffer_size Sets the maximum size for the page buffer and the minimum percentage for metadata and raw data pages.

H5Fget_page_buffering_stats Gathers page buffer statistics regarding page access when it is enabled.

H5Freset_page_buffering_stats Resets the page buffer statistics.

Tools

h5repack

The following options were added to h5repack:

-G FS_PAGESIZE,--fs_pagesize=FS_PAGESIZE enables the file space page size to be changed to FS_PAGESIZE.

-P FS_PERSIST,--fs_persist=FS_PERSIST sets the persisting free space to persist (1) or to not persist (0).

-S FS_STRATEGY, --fs_strategy=FS_STRATEGY sets the file space management strategy.

-T FS_THRESHOLD, --fs_threshold=FS_THRESHOLD sets the free-space section threshold.

The HDF Group Help Desk:

Last modified: April 2017