The HDF Group logo The HDF Group Documentation
 
 
Other HDF Links

HDF5 1.10 Documentation


New Features in HDF5 1.10
Additional New APIs

Collective Metadata I/O

Fine-tuning the Metadata Cache

File Space Management

Page Buffering

Partial Edge Chunks

Reference

SWMR

Virtual Datasets (VDS)
 

Page Buffering Documentation
This page describes the new HDF5 Page Buffering option and provides links to available reference manual entries. The page includes the following sections:
 
Overview of the Page Buffering Feature

Summary

Page buffering adds a layer immediately above the Virtual File Driver I/O layer within the library, which acts as a page cache for metadata and raw data accesses that are smaller than the page size for file space allocation. For most applications the HDF5 library generates very small amounts of metadata and raw data accesses in HDF5 datasets, even though the total amount of metadata in an HDF5 file varies between applications. These small and random I/O accesses on parallel file systems result in poor performance for applications.

The page buffering feature, in conjunction with paged aggregation, can be used to reduce the number of small accesses in HDF5. Together these features give an application control over minimizing the granularity and alignment of HDF5 I/O requests. See File Space Management for information on paged aggregation.

Background Information

See the RFC on this feature for complete details.


HDF5 Library APIs
Following are the APIs introduced for this feature.
 
New Functions for Page Buffering
 
H5Pget_file_space_strategy  Retrieves the File Space Strategy for a file creation property list. See File Space Management for more details.
 
H5Pset_file_space_strategy  H5F_FSPACE_STRATEGY_PAGE sets the file space handling strategy to use Page Buffering. See File Space Management for more details.
 
H5Pget_file_space_page_size Retrieves the file space page size.
 
H5Pset_file_space_page_size Sets the file space page size when file space allocation is H5F_FSPACE_STRATEGY_PAGE.
 
H5Pget_page_buffer_size  Retrieves the maximum size for the page buffer and the minimum percentage for metadata and raw data pages.
 
H5Pset_page_buffer_size  Sets the maximum size for the page buffer and the minimum percentage for metadata and raw data pages.
 
H5Fget_page_buffering_stats  Gathers page buffer statistics regarding page access when it is enabled.
 
H5Freset_page_buffering_stats  Resets the page buffer statistics.
 
Tools
h5repack The following options were added to h5repack:

-G FS_PAGESIZE,--fs_pagesize=FS_PAGESIZE enables the file space page size to be changed to FS_PAGESIZE.

-P FS_PERSIST,--fs_persist=FS_PERSIST sets the persisting free space to persist (1) or to not persist (0).

-S FS_STRATEGY, --fs_strategy=FS_STRATEGY sets the file space management strategy.

-T FS_THRESHOLD, --fs_threshold=FS_THRESHOLD sets the free-space section threshold.

 



The HDF Group Help Desk: The HDF Group Help Desk
Last modified: April 2017

  Copyright by The HDF Group.
All rights reserved.