Dataset Transfer Properties

Detailed Description

Use dataset transfer properties to customize certain aspects of reading and writing datasets such as transformations, MPI-IO I/O mode, error detection, etc. These properties are not persisted with datasets, and can be adjusted at runtime before a dataset is read or written.

Data transfer property list functions (H5P)

C Function	Purpose
H5Pset_btree_ratios/H5Pget_btree_ratios	Sets/gets B-tree split ratios for a dataset transfer property list.
H5Pset_buffer	Maximum size for the type conversion buffer and the background buffer. May also supply pointers to application-allocated buffers.
H5Pset_data_transform/H5Pget_data_transform	Sets/gets a data transform expression.
H5Pset_dataset_io_hyperslab_selection	Sets a hyperslab file selection for a dataset I/O operation.
H5Pset_edc_check/H5Pget_edc_check	Sets/gets whether to enable error-detection when reading a dataset.
H5Pset_hyper_vector_size	set the number of "I/O vectors" (offset and length pairs) which are to be accumulated in memory before being issued to the lower levels of the library for reading or writing the actual data.
H5Pset_filter_callback	Sets user-defined filter callback function.
H5Pset_hyper_vector_size/H5Pget_hyper_vector_size	Sets/gets number of I/O vectors to be read/written in hyperslab I/O.
H5Pset_type_conv_cb/H5Pget_type_conv_cb	Sets/gets user-defined datatype conversion callback function.
H5Pset_vlen_mem_manager/H5Pget_vlen_mem_manager	Sets/gets the memory manager for variable-length datatype allocation in H5Dread and H5Dvlen_reclaim.
H5Pset_dxpl_mpio/H5Pget_dxpl_mpio	Sets/gets data transfer mode.
H5Pset_dxpl_mpio_chunk_opt	Sets a flag specifying linked-chunk I/O or multi-chunk I/O.
H5Pset_dxpl_mpio_chunk_opt_num	Sets a numeric threshold for linked-chunk I/O.
H5Pset_dxpl_mpio_chunk_opt_ratio	Sets a ratio threshold for collective I/O.
H5Pset_dxpl_mpio_collective_opt	Sets a flag governing the use of independent versus collective I/O.
H5Pget_mpio_actual_chunk_opt_mode	Gets the type of chunk optimization that HDF5 actually performed on the last parallel I/O call.
H5Pget_mpio_actual_io_mode	Gets the type of I/O that HDF5 actually performed on the last parallel I/O call.
H5Pget_mpio_no_collective_cause	Gets local and global causes that broke collective I/O on the last parallel I/O call.
H5Pset_selection_io/H5Pget_selection_io	Sets/gets the selection I/O mode.
H5Pget_no_selection_io_cause	Gets the cause for not performing selection or vector I/O on the last parallel I/O call.
H5Pget_actual_selection_io_mode	Gets the type(s) (scalar, vector, selection) of raw data I/O performed on the last I/O call.
H5Pset_modify_write_buf/H5Pget_modify_write_buf	Sets/gets a flag allowing the library to modify the contents of the write buffer.
H5Pset_preserve/H5Pget_preserve	No longer available, deprecated as it no longer has any effect.

Collaboration diagram for Dataset Transfer Properties:

Functions
H5_DLL herr_t	H5Pset_dxpl_mpio (hid_t dxpl_id, H5FD_mpio_xfer_t xfer_mode)
	Sets data transfer mode.
H5_DLL herr_t	H5Pget_dxpl_mpio (hid_t dxpl_id, H5FD_mpio_xfer_t *xfer_mode)
	Returns the data transfer mode.
H5_DLL herr_t	H5Pset_dxpl_mpio_collective_opt (hid_t dxpl_id, H5FD_mpio_collective_opt_t opt_mode)
	Sets low-level data transfer mode.
H5_DLL herr_t	H5Pset_dxpl_mpio_chunk_opt (hid_t dxpl_id, H5FD_mpio_chunk_opt_t opt_mode)
	Sets a flag specifying linked-chunk I/O or multi-chunk I/O.
H5_DLL herr_t	H5Pset_dxpl_mpio_chunk_opt_num (hid_t dxpl_id, unsigned num_chunk_per_proc)
	Sets a numeric threshold for linked-chunk I/O.
H5_DLL herr_t	H5Pset_dxpl_mpio_chunk_opt_ratio (hid_t dxpl_id, unsigned percent_num_proc_per_chunk)
	Sets a ratio threshold for collective I/O.
herr_t	H5Pget_btree_ratios (hid_t plist_id, double *left, double *middle, double *right)
	Gets B-tree split ratios for a dataset transfer property list.
size_t	H5Pget_buffer (hid_t plist_id, void **tconv, void **bkg)
	Reads buffer settings.
ssize_t	H5Pget_data_transform (hid_t plist_id, char *expression, size_t size)
	Retrieves a data transform expression.
H5Z_EDC_t	H5Pget_edc_check (hid_t plist_id)
	Determines whether error-detection is enabled for dataset reads.
herr_t	H5Pget_hyper_vector_size (hid_t fapl_id, size_t *size)
	Retrieves number of I/O vectors to be read/written in hyperslab I/O.
int	H5Pget_preserve (hid_t plist_id)
	Checks status of the dataset transfer property list (DEPRECATED)
herr_t	H5Pget_type_conv_cb (hid_t dxpl_id, H5T_conv_except_func_t *op, void **operate_data)
	Gets user-defined datatype conversion callback function.
herr_t	H5Pget_vlen_mem_manager (hid_t plist_id, H5MM_allocate_t *alloc_func, void **alloc_info, H5MM_free_t *free_func, void **free_info)
	Gets the memory manager for variable-length datatype allocation in H5Dread() and H5Dvlen_reclaim()
herr_t	H5Pset_btree_ratios (hid_t plist_id, double left, double middle, double right)
	Sets B-tree split ratios for a dataset transfer property list.
herr_t	H5Pset_buffer (hid_t plist_id, size_t size, void *tconv, void *bkg)
	Sets type conversion and background buffers.
herr_t	H5Pset_data_transform (hid_t plist_id, const char *expression)
	Sets a data transform expression.
herr_t	H5Pset_edc_check (hid_t plist_id, H5Z_EDC_t check)
	Sets the dataset transfer property list to enable or disable error detection when reading data.
herr_t	H5Pset_filter_callback (hid_t plist_id, H5Z_filter_func_t func, void *op_data)
	Sets user-defined filter callback function.
herr_t	H5Pset_hyper_vector_size (hid_t plist_id, size_t size)
	Sets number of I/O vectors to be read/written in hyperslab I/O.
herr_t	H5Pset_preserve (hid_t plist_id, hbool_t status)
	Sets the dataset transfer property list status.
herr_t	H5Pset_type_conv_cb (hid_t dxpl_id, H5T_conv_except_func_t op, void *operate_data)
	Sets user-defined datatype conversion callback function.
herr_t	H5Pset_vlen_mem_manager (hid_t plist_id, H5MM_allocate_t alloc_func, void *alloc_info, H5MM_free_t free_func, void *free_info)
	Sets the memory manager for variable-length datatype allocation in H5Dread() and H5Dvlen_reclaim()
herr_t	H5Pget_mpio_actual_chunk_opt_mode (hid_t plist_id, H5D_mpio_actual_chunk_opt_mode_t *actual_chunk_opt_mode)
	Retrieves the type of chunk optimization that HDF5 actually performed on the last parallel I/O call (not necessarily the type requested)
herr_t	H5Pget_mpio_actual_io_mode (hid_t plist_id, H5D_mpio_actual_io_mode_t *actual_io_mode)
	Retrieves the type of I/O that HDF5 actually performed on the last parallel I/O call (not necessarily the type requested)
herr_t	H5Pget_mpio_no_collective_cause (hid_t plist_id, uint32_t *local_no_collective_cause, uint32_t *global_no_collective_cause)
	Retrieves local and global causes that broke collective I/O on the last parallel I/O call.
herr_t	H5Pset_dataset_io_hyperslab_selection (hid_t plist_id, unsigned rank, H5S_seloper_t op, const hsize_t start[], const hsize_t stride[], const hsize_t count[], const hsize_t block[])
	Sets a hyperslab file selection for a dataset I/O operation.
herr_t	H5Pset_selection_io (hid_t plist_id, H5D_selection_io_mode_t selection_io_mode)
	Sets the selection I/O mode.
herr_t	H5Pget_selection_io (hid_t plist_id, H5D_selection_io_mode_t *selection_io_mode)
	Retrieves the selection I/O mode.
herr_t	H5Pget_no_selection_io_cause (hid_t plist_id, uint32_t *no_selection_io_cause)
	Retrieves the cause for not performing selection or vector I/O on the last parallel I/O call.
herr_t	H5Pget_actual_selection_io_mode (hid_t plist_id, uint32_t *actual_selection_io_mode)
	Retrieves the type(s) of I/O that HDF5 actually performed on raw data during the last I/O call.
herr_t	H5Pset_modify_write_buf (hid_t plist_id, hbool_t modify_write_buf)
	Allows the library to modify the contents of the write buffer.
herr_t	H5Pget_modify_write_buf (hid_t plist_id, hbool_t *modify_write_buf)
	Retrieves the "modify write buffer" property.

Function Documentation

H5Pget_actual_selection_io_mode()

herr_t H5Pget_actual_selection_io_mode	(	hid_t	plist_id,
		uint32_t *	actual_selection_io_mode
	)

Retrieves the type(s) of I/O that HDF5 actually performed on raw data during the last I/O call.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[out]	actual_selection_io_mode	A bitwise set value indicating the type(s) of I/O performed

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

Motivation:

A user can request selection I/O to be performed via a data transfer property list (DXPL). This can be used to enable collective I/O with type conversion, or with custom VFDs that support vector or selection I/O. However, there are conditions that can cause HDF5 to forgo selection or vector I/O and perform legacy (scalar) I/O instead. This function allows the user to determine which type or types of I/O were actually performed.

H5Pget_actual_selection_io_mode() allows the user to determine which type(s) of I/O were actually performed on raw data during the last I/O operation which used plist_id. This property is set after all I/O is completed; if I/O fails, it will not be set.

H5Pget_no_selection_io_cause() can be used to determine the reason why selection or vector I/O was not performed.

Valid bitflags returned in actual_selection_io_mode are listed as follows.

• H5D_SCALAR_IO Scalar (or legacy MPIO) I/O was performed
• H5D_VECTOR_IO Vector I/O was performed
• H5D_SELECTION_IO Selection I/O was performed

0 or more of these can be present in actual_selection_io_mode in a bitwise fashion, since a single operation can trigger multiple instances of I/O, possibly with different types. A value of 0 indicates no raw data I/O was performed during the operation.

Be aware that this function will only include raw data I/O performed to/from disk as part of the last I/O operation. Any metadata I/O, including attribute and compact dataset I/O, is disregarded. It is also possible that data was cached in the dataset chunk cache or sieve buffer, which may prevent I/O from hitting the disk, and thereby prevent it from being counted by this function.

Since

1.14.3

H5Pget_btree_ratios()

herr_t H5Pget_btree_ratios	(	hid_t	plist_id,
		double *	left,
		double *	middle,
		double *	right
	)

Gets B-tree split ratios for a dataset transfer property list.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[out]	left	The B-tree split ratio for left-most nodes
[out]	middle	The B-tree split ratio for right-most nodes and lone nodes
[out]	right	The B-tree split ratio for all other nodes

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pget_btree_ratios() returns the B-tree split ratios for a dataset transfer property list.

The B-tree split ratios are returned through the non-NULL arguments left, middle, and right, as set by the H5Pset_btree_ratios() function.

Since

1.0.0

H5Pget_buffer()

size_t H5Pget_buffer	(	hid_t	plist_id,
		void **	tconv,
		void **	bkg
	)

Reads buffer settings.

Parameters

[in]	plist_id	Identifier for the dataset transfer property list
[out]	tconv	Address of the pointer to application-allocated type conversion buffer
[out]	bkg	Address of the pointer to application-allocated background buffer

Returns

Returns buffer size, in bytes, if successful; otherwise 0 on failure.

H5Pget_buffer() reads values previously set with H5Pset_buffer().

Version

1.6.0 The return type changed from hsize_t to size_t.

1.4.0 The return type changed to hsize_t.

Since

1.0.0

H5Pget_data_transform()

ssize_t H5Pget_data_transform	(	hid_t	plist_id,
		char *	expression,
		size_t	size
	)

Retrieves a data transform expression.

Parameters

[in]	plist_id	Identifier of the property list or class
[out]	expression	Pointer to memory where the transform expression will be copied
[in]	size	Number of bytes of the transform expression to copy to

Returns

Success: the size of the transform expression. Failure: a negative value.

H5Pget_data_transform() retrieves the data transform expression previously set in the dataset transfer property list plist_id by H5Pset_data_transform().

H5Pget_data_transform() can be used to both retrieve the transform expression and query its size.

If expression is non-NULL, up to size bytes of the data transform expression are written to the buffer. If expression is NULL, size is ignored, and the function does not write anything to the buffer. The function always returns the size of the data transform expression.

If 0 is returned for the size of the expression, no data transform expression exists for the property list.

If an error occurs, the buffer pointed to by expression is unchanged, and the function returns a negative value.

Since

1.8.0

H5Pget_dxpl_mpio()

H5_DLL herr_t H5Pget_dxpl_mpio	(	hid_t	dxpl_id,
		H5FD_mpio_xfer_t *	xfer_mode
	)

Returns the data transfer mode.

Parameters

[in]	dxpl_id	Dataset transfer property list identifier
[out]	xfer_mode	Transfer mode

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pget_dxpl_mpio() queries the data transfer mode currently set in the data transfer property list dxpl_id.

Upon return, xfer_mode contains the data transfer mode, if it is non-null.

H5Pget_dxpl_mpio() is not a collective function.

Since

1.4.0

H5Pget_edc_check()

H5Z_EDC_t H5Pget_edc_check

(

hid_t

plist_id

)

Determines whether error-detection is enabled for dataset reads.

Parameters

[in]

plist_id

Dataset transfer property list identifier

Returns

Returns H5Z_ENABLE_EDC or H5Z_DISABLE_EDC if successful; otherwise returns a negative value.

H5Pget_edc_check() queries the dataset transfer property list plist to determine whether error detection is enabled for data read operations.

Since

1.6.0

H5Pget_hyper_vector_size()

herr_t H5Pget_hyper_vector_size	(	hid_t	fapl_id,
		size_t *	size
	)

Retrieves number of I/O vectors to be read/written in hyperslab I/O.

Parameters

[in]	fapl_id	Dataset transfer property list identifier
[out]	size	Number of I/O vectors to accumulate in memory for I/O operations

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pget_hyper_vector_size() retrieves the number of I/O vectors to be accumulated in memory before being issued to the lower levels of the HDF5 library for reading or writing the actual data.

The number of I/O vectors set in the dataset transfer property list fapl_id is returned in size. Unless the default value is in use, size was previously set with a call to H5Pset_hyper_vector_size().

Since

1.6.0

H5Pget_modify_write_buf()

herr_t H5Pget_modify_write_buf	(	hid_t	plist_id,
		hbool_t *	modify_write_buf
	)

Retrieves the "modify write buffer" property.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[out]	modify_write_buf	Whether the library can modify the contents of the write buffer

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pget_modify_write_buf() gets the "modify write buffer" property from the dataset transfer property list plist_id. This property determines whether the library is allowed to modify the contents of write buffers passed to HDF5 API routines that are passed plist_id. The default value for modify_write_buf is false.

Since

1.14.1

H5Pget_mpio_actual_chunk_opt_mode()

herr_t H5Pget_mpio_actual_chunk_opt_mode	(	hid_t	plist_id,
		H5D_mpio_actual_chunk_opt_mode_t *	actual_chunk_opt_mode
	)

Retrieves the type of chunk optimization that HDF5 actually performed on the last parallel I/O call (not necessarily the type requested)

Parameters

[in]	plist_id	Dataset transfer property list identifier
[out]	actual_chunk_opt_mode	The type of chunk optimization performed by HDF5

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

Motivation:

A user can request collective I/O via a data transfer property list (DXPL) that has been suitably modified with H5Pset_dxpl_mpio(). However, HDF5 will sometimes ignore this request and perform independent I/O instead. This property allows the user to see what kind of I/O HDF5 actually performed. Used in conjunction with H5Pget_mpio_actual_io_mode(), this property allows the user to determine exactly what HDF5 did when attempting collective I/O.

H5Pget_mpio_actual_chunk_opt_mode() retrieves the type of chunk optimization performed when collective I/O was requested. This property is set before I/O takes place, and will be set even if I/O fails.

Valid values returned in actual_chunk_opt_mode:

 
typedef enum H5D_mpio_actual_chunk_opt_mode_t {
    H5D_MPIO_NO_CHUNK_OPTIMIZATION = 0,
    H5D_MPIO_LINK_CHUNK,
    H5D_MPIO_MULTI_CHUNK
} H5D_mpio_actual_chunk_opt_mode_t;

(Click on a enumerator, field, or type for more information.)

Since

1.8.8

H5Pget_mpio_actual_io_mode()

herr_t H5Pget_mpio_actual_io_mode	(	hid_t	plist_id,
		H5D_mpio_actual_io_mode_t *	actual_io_mode
	)

Retrieves the type of I/O that HDF5 actually performed on the last parallel I/O call (not necessarily the type requested)

Parameters

[in]	plist_id	Dataset transfer property list identifier
[out]	actual_io_mode	The type of I/O performed by this process

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

Motivation:

A user can request collective I/O via a data transfer property list (DXPL) that has been suitably modified with H5Pset_dxpl_mpio(). However, HDF5 will sometimes ignore this request and perform independent I/O instead. This property allows the user to see what kind of I/O HDF5 actually performed. Used in conjunction with H5Pget_mpio_actual_chunk_opt_mode(), this property allows the user to determine exactly HDF5 did when attempting collective I/O.

H5Pget_mpio_actual_io_mode() retrieves the type of I/O performed on the selection of the current process. This property is set after all I/O is completed; if I/O fails, it will not be set.

Valid values returned in actual_io_mode:

 
typedef enum H5D_mpio_actual_io_mode_t {
    H5D_MPIO_NO_COLLECTIVE = 0x0,
    H5D_MPIO_CHUNK_INDEPENDENT = 0x1,
    H5D_MPIO_CHUNK_COLLECTIVE = 0x2,
    H5D_MPIO_CHUNK_MIXED = 0x1 | 0x2,
    H5D_MPIO_CONTIGUOUS_COLLECTIVE = 0x4
} H5D_mpio_actual_io_mode_t;

(Click on a enumerator, field, or type for more information.)

Attention

All processes do not need to have the same value. For example, if I/O is being performed using the multi chunk optimization scheme, one process's selection may include only chunks accessed collectively, while another may include chunks accessed independently. In this case, the first process will report H5D_MPIO_CHUNK_COLLECTIVE while the second will report H5D_MPIO_CHUNK_INDEPENDENT.

See also

H5Pget_mpio_no_collective_cause(), H5Pget_mpio_actual_chunk_opt_mode()

Since

1.8.8

H5Pget_mpio_no_collective_cause()

herr_t H5Pget_mpio_no_collective_cause	(	hid_t	plist_id,
		uint32_t *	local_no_collective_cause,
		uint32_t *	global_no_collective_cause
	)

Retrieves local and global causes that broke collective I/O on the last parallel I/O call.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[out]	local_no_collective_cause	An enumerated set value indicating the causes that prevented collective I/O in the local process
[out]	global_no_collective_cause	An enumerated set value indicating the causes across all processes that prevented collective I/O

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

Motivation:

A user can request collective I/O via a data transfer property list (DXPL) that has been suitably modified with H5P_SET_DXPL_MPIO. However, there are conditions that can cause HDF5 to forgo collective I/O and perform independent I/O. Such causes can be different across the processes of a parallel application. This function allows the user to determine what caused the HDF5 library to skip collective I/O locally, that is in the local process, and globally, across all processes.

H5Pget_mpio_no_collective_cause() serves two purposes. It can be used to determine whether collective I/O was used for the last preceding parallel I/O call. If collective I/O was not used, the function retrieves the local and global causes that broke collective I/O on that parallel I/O call. The properties retrieved by this function are set before I/O takes place and are retained even when I/O fails.

Valid values returned in local_no_collective_cause and global_no_collective_cause are as follows or, if there are multiple causes, a bitwise OR of the relevant causes; the numbers in the center column are the bitmask values:

 
typedef enum H5D_mpio_no_collective_cause_t {
    H5D_MPIO_COLLECTIVE = 0x00,
    H5D_MPIO_SET_INDEPENDENT = 0x01,
    H5D_MPIO_DATATYPE_CONVERSION = 0x02,
    H5D_MPIO_DATA_TRANSFORMS = 0x04,
    H5D_MPIO_MPI_OPT_TYPES_ENV_VAR_DISABLED = 0x08,
    H5D_MPIO_NOT_SIMPLE_OR_SCALAR_DATASPACES = 0x10,
    H5D_MPIO_NOT_CONTIGUOUS_OR_CHUNKED_DATASET = 0x20,
    H5D_MPIO_PARALLEL_FILTERED_WRITES_DISABLED = 0x40,
    H5D_MPIO_ERROR_WHILE_CHECKING_COLLECTIVE_POSSIBLE = 0x80,
    H5D_MPIO_NO_SELECTION_IO = 0x100,
    H5D_MPIO_NO_COLLECTIVE_MAX_CAUSE = 0x200
} H5D_mpio_no_collective_cause_t;

(Click on a enumerator, field, or type for more information.)

Attention

Each process determines whether it can perform collective I/O and broadcasts the result. Those results are combined to make a collective decision; collective I/O will be performed only if all processes can perform collective I/O.
If collective I/O was not used, the causes that prevented it are reported by individual process by means of an enumerated set. The causes may differ among processes, so H5Pget_mpio_no_collective_cause() returns two property values. The first value is the one produced by the local process to report local causes. This local information is encoded in an enumeration, the H5D_mpio_no_collective_cause_t described above, with all individual causes combined into a single enumeration value by means of a bitwise OR operation. The second value reports global causes; this global value is the result of a bitwise-OR operation across the values returned by all the processes.

Since

1.8.10

H5Pget_no_selection_io_cause()

herr_t H5Pget_no_selection_io_cause	(	hid_t	plist_id,
		uint32_t *	no_selection_io_cause
	)

Retrieves the cause for not performing selection or vector I/O on the last parallel I/O call.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[out]	no_selection_io_cause	A bitwise set value indicating the relevant causes that prevented selection I/O from being performed

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

Motivation:

A user can request selection I/O to be performed via a data transfer property list (DXPL). This can be used to enable collective I/O with type conversion, or with custom VFDs that support vector or selection I/O. However, there are conditions that can cause HDF5 to forgo selection or vector I/O and perform legacy (scalar) I/O instead.

H5Pget_no_selection_io_cause() can be used to determine whether selection or vector I/O was applied for the last preceding I/O call. If selection or vector I/O was not used, this function retrieves the cause(s) that prevent selection or vector I/O to be performed on that I/O call. The properties retrieved by this function are set before I/O takes place and are retained even when I/O fails.

If a selection I/O request falls back to vector I/O, that is not considered "breaking" selection I/O by this function, since vector I/O still passes all information to the file driver in a single callback.

Valid values returned in no_selection_io_cause are listed as follows. If there are multiple causes, it is a bitwise OR of the relevant causes.

• H5D_SEL_IO_DISABLE_BY_API Selection I/O was not performed because the feature was disabled by the API
• H5D_SEL_IO_NOT_CONTIGUOUS_OR_CHUNKED_DATASET Selection I/O was not performed because the dataset was neither contiguous nor chunked
• H5D_SEL_IO_CONTIGUOUS_SIEVE_BUFFER Selection I/O was not performed because of sieve buffer for contiguous dataset
• H5D_SEL_IO_NO_VECTOR_OR_SELECTION_IO_CB Selection I/O was not performed because the VFD does not have vector or selection I/O callback
• H5D_SEL_IO_PAGE_BUFFER Selection I/O was not performed because of page buffer
• H5D_SEL_IO_DATASET_FILTER Selection I/O was not performed because of dataset filters
• H5D_SEL_IO_CHUNK_CACHE Selection I/O was not performed because of chunk cache
• H5D_SEL_IO_TCONV_BUF_TOO_SMALL Selection I/O was not performed because the type conversion buffer is too small
• H5D_SEL_IO_BKG_BUF_TOO_SMALL Selection I/O was not performed because the type conversion background buffer is too small
• H5D_SEL_IO_DEFAULT_OFF Selection I/O was not performed because the selection I/O mode is DEFAULT and the library chose it to be off for this case

Since

1.14.1

H5Pget_preserve()

int H5Pget_preserve

(

hid_t

plist_id

)

Checks status of the dataset transfer property list (DEPRECATED)

Deprecated:

Parameters

[in]

plist_id

Identifier for the dataset transfer property list

Returns

Returns 1 or 0 if successful; otherwise returns a negative value.

H5Pget_preserve() checks the status of the dataset transfer property list.

Since

1.0.0

Version

1.6.0 The flag parameter was changed from INTEGER to LOGICAL to better match the C API. (Fortran 90)

1.8.2 Deprecated.

H5Pget_selection_io()

herr_t H5Pget_selection_io	(	hid_t	plist_id,
		H5D_selection_io_mode_t *	selection_io_mode
	)

Retrieves the selection I/O mode.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[out]	selection_io_mode	The selection I/O mode

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pget_selection_io() queries the selection I/O mode set in in the dataset transfer property list plist_id.

Values returned in selection_io_mode:

 
typedef enum H5D_selection_io_mode_t {
    H5D_SELECTION_IO_MODE_DEFAULT = 0,
    H5D_SELECTION_IO_MODE_OFF,
    H5D_SELECTION_IO_MODE_ON
} H5D_selection_io_mode_t;

(Click on a enumerator, field, or type for more information.)

Note

The library may not perform selection I/O as it asks for if the layout callback determines that it is not feasible to do so. Please refer to H5Pget_no_selection_io_cause() for details.

Since

1.14.1

H5Pget_type_conv_cb()

herr_t H5Pget_type_conv_cb	(	hid_t	dxpl_id,
		H5T_conv_except_func_t *	op,
		void **	operate_data
	)

Gets user-defined datatype conversion callback function.

Parameters

[in]	dxpl_id	Dataset transfer property list identifier
[out]	op	User-defined type conversion callback function
[out]	operate_data	User-defined input data for the callback function

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pget_type_conv_cb() gets the user-defined datatype conversion callback function op in the dataset transfer property list dxpl_id.

The parameter operate_data is a pointer to user-defined input data for the callback function.

The callback function op defines the actions an application is to take when there is an exception during datatype conversion.

Please refer to the function H5Pset_type_conv_cb() for more details.

Since

1.8.0

H5Pget_vlen_mem_manager()

herr_t H5Pget_vlen_mem_manager	(	hid_t	plist_id,
		H5MM_allocate_t *	alloc_func,
		void **	alloc_info,
		H5MM_free_t *	free_func,
		void **	free_info
	)

Gets the memory manager for variable-length datatype allocation in H5Dread() and H5Dvlen_reclaim()

Parameters

[in]	plist_id	Identifier for the dataset transfer property list
[out]	alloc_func	User's allocate routine, or NULL for system malloc
[out]	alloc_info	Extra parameter for user's allocation routine. Contents are ignored if preceding parameter is NULL
[out]	free_func	User's free routine, or NULL for system free
[out]	free_info	Extra parameter for user's free routine. Contents are ignored if preceding parameter is NULL

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pget_vlen_mem_manager() is the companion function to H5Pset_vlen_mem_manager(), returning the parameters set by that function.

Since

1.0.0

H5Pset_btree_ratios()

herr_t H5Pset_btree_ratios	(	hid_t	plist_id,
		double	left,
		double	middle,
		double	right
	)

Sets B-tree split ratios for a dataset transfer property list.

Parameters

[in]	plist_id	The dataset transfer property list identifier
[in]	left	The B-tree split ratio for left-most nodes
[in]	middle	The B-tree split ratio for all other nodes
[in]	right	The B-tree split ratio for right-most nodes and lone nodes

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_btree_ratios() sets the B-tree split ratios for a dataset transfer property list. The split ratios determine what percent of children go in the first node when a node splits.

The ratio left is used when the splitting node is the left-most node at its level in the tree; the ratio right is used when the splitting node is the right-most node at its level; and the ratio middle is used for all other cases.

A node that is the only node at its level in the tree uses the ratio right when it splits.

All ratios are real numbers between 0 and 1, inclusive.

Since

1.0.0

H5Pset_buffer()

herr_t H5Pset_buffer	(	hid_t	plist_id,
		size_t	size,
		void *	tconv,
		void *	bkg
	)

Sets type conversion and background buffers.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[in]	size	Size, in bytes, of the type conversion and background buffers
[in]	tconv	Pointer to application-allocated type conversion buffer
[in]	bkg	Pointer to application-allocated background buffer

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

Given a dataset transfer property list, H5Pset_buffer() sets the maximum size for the type conversion buffer and background buffer and optionally supplies pointers to application-allocated buffers. If the buffer size is smaller than the entire amount of data being transferred between the application and the file, and a type conversion buffer or background buffer is required, then strip mining will be used.

Note that there are minimum size requirements for the buffer. Strip mining can only break the data up along the first dimension, so the buffer must be large enough to accommodate a complete slice that encompasses all of the remaining dimensions. For example, when strip mining a 100x200x300 hyperslab of a simple data space, the buffer must be large enough to hold 1x200x300 data elements. When strip mining a 100x200x300x150 hyperslab of a simple data space, the buffer must be large enough to hold 1x200x300x150 data elements.

If tconv and/or bkg are null pointers, then buffers will be allocated and freed during the data transfer.

The default value for the maximum buffer is 1 MiB.

Version

1.6.0 The size parameter has changed from type hsize_t to size_t.

1.4.0 The size parameter has changed to type hsize_t.

Since

1.0.0

H5Pset_data_transform()

herr_t H5Pset_data_transform	(	hid_t	plist_id,
		const char *	expression
	)

Sets a data transform expression.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[in]	expression	Pointer to the null-terminated data transform expression

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_data_transform() sets the data transform to be used for reading and writing data. This function operates on the dataset transfer property list plist_id.

The expression parameter is a string containing an algebraic expression, such as (5/9.0)*(x-32) or x*(x-5). When a dataset is read or written with this property list, the transform expression is applied with the x being replaced by the values in the dataset. When reading data, the values in the file are not changed and the transformed data is returned to the user.

Data transforms can only be applied to integer or floating-point datasets. Order of operations is obeyed and the only supported operations are +, -, *, and /. Parentheses can be nested arbitrarily and can be used to change precedence. When writing data back to the dataset, the transformed data is written to the file and there is no way to recover the original values to which the transform was applied.

Since

1.8.0

H5Pset_dataset_io_hyperslab_selection()

herr_t H5Pset_dataset_io_hyperslab_selection	(	hid_t	plist_id,
		unsigned	rank,
		H5S_seloper_t	op,
		const hsize_t	start[],
		const hsize_t	stride[],
		const hsize_t	count[],
		const hsize_t	block[]
	)

Sets a hyperslab file selection for a dataset I/O operation.

Parameters

[in]	plist_id	Property list identifier
[in]	rank	Number of dimensions of selection
[in]	op	Operation to perform on current selection
[in]	start	Offset of start of hyperslab
[in]	stride	Hyperslab stride
[in]	count	Number of blocks included in hyperslab
[in]	block	Size of block in hyperslab

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_dataset_io_hyperslab_selection() is designed to be used in conjunction with using H5S_PLIST for the file dataspace ID when making a call to H5Dread() or H5Dwrite(). When used with H5S_PLIST, the selection created by one or more calls to this routine is used for determining which dataset elements to access.

rank is the dimensionality of the selection and determines the size of the start, stride, count, and block arrays. rank must be between 1 and H5S_MAX_RANK, inclusive.

The op, start, stride, count, and block parameters behave identically to their behavior for H5Sselect_hyperslab(), please see the documentation for that routine for details about their use.

Since

1.14.0

H5Pset_dxpl_mpio()

H5_DLL herr_t H5Pset_dxpl_mpio	(	hid_t	dxpl_id,
		H5FD_mpio_xfer_t	xfer_mode
	)

Sets data transfer mode.

Parameters

[in]	dxpl_id	Dataset transfer property list identifier
[in]	xfer_mode	Transfer mode

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_dxpl_mpio() sets the data transfer property list dxpl_id to use transfer mode xfer_mode. The property list can then be used to control the I/O transfer mode during data I/O operations.

Valid transfer modes are H5FD_MPIO_INDEPENDENT (default) and H5FD_MPIO_COLLECTIVE.

Since

1.4.0

H5Pset_dxpl_mpio_chunk_opt()

H5_DLL herr_t H5Pset_dxpl_mpio_chunk_opt	(	hid_t	dxpl_id,
		H5FD_mpio_chunk_opt_t	opt_mode
	)

Sets a flag specifying linked-chunk I/O or multi-chunk I/O.

Parameters

[in]	dxpl_id	Dataset transfer property list identifier
[in]	opt_mode	Transfer mode

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_dxpl_mpio_chunk_opt() specifies whether I/O is to be performed as linked-chunk I/O or as multi-chunk I/O. This function overrides the HDF5 library's internal algorithm for determining which mechanism to use.

When an application uses collective I/O with chunked storage, the HDF5 library normally uses an internal algorithm to determine whether that I/O activity should be conducted as one linked-chunk I/O or as multi-chunk I/O. H5Pset_dxpl_mpio_chunk_opt() is provided so that an application can override the library's algorithm in circumstances where the library might lack the information needed to make an optimal decision.

H5Pset_dxpl_mpio_chunk_opt() works by setting one of the following flags in the parameter opt_mode:

• H5FD_MPIO_CHUNK_ONE_IO - Do one-link chunked I/O
• H5FD_MPIO_CHUNK_MULTI_IO - Do multi-chunked I/O

This function works by setting a corresponding property in the dataset transfer property list dxpl_id.

The library performs I/O in the specified manner unless it determines that the low-level MPI IO package does not support the requested behavior; in such cases, the HDF5 library will internally use independent I/O.

Use of this function is optional.

Since

1.8.0

H5Pset_dxpl_mpio_chunk_opt_num()

H5_DLL herr_t H5Pset_dxpl_mpio_chunk_opt_num	(	hid_t	dxpl_id,
		unsigned	num_chunk_per_proc
	)

Sets a numeric threshold for linked-chunk I/O.

Parameters

[in]	dxpl_id	Dataset transfer property list identifier
[in]	num_chunk_per_proc

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_dxpl_mpio_chunk_opt_num() sets a numeric threshold for the use of linked-chunk I/O.

The library will calculate the average number of chunks selected by each process when doing collective access with chunked storage. If the number is greater than the threshold set in num_chunk_per_proc, the library will use linked-chunk I/O; otherwise, a separate I/O process will be invoked for each chunk (multi-chunk I/O).

Since

1.8.0

H5Pset_dxpl_mpio_chunk_opt_ratio()

H5_DLL herr_t H5Pset_dxpl_mpio_chunk_opt_ratio	(	hid_t	dxpl_id,
		unsigned	percent_num_proc_per_chunk
	)

Sets a ratio threshold for collective I/O.

Parameters

[in]	dxpl_id	Dataset transfer property list identifier
[in]	percent_num_proc_per_chunk

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_dxpl_mpio_chunk_opt_ratio() sets a threshold for the use of collective I/O based on the ratio of processes with collective access to a dataset with chunked storage. The decision whether to use collective I/O is made on a per-chunk basis.

The library will calculate the percentage of the total number of processes, the ratio, that hold selections in each chunk. If that percentage is greater than the threshold set in percent_proc_per_chunk, the library will do collective I/O for this chunk; otherwise, independent I/O will be done for the chunk.

Since

1.8.0

H5Pset_dxpl_mpio_collective_opt()

H5_DLL herr_t H5Pset_dxpl_mpio_collective_opt	(	hid_t	dxpl_id,
		H5FD_mpio_collective_opt_t	opt_mode
	)

Sets low-level data transfer mode.

Parameters

[in]	dxpl_id	Dataset transfer property list identifier
[in]	opt_mode	Transfer mode

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_dxpl_mpio_collective_opt() sets the data transfer property list dxpl_id to use transfer mode opt_mode when performing I/O. This allows the application to specify collective I/O at the HDF5 interface level (with the H5Pset_dxpl_mpio() API routine), while controlling whether the actual I/O is performed collectively (e.g., via MPI_File_write_at_all) or independently (e.g., via MPI_File_write_at). If the collectivity setting at the HDF5 interface level (set via H5Pset_dxpl_mpio()) is not set to H5FD_MPIO_COLLECTIVE, this setting will be ignored.

Valid transfer modes are H5FD_MPIO_COLLECTIVE_IO (default) and H5FD_MPIO_INDIVIDUAL_IO.

Since

1.4.0

H5Pset_edc_check()

herr_t H5Pset_edc_check	(	hid_t	plist_id,
		H5Z_EDC_t	check
	)

Sets the dataset transfer property list to enable or disable error detection when reading data.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[in]	check	Specifies whether error checking is enabled or disabled for dataset read operations

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_edc_check() sets the dataset transfer property list plist to enable or disable error detection when reading data.

Whether error detection is enabled or disabled is specified in the check parameter. Valid values are H5Z_ENABLE_EDC (default) and H5Z_DISABLE_EDC.

Note

The initial error detection implementation, Fletcher32 checksum, supports error detection for chunked datasets only.

Attention

The Fletcher32 EDC checksum filter, set with H5Pset_fletcher32(), was added in HDF5 Release 1.6.0. In the original implementation, however, the checksum value was calculated incorrectly on little-endian systems. The error was fixed in HDF5 Release 1.6.3.
As a result of this fix, an HDF5 library of Release 1.6.0 through Release 1.6.2 cannot read a dataset created or written with Release 1.6.3 or later if the dataset was created with the checksum filter and the filter is enabled in the reading library. (Libraries of Release 1.6.3 and later understand the earlier error and compensate appropriately.)
Work-around: An HDF5 library of Release 1.6.2 or earlier will be able to read a dataset created or written with the checksum filter by an HDF5 library of Release 1.6.3 or later if the checksum filter is disabled for the read operation. This can be accomplished via an H5Pset_edc_check() call with the value H5Z_DISABLE_EDC in the second parameter. This has the obvious drawback that the application will be unable to verify the checksum, but the data does remain accessible.

Version

1.6.3 Error in checksum calculation on little-endian systems corrected in this release.

Since

1.6.0

H5Pset_filter_callback()

herr_t H5Pset_filter_callback	(	hid_t	plist_id,
		H5Z_filter_func_t	func,
		void *	op_data
	)

Sets user-defined filter callback function.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[in]	func	User-defined filter callback function
[in]	op_data	User-defined input data for the callback function

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_filter_callback() sets the user-defined filter callback function func in the dataset transfer property list plist_id.

The parameter op_data is a pointer to user-defined input data for the callback function and will be passed through to the callback function.

The callback function func defines the actions an application is to take when a filter fails. The function prototype is as follows:

 
typedef H5Z_cb_return_t (*H5Z_filter_func_t)(H5Z_filter_t filter, void *buf, size_t buf_size, void *op_data);

where filter indicates which filter has failed, buf and buf_size are used to pass in the failed data, and op_data is the required input data for this callback function.

Valid callback function return values are H5Z_CB_FAIL and H5Z_CB_CONT.

Since

1.6.0

H5Pset_hyper_vector_size()

herr_t H5Pset_hyper_vector_size	(	hid_t	plist_id,
		size_t	size
	)

Sets number of I/O vectors to be read/written in hyperslab I/O.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[in]	size	Number of I/O vectors to accumulate in memory for I/O operations Must be greater than 1 (one) Default value: 1024

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_hyper_vector_size() sets the number of I/O vectors to be accumulated in memory before being issued to the lower levels of the HDF5 library for reading or writing the actual data.

The I/O vectors are hyperslab offset and length pairs and are generated during hyperslab I/O.

The number of I/O vectors is passed in size to be set in the dataset transfer property list plist_id. size must be greater than 1 (one).

H5Pset_hyper_vector_size() is an I/O optimization function; increasing vector_size should provide better performance, but the library will use more memory during hyperslab I/O. The default value of size is 1024.

Since

1.6.0

H5Pset_modify_write_buf()

herr_t H5Pset_modify_write_buf	(	hid_t	plist_id,
		hbool_t	modify_write_buf
	)

Allows the library to modify the contents of the write buffer.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[in]	modify_write_buf	Whether the library can modify the contents of the write buffer

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_modify_write_buf() sets whether the library is allowed to modify the contents of write buffers passed to HDF5 API routines that are passed the dataset transfer property list plist_id. The default value for modify_write_buf is false.

This function can be used to allow the library to perform in-place type conversion on write operations to save memory space. After making an API call with this parameter set to true, the contents of the write buffer are undefined.

Note

When modify_write_buf is set to true the library may violate the const qualifier on the API parameter for the write buffer.

Since

1.14.1

H5Pset_preserve()

herr_t H5Pset_preserve	(	hid_t	plist_id,
		hbool_t	status
	)

Sets the dataset transfer property list status.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[in]	status	Status toggle of the dataset transfer property list

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

Deprecated:

This function is deprecated as it no longer has any effect; compound datatype field preservation is now core functionality in the HDF5 library.

H5Pset_preserve() sets the dataset transfer property list status to 1 or 0.

When reading or writing compound datatypes and the destination is partially initialized and the read/write is intended to initialize the other members, one must set this property to 1. Otherwise the I/O pipeline treats the destination datapoints as completely uninitialized.

Since

1.0.0

Version

1.8.2 Deprecated.

H5Pset_selection_io()

herr_t H5Pset_selection_io	(	hid_t	plist_id,
		H5D_selection_io_mode_t	selection_io_mode
	)

Sets the selection I/O mode.

Parameters

[in]	plist_id	Dataset transfer property list identifier
[in]	selection_io_mode	The selection I/O mode to be set

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_selection_io() sets the selection I/O mode selection_io_mode in the dataset transfer property list plist_id.

This can be used to enable collective I/O with type conversion, or with custom VFDs that support vector or selection I/O.

Values that can be set in selection_io_mode:

 
typedef enum H5D_selection_io_mode_t {
    H5D_SELECTION_IO_MODE_DEFAULT = 0,
    H5D_SELECTION_IO_MODE_OFF,
    H5D_SELECTION_IO_MODE_ON
} H5D_selection_io_mode_t;

(Click on a enumerator, field, or type for more information.)

Note

The library may not perform selection I/O as it asks for if the layout callback determines that it is not feasible to do so. Please refer to H5Pget_no_selection_io_cause() for details.

When used with type conversion, selection I/O requires the type conversion buffer (and the background buffer if applicable) be large enough to hold the entirety of the data involved in the I/O. For read operations, the library will use the application's read buffer as the type conversion buffer if the memory type is not smaller than the file type, eliminating the need for a separate type conversion buffer (a background buffer may still be required). For write operations, the library will similarly use the write buffer as a type conversion buffer, but only if H5Pset_modify_write_buf() is used to allow the library to modify the contents of the write buffer.

Since

1.14.1

H5Pset_type_conv_cb()

herr_t H5Pset_type_conv_cb	(	hid_t	dxpl_id,
		H5T_conv_except_func_t	op,
		void *	operate_data
	)

Sets user-defined datatype conversion callback function.

Parameters

[in]	dxpl_id	Dataset transfer property list identifier
[in]	op	User-defined type conversion callback function
[in]	operate_data	User-defined input data for the callback function

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_type_conv_cb() sets the user-defined datatype conversion callback function op in the dataset transfer property list dxpl_id

The parameter operate_data is a pointer to user-defined input data for the callback function and will be passed through to the callback function.

The callback function op defines the actions an application is to take when there is an exception during datatype conversion. The function prototype is as follows:

 
typedef H5T_conv_ret_t (*H5T_conv_except_func_t)(H5T_conv_except_t except_type, hid_t src_id, hid_t dst_id,
                                                 void *src_buf, void *dst_buf, void *user_data);

Since

1.8.0

H5Pset_vlen_mem_manager()

herr_t H5Pset_vlen_mem_manager	(	hid_t	plist_id,
		H5MM_allocate_t	alloc_func,
		void *	alloc_info,
		H5MM_free_t	free_func,
		void *	free_info
	)

Sets the memory manager for variable-length datatype allocation in H5Dread() and H5Dvlen_reclaim()

Parameters

[in]	plist_id	Dataset transfer property list identifier
[in]	alloc_func	User's allocate routine, or NULL for system malloc
[in]	alloc_info	Extra parameter for user's allocation routine. Contents are ignored if preceding parameter is NULL.
[in]	free_func	User's free routine, or NULL for system free
[in]	free_info	Extra parameter for user's free routine. Contents are ignored if preceding parameter is NULL

Returns

Returns a non-negative value if successful; otherwise, returns a negative value.

H5Pset_vlen_mem_manager() sets the memory manager for variable-length datatype allocation in H5Dread() and free in H5Dvlen_reclaim().

The alloc_func and free_func parameters identify the memory management routines to be used. If the user has defined custom memory management routines, alloc_func and/or free_func should be set to make those routine calls (i.e., the name of the routine is used as the value of the parameter); if the user prefers to use the system's malloc and/or free, the alloc_func and free_func parameters, respectively, should be set to NULL

The prototypes for these user-defined functions are as follows:

typedef void *(*H5MM_allocate_t)(size_t size, void *alloc_info);

typedef void (*H5MM_free_t)(void *mem, void *free_info);

The alloc_info and free_info parameters can be used to pass along any required information to the user's memory management routines.

In summary, if the user has defined custom memory management routines, the name(s) of the routines are passed in the alloc_func and free_func parameters and the custom routines' parameters are passed in the alloc_info and free_info parameters. If the user wishes to use the system malloc and free functions, the alloc_func and/or free_func parameters are set to NULL and the alloc_info and free_info parameters are ignored.

Since

1.0.0