In developing and testing these examples, the Sun Solaris OS version supported by HDF version 4.1 release 1 was used. Version 2.0 of the Quantify performance profiler was used to measure the relative differences in library performance between the SDS models in each pair. It should be noted that, while the examples reliably reflect which SDS configurations result in better performance, the specifics of how much performance will be improved depend on many factors such as OS configuration, compiler used and profiler used. Therefore, any specific measurements of performance mentioned in the chapter should be interpreted only as general indicators.
The reader should keep in mind that the following examples have been designed for illustrative purposes only, and should not be considered as real-world examples. It is expected that the reader will apply the library performance concepts covered by these examples to their specific usage of the HDF library.
An opportunity for performance enhancement can exist when the size of the metadata far exceeds the size of the data described by the metadata. In this situation, more CPU time and disk space will be used to maintain the metadata than the data contained in the SDS. Consolidating the data into fewer, or even one, SDS can increase performance.
To illustrate this, consider 1,000 1 x 1 x 1 element scientific data sets of 32-bit floating-point numbers. No user-defined dimension, dimension scales or fill values have been defined or created.
In this example, 1,000 32-bit floating-point numbers are first buffered in-core, then written as 1,000 SDSs.
In Table 14A, the results of this operation are reflected in two metrics: the total number of CPU cycles used by the example program, and the size of the HDF file after the write operation.
TABLE 14A - Results of the Write Operation to 1,000 1 x 1 x 1 Element Scientific Data Sets
|
|
|
|
As with the last example, 1,000 32-bit floating-point numbers are first buffered in-core, then written to a single SDS. The following table contains the performance metrics of this operation.
|
|
|
|
The extent to which the data consolidation described in this section should be done is dependent on the specific I/O requirements of the HDF user application.
To illustrate this, again consider the example of 1,000 1 x 1 x 1 scientific data sets of 32-bit floating point numbers. Three dimensions are attached by default to each scientific data set by the HDF library. The HDF library assigns each of these dimensions a default name prefaced by the string
fakeDim
. See Chapter 3, Scientific Data Sets (SD API), for a specific explanation of default dimension naming conventions.
FIGURE 14c - 1,000 1 x 1 x 1 Element Scientific Data Sets
TABLE 14C - Results of the Write Operation to 1,000 1 x 1 x 1 Element Scientific Data Sets
|
|
|
|
X_Axis
, Y_Axis
and Z_Axis
dimensions as illustrated in the following figure.
The performance metrics that result from writing one 32-bit floating-point number to each dataset are in the following table.
|
|
|
|
The library's default writing of fill values can degrade performance when, after the fill values have been written, every element in the dataset is written to again. This operation involves writing every element in the SDS twice. This section will demonstrate that disabling the initial fill value write operation by calling SDsetfillmode can improve library performance.
Consider 50 10 x 10 x 10 scientific data sets of 32-bit floating-point numbers.
By default, the fill value is written to every element in all 50 SDSs. The contents of a two-dimensional buffer containing 32-bit floating-point numbers is then written to these datasets. The way these two-dimensional slices are written to the three-dimensional SDSs is illustrated in the following figure. Each slice (represented by each shaded area in the figure) is written along the third dimension of each SDS, or if the dimensions are related to a Cartesian grid, the z-dimension, until the entire SDS is filled.
It should be noted that the reason each SDS is not rewritten to in one write operation is because the HDF library will detect this and automatically disable the initial write of the fill values as a performance-saving measure. Hence, the partial writes in two-dimensional slabs.
The following table shows the number of CPU cycles needed in our tests to perform this write operation with the fill value write enabled. The "Size of the HDF File" metric has been left out of this table, because it will not change substantially regardless of whether the default fill value write operation is enabled.
|
|
|
|
In HDF version 4.0 a new representation of the dimension scale was implemented alongside the old one -- a vdata containing only one value representing the total number of values in the dimension scale. In version 4.1 release 2, this representation was made the default. A compatible mode is also supported where both the older and newer representations of the dimension scale are written to file.
In the earlier representation, a substantial amount of I/O overhead is involved in writing the fake dimension scale values into the vdata. When one of the dimensions of the SDS array is very large, performance can be improved, and the size of the HDF file can be reduced, if the old representation of dimension scales is disabled by a call to the SDsetdimval_comp routine. The examples in this section will illustrate this.
First, consider one 10,000 element array of 32-bit floating point numbers, as shown in the following figure. Both the new and old dimension scale representations are enabled by the library.
10,000 32-bit floating-point numbers are buffered in-core, then written to the scientific data set. In addition, 10,000 integers are written to the SDS as dimension scale values. The following table contains the results of this operation from our tests.
|
|
|
|
The following table contains the performance metrics of this write operation.
|
|
|
|