Help us improve by taking our short survey: https://www.hdfgroup.org/website-survey/

| HDF5 Last Updated on 2025-11-11 The HDF5 Field Guide

- Main Page
- Getting started
- User Guide
- Reference Manual
- Cookbook
- Technical Notes
- RFCs
- Specifications
- Glossary
- Full-Text Search
- About

Loading...
Searching...
No Matches
Installation Instructions for Parallel HDF5

# 1. Overview

This file contains instructions for installing parallel HDF5 (PHDF5) using CMake. The document covers:

- Section 1: Requirements and prerequisites
- Section 2: Obtaining HDF5 source
- Section 3: Quick start instructions
- Section 4: Automated builds with ctest (HPC systems)
- Section 5: Manual CMake configuration
- Section 6: Cross-compiling for HPC hardware
- Section 7: Running parallel tests
- Appendix A: Sample MPI-IO programs

## 1.1. Requirements

PHDF5 requires:

- CMake version 3.26 or greater
- An MPI compiler with MPI-IO support
- A POSIX compliant parallel file system (see References)

You should first consult with your system support staff for information on:

- How to compile an MPI program
- How to run an MPI application
- How to access the parallel file system

Sample MPI-IO C and Fortran programs are provided in Appendix A. Use them to test your MPI compiler and parallel file system before building HDF5.

## 1.2. Prerequisites for HPC Systems

When building on HPC systems:

1. Create a working directory accessible from compute nodes for running tests. Use a scratch space or parallel file system - NOT your home directory, as this typically causes test failures.
2. Load required modules:
   - Desired compiler modules (and set CC, FC, CXX if needed)
   - CMake version 3.26 or greater
   - MPI implementation module

3. For Cray and other systems with recommend compiler wrappers, set compiler environment variables AFTER loading modules:

   export CC=cc
   export FC=ftn
   export CXX=CC

## 1.3. Further Help

For installation help, post questions to the HDF Forum or HDF Support:

- HDF Forum: https://forum.hdfgroup.org/
- HDF Support: https://support.hdfgroup.org/

Include the output of `uname -a` and the contents of `CMakeCache.txt` and `CMakeError.log` from your build directory, and the loaded modules if applicable.

# 2. Obtaining HDF5 Source

Obtain HDF5 source code from the HDF5 repository or from a release tar file:

git clone https://github.com/HDFGroup/hdf5.git [-b branch] [directory]

If no branch is specified, the `develop` branch will be checked out. If no directory is specified, source will be in the `hdf5` directory.

For release or snapshot tar files, extract them to your working directory.

> Note: When using the ctest automated build method (Section 4), the source directory should be named `hdf5-<version>`, where version uses format `1.xx.xx` or `2.xx.xx`. Use `bin/h5vers` to determine the version string if needed.

# 3. Quick Start Instructions

## 3.1. Using CMake Presets (Recommended for General Builds)

For building with CMake 3.26 or greater using presets:

cd hdf5
cmake --workflow --preset ci-StdShar-GNUC --fresh

> Note: Standard presets do not enable parallel by default. To enable parallel support, you need to create a custom preset in `CMakeUserPresets.json` that sets `-DHDF5_ENABLE_PARALLEL=ON`, or use the standard CMake build approach below.

## 3.2. Standard CMake Build (Recommended for Parallel)

For a basic parallel build:

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release \
-DHDF5_ENABLE_PARALLEL=ON \
-DBUILD_SHARED_LIBS=ON \
-DBUILD_TESTING=ON \
..
cmake --build . --config Release
ctest . -C Release
cmake --install . --prefix /path/to/install

## 3.3. Specifying MPI Compiler

CMake can usually find MPI automatically. To specify explicitly:

cmake -DCMAKE_C_COMPILER=mpicc \
-DCMAKE_Fortran_COMPILER=mpif90 \
-DHDF5_ENABLE_PARALLEL=ON \
..

Or use MPI compiler wrappers:

export CC=mpicc
export FC=mpif90
cmake -DHDF5_ENABLE_PARALLEL=ON ..

## 3.4. Important Configuration Options

Option | Description
--- | ---
`-DHDF5_ENABLE_PARALLEL=ON` | Enable parallel HDF5 (required)
`-DBUILD_SHARED_LIBS=ON` | Build shared libraries
`-DBUILD_STATIC_LIBS=ON` | Build static libraries
`-DHDF5_BUILD_FORTRAN=ON` | Build Fortran interface
`-DHDF5_BUILD_CPP_LIB=OFF` | C++ disabled in parallel builds
`-DHDF5_ENABLE_THREADSAFE=OFF` | Thread safety disabled in parallel builds
`-DHDF5_ENABLE_SUBFILING_VFD=ON` | Enable subfiling VFD (parallel I/O optimization)
`-DMPIEXEC_EXECUTABLE=mpiexec` | MPI launcher executable
`-DMPIEXEC_NUMPROC_FLAG=-n` | MPI flag for number of processes
`-DMPIEXEC_MAX_NUMPROCS=6` | Number of processes for tests

> Note: Some MPI implementations (e.g., OpenMPI 4.0+) disallow oversubscribing by default. Set `MPIEXEC_MAX_NUMPROCS` to available processors or fewer, or add appropriate flags (e.g., `--oversubscribe` for OpenMPI).

# 4. Automated Builds with ctest (HPC Systems)

The ctest command provides automated configure, build, test, and package workflow for HPC systems with batch schedulers.

## 4.1. Setup Steps

1. Rename source directory to `hdf5-<version>` (e.g., `hdf5-2.0.0`)
2. Copy or link these CMake scripts to your working directory:
   - `hdf5-<version>/config/cmake/scripts/HDF5config.cmake`
   - `hdf5-<version>/config/cmake/scripts/CTestScript.cmake`
   - `hdf5-<version>/config/cmake/scripts/HDF5options.cmake`
3. Your working directory should contain:
   CTestScript.cmake
   HDF5config.cmake
   HDF5options.cmake
   hdf5-<version>/

## 4.2. Running ctest

Basic ctest command:

ctest -S HDF5config.cmake,BUILD_GENERATOR=Unix -C Release -V -O hdf5.log

For parallel builds on HPC systems with batch schedulers:

ctest -S HDF5config.cmake,HPC=sbatch,MPI=true -C Release -V -O hdf5.log

### Available HPC Options

Add after `HDF5config.cmake,` separated by commas:

Option | Description
--- | ---
`HPC=sbatch` | Use SLURM batch system
`HPC=bsub` | Use LSF batch system
`MPI=true` | Enable parallel (disables C++, Java, threadsafe)
`LOCAL_BATCH_SCRIPT_ARGS="--account=<acct>"` | Supply batch job account information

### Examples

SLURM system with parallel:

ctest -S HDF5config.cmake,HPC=sbatch,MPI=true \
-C Release -V -O hdf5.log

Use `-VV` instead of `-V` for more detailed logging.

# 5. Manual CMake Configuration

If the automated ctest approach is not suitable, you can manually configure and build HDF5.

## 5.1. Create Build Directory

mkdir build && cd build

## 5.2. Run CMake Configure

Example for parallel build with Fortran on HPC system:

cmake \
-C ../hdf5-<version>/config/cmake/cacheinit.cmake \
-DCMAKE_BUILD_TYPE:STRING=Release \
-DCMAKE_INSTALL_PREFIX:PATH=/install/path \
-DHDF5_ENABLE_PARALLEL:BOOL=ON \
-DHDF5_BUILD_FORTRAN:BOOL=ON \
-DHDF5_BUILD_CPP_LIB:BOOL=OFF \
-DHDF5_BUILD_JAVA:BOOL=OFF \
-DHDF5_ENABLE_THREADSAFE:BOOL=OFF \
-DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF \
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
-DMPIEXEC_EXECUTABLE:STRING=srun \
-DMPIEXEC_NUMPROC_FLAG:STRING=-n \
-DMPIEXEC_MAX_NUMPROCS:STRING=6 \
-G"Unix Makefiles" \
../hdf5-<version>

## 5.3. Build

cmake --build . --config Release -j 8

## 5.4. Test

For systems where you can run MPI directly:

ctest . -C Release

For batch systems, create and submit batch scripts (see section 7.3).

## 5.5. Install

cmake --install . --prefix /install/path

# 6. Cross-Compiling for HPC Hardware

## 6.1. Overview

Cross-compiling is the process of building software on one system architecture (like a login node) to be run on a different architecture (like a compute node). This section provides a historical overview of how this was done on systems that are no longer in service.

## 6.2. Historical Example: Knights Landing (KNL) on Cray XC40

A common historical use case was compiling for Intel Knights Landing (KNL) nodes on Cray XC40 systems, such as the retired Mutrino and Cori machines. These supercomputers had login nodes with a standard CPU architecture (e.g., Haswell) but used the different KNL architecture for their compute nodes.

To build software for KNL, a "module swapping" technique was required. The build process involved:

1. Loading the compiler module for the login node architecture (e.g., `craype-haswell`) to configure the project.
2. Switching to the compiler module for the compute node architecture (e.g., `craype-mic-knl`) before starting the actual compilation.

This process was managed by special CMake toolchain files and custom batch scripts, which were often automated within the `ctest` framework.

## 6.3. Cross-Compilation on Current Systems

The specific hardware (Cray XC40, KNL) and the build procedures described above are historical and no longer in use.

While cross-compilation is less common on many modern, homogeneous HPC clusters, it is still a necessary technique for advanced architectures, such as systems with different processor types or accelerators (e.g., GPUs).

If you need assistance with cross-compiling for a current HPC system, please contact the facility administrators or The HDF Group (Section 1.3).

# 7. Running Parallel Tests

## 7.1. Test Directory Location

The parallel test suite (`testpar/`) contains tests for Parallel HDF5 and MPI-IO.

By default, tests use the current directory for test files. To specify a different location (e.g., parallel file system):

export HDF5_PARAPREFIX=/scratch/username/hdf5test
ctest . -C Release

> Important: This is critical for performance - avoid using NFS or home directories for parallel testing.

## 7.2. Important Test Notes

### t_mpi

Tests basic MPI-IO features used by Parallel HDF5. Returns non-zero exit code if required features fail.

Exception: Testing files >2GB will print informational messages but not fail, as HDF5 can use alternative file drivers (e.g., family driver) to handle size limits.

### t_cache

Performs many small I/O requests. May run slowly on NFS or other non-parallel file systems. Set `HDF5_PARAPREFIX` to a parallel file system if this test is slow.

### Test Express Level

Controls test thoroughness:

export HDF5_TEST_EXPRESS=3 # Quick tests (default)
export HDF5_TEST_EXPRESS=0 # Exhaustive tests

### Test Timeout

Default is 1200 seconds (20 minutes). Modify in CMake with:

-DDART_TESTING_TIMEOUT=3600

## 7.3. Running Tests on Batch Systems

For SLURM systems:

sbatch -C quad,cache build/bin/batch/ctestS.sl # Serial tests
sbatch -C quad,cache build/bin/batch/ctestP.sl # Parallel tests

For LSF systems:

bsub build/bin/batch/ctestS.lsf # Serial tests
bsub build/bin/batch/ctestP.lsf # Parallel tests

Batch scripts are generated during CMake configuration in the build directory.

## 7.4. Running Specific Test Categories

To run specific test suites:

ctest -R "H5TEST" # Core library tests
ctest -R "MPI_TEST" # Parallel/MPI tests
ctest -R "CPP|FORTRAN" # C++ and Fortran tests
ctest -E "MPI_TEST" # Exclude parallel tests
ctest --parallel 4 # Run 4 tests in parallel

# 8. Known Platform Notes

## 8.1. Linux Systems

For MPICH on Linux, ensure >2GB file support by configuring MPICH with:

-cflags="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64"

This is available on Linux kernels 2.4 and greater.

## 8.2. Cray Systems

- Use `CC=cc`, `FC=ftn`, `CXX=CC` after loading compiler modules
- Unload `craype-hugepages2M` if loaded (Note: This is situational advice and is not a universal rule, but it may be a valid troubleshooting step if you encounter memory-related performance issues or allocation errors.)
- Disable shared libraries if encountering linking issues:
  -DBUILD_SHARED_LIBS=OFF

## 8.3. OpenMPI

OpenMPI 4.0+ disallows oversubscribing by default. Either:

- Set `MPIEXEC_MAX_NUMPROCS` to actual processor count, or
- Add `--oversubscribe` flag: `-DMPIEXEC_PREFLAGS=--oversubscribe`

# References

POSIX Compliant: After a write() to a regular file has successfully returned, any successful read() from each byte position modified by that write() will return the data that was written. A subsequent write() to the same byte will overwrite the file data. If a read() can be proven by any means [e.g., MPI_Barrier()] to occur after a write() of that data, it must reflect that write(), even if calls are made by different processes.

> Lewin, D. (1994). "POSIX Programmer's Guide (pg. 513-4)". O'Reilly & Associates.

# Appendix A. Sample MPI-IO Programs

Here are sample MPI-IO C and Fortran programs to test your MPI compiler and parallel file system before building HDF5. The programs assume they run in a parallel file system (create test files in current directory). For more examples, please refer to the following directories: HDF5Examples/C/H5PAR and HDF5Examples/FORTRAN/H5PAR

## Example Compiling and Running

mpicc Sample_mpio.c -o c.out
mpiexec -np 4 c.out
mpif90 Sample_mpio.f90 -o f.out
mpiexec -np 4 f.out

## Sample_mpio.c

/* Simple MPI-IO program testing if a parallel file can be created.
* Default filename can be specified via first program argument.
* Each process writes something, then reads all data back.
*/
#include <stdio.h>
#include <unistd.h>
#include <mpi.h>
#ifndef MPI_FILE_NULL /*MPIO may be defined in mpi.h already */
# include <mpio.h>
#endif
#define DIMSIZE 10 /* dimension size, avoid powers of 2. */
#define PRINTID printf("Proc %d: ", mpi_rank)
int main(int ac, char **av)
{
char hostname[128];
int mpi_size, mpi_rank;
MPI_File fh;
char *filename = "./mpitest.data";
char mpi_err_str[MPI_MAX_ERROR_STRING];
int mpi_err_strlen;
int mpi_err;
char writedata[DIMSIZE], readdata[DIMSIZE];
char expect_val;
int i, irank;
int nerrors = 0; /* number of errors */
MPI_Offset mpi_off;
MPI_Status mpi_stat;
MPI_Init(&ac, &av);
MPI_Comm_size(MPI_COMM_WORLD, &mpi_size);
MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank);
/* get file name if provided */
if (ac > 1){
filename = *++av;
}
if (mpi_rank==0){
printf("Testing simple MPIO program with %d processes accessing file %s\n",
mpi_size, filename);
printf(" (Filename can be specified via program argument)\n");
}
/* show the hostname so that we can tell where the processes are running */
if (gethostname(hostname, 128) < 0){
PRINTID;
printf("gethostname failed\n");
return 1;
}
PRINTID;
printf("hostname=%s\n", hostname);
if ((mpi_err = MPI_File_open(MPI_COMM_WORLD, filename,
MPI_MODE_RDWR | MPI_MODE_CREATE | MPI_MODE_DELETE_ON_CLOSE,
MPI_INFO_NULL, &fh))
!= MPI_SUCCESS){
MPI_Error_string(mpi_err, mpi_err_str, &mpi_err_strlen);
PRINTID;
printf("MPI_File_open failed (%s)\n", mpi_err_str);
return 1;
}
/* each process writes some data */
for (i=0; i < DIMSIZE; i++)
writedata[i] = mpi_rank*DIMSIZE + i;
mpi_off = mpi_rank*DIMSIZE;
if ((mpi_err = MPI_File_write_at(fh, mpi_off, writedata, DIMSIZE, MPI_BYTE,
&mpi_stat))
!= MPI_SUCCESS){
MPI_Error_string(mpi_err, mpi_err_str, &mpi_err_strlen);
PRINTID;
printf("MPI_File_write_at offset(%ld), bytes (%d), failed (%s)\n",
(long) mpi_off, (int) DIMSIZE, mpi_err_str);
return 1;
};
/* make sure all processes has done writing. */
MPI_Barrier(MPI_COMM_WORLD);
/* each process reads all data and verify. */
for (irank=0; irank < mpi_size; irank++){
mpi_off = irank*DIMSIZE;
if ((mpi_err = MPI_File_read_at(fh, mpi_off, readdata, DIMSIZE, MPI_BYTE,
&mpi_stat))
!= MPI_SUCCESS){
MPI_Error_string(mpi_err, mpi_err_str, &mpi_err_strlen);
PRINTID;
printf("MPI_File_read_at offset(%ld), bytes (%d), failed (%s)\n",
(long) mpi_off, (int) DIMSIZE, mpi_err_str);
return 1;
};
for (i=0; i < DIMSIZE; i++){
expect_val = irank*DIMSIZE + i;
if (readdata[i] != expect_val){
PRINTID;
printf("read data[%d:%d] got %d, expect %d\n", irank, i,
readdata[i], expect_val);
nerrors++;
}
}
}
if (nerrors)
return 1;
MPI_File_close(&fh);
PRINTID;
printf("all tests passed\n");
MPI_Finalize();
return 0;
}

## Sample_mpio.f90

!
! The following example demonstrates how to create and close a parallel
! file using MPI-IO calls.
!
! USE MPI is the proper way to bring in MPI definitions but many
! MPI Fortran compiler supports the pseudo standard of INCLUDE.
! So, HDF5 uses the INCLUDE statement instead.
!
PROGRAM mpioexample
USE mpi
IMPLICIT NONE
CHARACTER(LEN=80), PARAMETER :: filename = "filef.h5" ! File name
INTEGER :: ierror ! Error flag
INTEGER :: fh ! File handle
INTEGER :: amode ! File access mode
call mpi_init(ierror)
amode = mpi_mode_rdwr + mpi_mode_create + mpi_mode_delete_on_close
call mpi_file_open(mpi_comm_world, filename, amode, mpi_info_null, fh, ierror)
print *, "Trying to create ", filename
if ( ierror .eq. mpi_success ) then
print *, "MPI_FILE_OPEN succeeded"
call mpi_file_close(fh, ierror)
else
print *, "MPI_FILE_OPEN failed"
endif
call mpi_finalize(ierror);
END PROGRAM