Help us improve by taking our short survey: https://www.hdfgroup.org/website-survey/
HDF5 Last Updated on 2025-11-26
The HDF5 Field Guide
Loading...
Searching...
No Matches
HDF5 Build System Summary

A comprehensive analysis of the HDF5 CMake-only build system and CI/CD infrastructure for development planning.

Build System Architecture

CMake-Only Migration (March 2025)

  • Complete transition: Autotools support was completely dropped in HDF5 2.0
  • CMake minimum version: 3.26 required
  • Out-of-source builds: Enforced - in-source builds are blocked with clear error messages

Core CMake Structure

Main Configuration Files

  • CMakeLists.txt - Root build configuration
  • CMakeBuildOptions.cmake - Centralized build option definitions
  • CMakeFilters.cmake - Compression filter support (zlib, szip, libaec)
  • CMakeTests.cmake - Testing infrastructure configuration
  • CMakeInstallation.cmake - Installation and packaging setup
  • CMakeVOL.cmake - Virtual Object Layer connector support
  • CMakePlugins.cmake - Plugin architecture support
  • java/src/hdf/hdf5lib/pom.xml.in - Maven POM template for Java artifacts

Build Options Categories

Library Types:

  • BUILD_STATIC_LIBS=ON - Static library builds
  • BUILD_SHARED_LIBS=ON - Shared library builds
  • HDF5_ONLY_SHARED_LIBS=OFF - Force shared-only builds
  • HDF5_BUILD_STATIC_TOOLS=OFF - Static vs shared tools

Language Bindings:

  • HDF5_BUILD_CPP_LIB=OFF - C++ bindings
  • HDF5_BUILD_FORTRAN=OFF - Fortran bindings
  • HDF5_BUILD_JAVA=OFF - Java bindings
  • HDF5_ENABLE_MAVEN_DEPLOY=OFF - Maven repository deployment support
  • HDF5_MAVEN_SNAPSHOT=OFF - Build Maven snapshot versions with -SNAPSHOT suffix

Core Features:

  • HDF5_ENABLE_PARALLEL=OFF - MPI parallel I/O support
  • HDF5_ENABLE_THREADSAFE=OFF - Thread safety (mutually exclusive with parallel)
  • HDF5_ENABLE_CONCURRENCY=OFF - Multi-threaded concurrency
  • HDF5_BUILD_HL_LIB=ON - High-level APIs
  • HDF5_BUILD_TOOLS=ON - Command-line utilities

Compression & Filters:

  • HDF5_ENABLE_ZLIB_SUPPORT=OFF - DEFLATE compression
  • HDF5_ENABLE_SZIP_SUPPORT=OFF - SZIP compression
  • HDF5_ENABLE_PLUGIN_SUPPORT=OFF - Runtime plugin loading
  • HDF5_USE_ZLIB_NG=OFF - Use zlib-ng instead of zlib

Advanced Features:

  • HDF5_ENABLE_SUBFILING_VFD=OFF - Parallel subfiling VFD
  • HDF5_ENABLE_MAP_API=OFF - Map API (experimental)
  • HDF5_ENABLE_HDFS=OFF - Hadoop HDFS support

CMake Preset System

Preset Architecture

  • Layered inheritance: Base presets + feature-specific + platform-specific
  • Hidden presets: Reusable components (ci-base, ci-Debug, ci-Release, ci-Maven, ci-Maven-Snapshot, ci-Maven-Minimal, ci-Maven-Minimal-Snapshot)
  • Platform presets: ci-GNUC, ci-Clang, ci-MSVC, ci-macos
  • Maven presets: Hidden base configurations for Maven deployment support
  • Minimal Maven presets: Streamlined configurations for Java artifact generation only
  • Build type matrix: Debug, Release (RelWithDebInfo + docs), Maven variants

Key Preset Patterns

# Standard shared library builds
cmake --workflow --preset ci-StdShar-GNUC --fresh # GCC
cmake --workflow --preset ci-StdShar-Clang --fresh # Clang
cmake --workflow --preset ci-StdShar-MSVC --fresh # MSVC
# Maven-enabled builds (Java artifacts with deployment support)
cmake --workflow --preset ci-StdShar-GNUC-Maven --fresh # Maven release (full build)
cmake --workflow --preset ci-StdShar-GNUC-Maven-Snapshot --fresh # Maven snapshot (full build)
cmake --workflow --preset ci-MinShar-GNUC-Maven --fresh # Maven release (minimal build)
cmake --workflow --preset ci-MinShar-GNUC-Maven-Snapshot --fresh # Maven snapshot (minimal build)
# Multi-platform Maven presets (minimal builds for Java artifacts only)
cmake --workflow --preset ci-MinShar-MSVC-Maven --fresh # Windows Maven
cmake --workflow --preset ci-MinShar-Clang-Maven --fresh # macOS Maven
# Naming convention: ci-[Features]-[Compiler][-Maven[-Snapshot]]
# Features: Std (standard), Min (minimal), StdShar (standard shared), MinShar (minimal shared)
# Maven: Adds Maven deployment support with platform-specific JARs
# Snapshot: Adds -SNAPSHOT suffix for development versions
# Minimal Maven presets: Skip examples, testing, tools, C++, Fortran - Java artifacts only
# Java Examples Maven Integration: Comprehensive testing of Java examples with Maven artifacts across all platforms

Preset Configuration Strategy

  • Binary directory: ${sourceParentDir}/build/${presetName}
  • Install directory: ${sourceParentDir}/install/${presetName}
  • Generator: Ninja (default for most presets)
  • External libraries: TGZ/GIT support for zlib, szip, libaec

Testing Infrastructure

Test Framework Structure

  • CTest integration: Primary test runner
  • Express levels: 0 (exhaustive) to 3 (quick) - default level 3
  • Timeout system: Base 1200s, with short/long/very-long variants
  • Parallel testing: Separate test infrastructure for MPI builds

Test Categories

  • Core tests: test/ - Library unit tests
  • API tests: test/API/ - Comprehensive API validation
  • Parallel tests: testpar/ - MPI-specific functionality
  • Tool tests: tools/test/ - Command-line utility validation
  • VFD tests: Multiple Virtual File Driver implementations
  • VOL tests: Virtual Object Layer connector testing
  • Language binding tests: C++, Fortran, Java specific tests

Test Execution Patterns

# Express testing (quick)
export HDF_TEST_EXPRESS=3
ctest -j4
# VFD testing matrix
ctest -R "VFD"
# Parallel testing
ctest -R "MPI\|parallel"
# API comprehensive tests
ctest -R "H5_api_test"

CI/CD Pipeline Architecture

GitHub Actions Matrix

  • Comprehensive testing across platforms/compilers
  • Multi-dimensional matrix:
    • Platforms: Windows (MSVC), Ubuntu (GCC), macOS (Clang)
    • Features: Serial, Parallel, Thread-safe, Various language bindings
    • Build types: Debug, Release, specialized configurations

Key Workflow Categories

Main CI Workflows:

  • main.yml - Primary CI across platforms
  • daily-build.yml - Nightly comprehensive builds
  • ctest.yml - Cross-platform testing with preset system

Specialized Testing:

  • par-*.yml - Parallel/MPI testing workflows
  • vfd-*.yml - Virtual File Driver testing
  • vol_*.yml - Virtual Object Layer connector testing
  • analysis.yml - Static analysis integration

Platform-Specific:

  • arm-main.yml - ARM architecture testing
  • cygwin.yml, msys2.yml - Windows alternative environments
  • intel.yml, aocc.yml, nvhpc.yml - Vendor compiler support

Release Infrastructure:

  • release.yml - Main release workflow with optional Maven deployment
  • release-files.yml - Automated release packaging
  • tarball.yml - Source distribution creation
  • maven-deploy.yml - Maven repository deployment workflow
  • maven-staging.yml - PR-based Maven testing and validation workflow
  • daily-schedule.yml - Scheduled builds with AWS integration

Build Matrix Strategy

  • Cross-platform validation: Windows/Linux/macOS for every PR
  • Compiler diversity: GCC, Clang, MSVC, Intel, AOCC, NVHPC
  • Feature combinations: Systematic testing of feature interactions
  • Performance variants: Debug vs Release vs specialized builds

Packaging & Distribution

Installation System

  • Component-based: Libraries, headers, tools, docs, examples
  • CMake integration: Full config package support
  • Cross-platform: Windows (NSIS/WiX), macOS (DMG/Framework), Linux (DEB/RPM/TGZ)

Release Process

  • Automated packaging: CPack integration with platform-specific installers
  • Version management: Automatic version extraction from source
  • SOVERSION handling: Complex library versioning for different components
  • External dependencies: Bundled compression libraries option

Distribution Formats

  • Source tarballs: Automated via GitHub Actions
  • Binary packages: Platform-specific installers
  • Maven repositories: GitHub Packages and Maven Central deployment
  • Java artifacts: Platform-specific JARs with classifiers (linux-x86_64, windows-x86_64, macos-x86_64, macos-aarch64)
  • Container support: Docker environments for CI
  • HPC integration: Specialized configurations for batch systems

Development Workflow Recommendations

Build Strategy

  1. Use presets for consistency: Leverage the preset system for reproducible builds
  2. Feature isolation: Test individual features before combining
  3. Express testing: Use level 3 for development, lower levels for validation
  4. Parallel development: Separate MPI builds from serial development

Testing Strategy

  1. Incremental testing: Start with testhdf5 core tests
  2. Feature-specific testing: Use test regex patterns for targeted testing
  3. Cross-platform validation: Test on primary CI platforms early
  4. Performance testing: Include Release builds for performance-critical changes

CI Integration

  1. Workflow triggers: Understand which changes trigger which test suites
  2. Matrix optimization: Consider CI time costs for comprehensive testing
  3. Failure isolation: Use workflow categories to isolate platform/feature issues
  4. External dependency management: Plan for compression library updates

Maven Integration Workflow

  1. Java Build Configuration: Enable Maven deployment with HDF5_ENABLE_MAVEN_DEPLOY=ON
  2. Version Management: Use HDF5_MAVEN_SNAPSHOT=ON for development builds with -SNAPSHOT suffix
  3. Preset Selection: Choose Maven-enabled presets (ci-StdShar-GNUC-Maven or ci-StdShar-GNUC-Maven-Snapshot)
  4. Platform Artifacts: Automatic generation of platform-specific JARs with classifiers (linux-x86_64, windows-x86_64, macos-x86_64, macos-aarch64)
  5. CI Integration: Conditional Maven artifact generation in ctest.yml workflow via preset system
  6. PR Testing: Automated Maven artifact validation for pull requests via maven-staging.yml
  7. Validation Framework: Pre-deployment validation using .github/scripts/validate-maven-artifacts.sh
  8. Repository Selection: Choose between GitHub Packages and Maven Central via workflow inputs
  9. Release Integration: Optional Maven deployment in release workflow with user control

Critical Dependencies

Build Dependencies

  • CMake 3.26+: Required for preset support and modern features
  • Ninja: Preferred generator for cross-platform consistency
  • Compression libraries: zlib, szip/libaec (optional but commonly used)
  • MPI: Required for parallel builds (MPI-3 standard minimum)
  • Java 11+: Required for Java bindings and Maven deployment (when HDF5_BUILD_JAVA=ON)
  • Maven: Optional for local Maven operations and validation

Platform-Specific Requirements

  • Windows: Visual Studio 2022, optional NSIS/WiX for packaging
  • macOS: Xcode command line tools, universal binary support
  • Linux: GCC/Clang, various package managers for dependencies

Development Tools Integration

  • Static analysis: Clang tools integration available
  • Code formatting: clang-format integration
  • Coverage: Code coverage support for testing
  • Sanitizers: Runtime error detection support
  • Maven validation: .github/scripts/validate-maven-artifacts.sh for pre-deployment validation

Future Considerations

Build System Evolution

  • CMake modernization: Potential for newer CMake features as minimum version increases
  • Preset expansion: More specialized presets for emerging use cases
  • Container integration: Enhanced Docker/container support for development

Testing Infrastructure

  • Test parallelization: Opportunities for faster CI execution
  • Cloud testing: Integration with cloud-native testing platforms
  • Performance regression: Automated performance monitoring integration

Platform Support

  • Emerging architectures: ARM64, RISC-V support expansion
  • New compilers: Integration with emerging compiler technologies
  • HPC evolution: Adaptation to evolving supercomputing environments