Help us improve by taking our short survey: https://www.hdfgroup.org/website-survey/

| HDF5 Last Updated on 2025-11-11 The HDF5 Field Guide

- Main Page
- Getting started
- User Guide
- Reference Manual
- Cookbook
- Technical Notes
- RFCs
- Specifications
- Glossary
- Full-Text Search
- About

Loading...
Searching...
No Matches
HDF5 Build System Summary

A comprehensive analysis of the HDF5 CMake-only build system and CI/CD infrastructure for development planning.

# Build System Architecture

## CMake-Only Migration (March 2025)

- Complete transition: Autotools support was completely dropped in HDF5 2.0
- CMake minimum version: 3.26 required
- Out-of-source builds: Enforced - in-source builds are blocked with clear error messages

## Core CMake Structure

### Main Configuration Files

- `CMakeLists.txt` - Root build configuration
- `CMakeBuildOptions.cmake` - Centralized build option definitions
- `CMakeFilters.cmake` - Compression filter support (zlib, szip, libaec)
- `CMakeTests.cmake` - Testing infrastructure configuration
- `CMakeInstallation.cmake` - Installation and packaging setup
- `CMakeVOL.cmake` - Virtual Object Layer connector support
- `CMakePlugins.cmake` - Plugin architecture support
- `java/src/hdf/hdf5lib/pom.xml.in` - Maven POM template for Java artifacts

### Build Options Categories

Library Types:

- `BUILD_STATIC_LIBS=ON` - Static library builds
- `BUILD_SHARED_LIBS=ON` - Shared library builds
- `HDF5_ONLY_SHARED_LIBS=OFF` - Force shared-only builds
- `HDF5_BUILD_STATIC_TOOLS=OFF` - Static vs shared tools

Language Bindings:

- `HDF5_BUILD_CPP_LIB=OFF` - C++ bindings
- `HDF5_BUILD_FORTRAN=OFF` - Fortran bindings
- `HDF5_BUILD_JAVA=OFF` - Java bindings
- `HDF5_ENABLE_MAVEN_DEPLOY=OFF` - Maven repository deployment support
- `HDF5_MAVEN_SNAPSHOT=OFF` - Build Maven snapshot versions with -SNAPSHOT suffix

Core Features:

- `HDF5_ENABLE_PARALLEL=OFF` - MPI parallel I/O support
- `HDF5_ENABLE_THREADSAFE=OFF` - Thread safety (mutually exclusive with parallel)
- `HDF5_ENABLE_CONCURRENCY=OFF` - Multi-threaded concurrency
- `HDF5_BUILD_HL_LIB=ON` - High-level APIs
- `HDF5_BUILD_TOOLS=ON` - Command-line utilities

Compression & Filters:

- `HDF5_ENABLE_ZLIB_SUPPORT=OFF` - DEFLATE compression
- `HDF5_ENABLE_SZIP_SUPPORT=OFF` - SZIP compression
- `HDF5_ENABLE_PLUGIN_SUPPORT=OFF` - Runtime plugin loading
- `HDF5_USE_ZLIB_NG=OFF` - Use zlib-ng instead of zlib

Advanced Features:

- `HDF5_ENABLE_SUBFILING_VFD=OFF` - Parallel subfiling VFD
- `HDF5_ENABLE_MAP_API=OFF` - Map API (experimental)
- `HDF5_ENABLE_HDFS=OFF` - Hadoop HDFS support

# CMake Preset System

## Preset Architecture

- Layered inheritance: Base presets + feature-specific + platform-specific
- Hidden presets: Reusable components (`ci-base`, `ci-Debug`, `ci-Release`, `ci-Maven`, `ci-Maven-Snapshot`, `ci-Maven-Minimal`, `ci-Maven-Minimal-Snapshot`)
- Platform presets: `ci-GNUC`, `ci-Clang`, `ci-MSVC`, `ci-macos`
- Maven presets: Hidden base configurations for Maven deployment support
- Minimal Maven presets: Streamlined configurations for Java artifact generation only
- Build type matrix: Debug, Release (RelWithDebInfo + docs), Maven variants

## Key Preset Patterns

# Standard shared library builds
cmake --workflow --preset ci-StdShar-GNUC --fresh # GCC
cmake --workflow --preset ci-StdShar-Clang --fresh # Clang
cmake --workflow --preset ci-StdShar-MSVC --fresh # MSVC
# Maven-enabled builds (Java artifacts with deployment support)
cmake --workflow --preset ci-StdShar-GNUC-Maven --fresh # Maven release (full build)
cmake --workflow --preset ci-StdShar-GNUC-Maven-Snapshot --fresh # Maven snapshot (full build)
cmake --workflow --preset ci-MinShar-GNUC-Maven --fresh # Maven release (minimal build)
cmake --workflow --preset ci-MinShar-GNUC-Maven-Snapshot --fresh # Maven snapshot (minimal build)
# Multi-platform Maven presets (minimal builds for Java artifacts only)
cmake --workflow --preset ci-MinShar-MSVC-Maven --fresh # Windows Maven
cmake --workflow --preset ci-MinShar-Clang-Maven --fresh # macOS Maven
# Naming convention: ci-[Features]-[Compiler][-Maven[-Snapshot]]
# Features: Std (standard), Min (minimal), StdShar (standard shared), MinShar (minimal shared)
# Maven: Adds Maven deployment support with platform-specific JARs
# Snapshot: Adds -SNAPSHOT suffix for development versions
# Minimal Maven presets: Skip examples, testing, tools, C++, Fortran - Java artifacts only
# Java Examples Maven Integration: Comprehensive testing of Java examples with Maven artifacts across all platforms

## Preset Configuration Strategy

- Binary directory: `${sourceParentDir}/build/${presetName}`
- Install directory: `${sourceParentDir}/install/${presetName}`
- Generator: Ninja (default for most presets)
- External libraries: TGZ/GIT support for zlib, szip, libaec

# Testing Infrastructure

## Test Framework Structure

- CTest integration: Primary test runner
- Express levels: 0 (exhaustive) to 3 (quick) - default level 3
- Timeout system: Base 1200s, with short/long/very-long variants
- Parallel testing: Separate test infrastructure for MPI builds

## Test Categories

- Core tests: `test/` - Library unit tests
- API tests: `test/API/` - Comprehensive API validation
- Parallel tests: `testpar/` - MPI-specific functionality
- Tool tests: `tools/test/` - Command-line utility validation
- VFD tests: Multiple Virtual File Driver implementations
- VOL tests: Virtual Object Layer connector testing
- Language binding tests: C++, Fortran, Java specific tests

## Test Execution Patterns

# Express testing (quick)
export HDF_TEST_EXPRESS=3
ctest -j4
# VFD testing matrix
ctest -R "VFD"
# Parallel testing
ctest -R "MPI\|parallel"
# API comprehensive tests
ctest -R "H5_api_test"

# CI/CD Pipeline Architecture

## GitHub Actions Matrix

- Comprehensive testing across platforms/compilers
- Multi-dimensional matrix:
  - Platforms: Windows (MSVC), Ubuntu (GCC), macOS (Clang)
  - Features: Serial, Parallel, Thread-safe, Various language bindings
  - Build types: Debug, Release, specialized configurations

## Key Workflow Categories

Main CI Workflows:

- `main.yml` - Primary CI across platforms
- `daily-build.yml` - Nightly comprehensive builds
- `ctest.yml` - Cross-platform testing with preset system

Specialized Testing:

- `par-*.yml` - Parallel/MPI testing workflows
- `vfd-*.yml` - Virtual File Driver testing
- `vol_*.yml` - Virtual Object Layer connector testing
- `analysis.yml` - Static analysis integration

Platform-Specific:

- `arm-main.yml` - ARM architecture testing
- `cygwin.yml`, `msys2.yml` - Windows alternative environments
- `intel.yml`, `aocc.yml`, `nvhpc.yml` - Vendor compiler support

Release Infrastructure:

- `release.yml` - Main release workflow with optional Maven deployment
- `release-files.yml` - Automated release packaging
- `tarball.yml` - Source distribution creation
- `maven-deploy.yml` - Maven repository deployment workflow
- `maven-staging.yml` - PR-based Maven testing and validation workflow
- `daily-schedule.yml` - Scheduled builds with AWS integration

## Build Matrix Strategy

- Cross-platform validation: Windows/Linux/macOS for every PR
- Compiler diversity: GCC, Clang, MSVC, Intel, AOCC, NVHPC
- Feature combinations: Systematic testing of feature interactions
- Performance variants: Debug vs Release vs specialized builds

# Packaging & Distribution

## Installation System

- Component-based: Libraries, headers, tools, docs, examples
- CMake integration: Full config package support
- Cross-platform: Windows (NSIS/WiX), macOS (DMG/Framework), Linux (DEB/RPM/TGZ)

## Release Process

- Automated packaging: CPack integration with platform-specific installers
- Version management: Automatic version extraction from source
- SOVERSION handling: Complex library versioning for different components
- External dependencies: Bundled compression libraries option

## Distribution Formats

- Source tarballs: Automated via GitHub Actions
- Binary packages: Platform-specific installers
- Maven repositories: GitHub Packages and Maven Central deployment
- Java artifacts: Platform-specific JARs with classifiers (linux-x86_64, windows-x86_64, macos-x86_64, macos-aarch64)
- Container support: Docker environments for CI
- HPC integration: Specialized configurations for batch systems

# Development Workflow Recommendations

## Build Strategy

1. Use presets for consistency: Leverage the preset system for reproducible builds
2. Feature isolation: Test individual features before combining
3. Express testing: Use level 3 for development, lower levels for validation
4. Parallel development: Separate MPI builds from serial development

## Testing Strategy

1. Incremental testing: Start with `testhdf5` core tests
2. Feature-specific testing: Use test regex patterns for targeted testing
3. Cross-platform validation: Test on primary CI platforms early
4. Performance testing: Include Release builds for performance-critical changes

## CI Integration

1. Workflow triggers: Understand which changes trigger which test suites
2. Matrix optimization: Consider CI time costs for comprehensive testing
3. Failure isolation: Use workflow categories to isolate platform/feature issues
4. External dependency management: Plan for compression library updates

## Maven Integration Workflow

1. Java Build Configuration: Enable Maven deployment with `HDF5_ENABLE_MAVEN_DEPLOY=ON`
2. Version Management: Use `HDF5_MAVEN_SNAPSHOT=ON` for development builds with `-SNAPSHOT` suffix
3. Preset Selection: Choose Maven-enabled presets (`ci-StdShar-GNUC-Maven` or `ci-StdShar-GNUC-Maven-Snapshot`)
4. Platform Artifacts: Automatic generation of platform-specific JARs with classifiers (linux-x86_64, windows-x86_64, macos-x86_64, macos-aarch64)
5. CI Integration: Conditional Maven artifact generation in `ctest.yml` workflow via preset system
6. PR Testing: Automated Maven artifact validation for pull requests via `maven-staging.yml`
7. Validation Framework: Pre-deployment validation using `.github/scripts/validate-maven-artifacts.sh`
8. Repository Selection: Choose between GitHub Packages and Maven Central via workflow inputs
9. Release Integration: Optional Maven deployment in release workflow with user control

# Critical Dependencies

## Build Dependencies

- CMake 3.26+: Required for preset support and modern features
- Ninja: Preferred generator for cross-platform consistency
- Compression libraries: zlib, szip/libaec (optional but commonly used)
- MPI: Required for parallel builds (MPI-3 standard minimum)
- Java 11+: Required for Java bindings and Maven deployment (when `HDF5_BUILD_JAVA=ON`)
- Maven: Optional for local Maven operations and validation

## Platform-Specific Requirements

- Windows: Visual Studio 2022, optional NSIS/WiX for packaging
- macOS: Xcode command line tools, universal binary support
- Linux: GCC/Clang, various package managers for dependencies

## Development Tools Integration

- Static analysis: Clang tools integration available
- Code formatting: clang-format integration
- Coverage: Code coverage support for testing
- Sanitizers: Runtime error detection support
- Maven validation: `.github/scripts/validate-maven-artifacts.sh` for pre-deployment validation

# Future Considerations

## Build System Evolution

- CMake modernization: Potential for newer CMake features as minimum version increases
- Preset expansion: More specialized presets for emerging use cases
- Container integration: Enhanced Docker/container support for development

## Testing Infrastructure

- Test parallelization: Opportunities for faster CI execution
- Cloud testing: Integration with cloud-native testing platforms
- Performance regression: Automated performance monitoring integration

## Platform Support

- Emerging architectures: ARM64, RISC-V support expansion
- New compilers: Integration with emerging compiler technologies
- HPC evolution: Adaptation to evolving supercomputing environments