Please see The HDF Group's new Support Portal for the latest information.
About the Project
The problem.
For NOAA and NASA, the data problem with their Joint Polar Satellite System (JPSS) is how to best handle a large volume data stream from the five different instruments on a satellite. JPSS is a new generation of low Earth orbiting satellites that monitor environmental conditions and provide data for long-range weather and climate forecasts.
Here is NOAA's description of one of the instruments from a 2013 news release:
CrIS is the Cross-track Infrared Sounder. CrIS is the first in a series of advanced operational sounders that provides more accurate, detailed atmospheric temperature and moisture observations for weather and climate applications. This high-spectral resolution infrared instrument measures the three-dimensional structure of atmospheric temperatures, water vapor and trace gases. It provides more than 1,000 infrared spectral channels at an improved horizontal spatial resolution and measure temperature profiles with keen vertical resolution to an accuracy approaching 1 Kelvin (the absolute temperature scale). This information helps significantly improve prediction, including both short-term weather “now casting” and longer-term forecasting. It also provides a vital tool for NOAA to continuously take the pulse of the planet and assist in understanding major seasonal and multi-year shifts.
The solution: HDF5 and custom tools.
We know from other projects such as NASA's Earth Science Data and Information System (ESDIS) project that HDF Group software can be used to store large amounts of climate data. ESDIS has in 13 years (as of September 30, 2013) archived in our software 9.8 petabytes of climate data. Each petabyte is a million gigabytes.
How might data be extracted in a timely manner from such a large dataset? A data granule holds the data from a short period of observation by an instrument. Data is stored by granule in HDF5 files. With a custom tool built by The HDF Group, data granules can be aggregated and extracted. This means that only the data for a certain period of time and for a limited location need be retrieved from the data for study. This is much more efficient than having to open the entire file to see any amount of data. Since the archived data is not changed, data granules can be extracted repeatedly, and the data files themselves only need to be downloaded once.
For more information, click the links on the left.
- - Last modified: 14 October 2016