From: General Support Subject: netCDF FAQ Frequently Asked Questions About netCDF This article contains answers to some of the most frequently asked netCDF questions on the netcdfgroup mailing list and in the email sent to support@unidata.ucar.edu. The list is archived in pub/netcdf/FAQ on unidata.ucar.edu, and is currently maintained by Russ Rew (russ@unidata.ucar.edu). List of Questions: 1: What Is netCDF? 2: How do I get the netCDF software package? 3: What does netCDF run on? 4: When will the next version be available? What changes are included? 5: What is the connection between netCDF and CDF? 6: What is the connection between netCDF and HDF? 7: Is netCDF available for the Apple Macintosh? 8: What are some references to netCDF? 9: What are the files in pub/netcdf/ on unidata.ucar.edu? 10: Why do I get XDR error messages when trying to write data? 11: What is the best way to represent [some particular data] using netCDF? 12: Are there plans to add facilities for data compression to netCDF? 13: What utilities are available for netCDF? 14: What is the status of implementing additional netCDF operators? 15: Is there a mailing list for netCDF discussions and questions? 16: How do I make a bug report? 17: Is there any WAIS or gopher access to netCDF information? 18: How widely is netCDF used? 1: What Is netCDF? A: NetCDF (network Common Data Form) is an interface for scientific data access and a freely-distributed software library that provides an implementation of the interface. It was developed by Glenn Davis, Russ Rew, and Steve Emmerson at the Unidata Program Center in Boulder, Colorado. The netCDF library also defines a machine-independent format for representing scientific data. Together, the interface, library, and format support the creation, access, and sharing of scientific data. netCDF data is: - Self-Describing. A netCDF file includes information about the data it contains. - Network-transparent. A netCDF file is represented in a form that can be accessed by computers with different ways of storing integers, characters, and floating-point numbers. - Direct-access. A small subset of a large dataset may be accessed efficiently, without first reading through all the preceding data. - Appendable. Data can be appended to a netCDF dataset along one dimension without copying the dataset or redefining its structure. The structure of a netCDF dataset can be changed, though this sometimes causes the dataset to be copied. - Sharable. One writer and multiple readers may simultaneously access the same netCDF file. 2: How do I get the netCDF software package? A: Via anonymous FTP from host: unidata.ucar.edu [128.117.140.3] file: pub/netcdf/netcdf.tar.Z Make sure the file is transmitted in BINARY mode. This is version 2.3.2, last updated in June 1993. 3: What does netCDF run on? A: Version 2.3.2 of netCDF has been tested on the following platforms: CRAY Y-MP UNICOS 6.1.6 DEC Alpha OSF/1 1.2 DEC VAX VMS 5.5-2 DEC VAX Ultrix 4.3 DECstation 3000 Ultrix 4.3 HP-9000/7xx HPUX 9.0 IBM PS/2 MSDOS 5.0 IBM PS/2 OS/2 1.2 IBM RS-6000 AIX 3.2 NeXT NeXTOS 3.0 SGI Iris IRIX 4.0.5F SPARCstation Solaris 2.1 SPARCstation SunOS 4.1.3 The experience of outside developers indicates that netCDF is relatively easy to port to any system that has a C compiler and an XDR library (eXternal Data Representation, used for NFS and usually supplied by vendors). 4: What has changed since the netCDF 2.2 release in December 1991? A: NetCDF 2.3, made available in April 1993, contains bug fixes, portability enhancements, performance enhancements, and a few new capabilities. Some new optimizations for the library result in significant speedups for accessing cross-sections involving non-contiguous data. New capabilities include some additional interfaces that provide a more general form of hyperslab access. This supports sub-sampling along specified dimensions and a mapping between the points of the hyperslab and the memory locations of the corresponding values. In a generalized hyperslab, an index mapping vector is used to define the mapping between points in the generalized hyperslab and the memory locations of the corresponding values, so data values that are written or read need no longer be contiguous in memory. There are also some new interfaces that can be used to write, read, and inquire about records, where a record may contain multiple variables of different types and shapes. Where before you had to access a record's worth of data using multiple calls, now you will be able to use a single call. The ncdump utility supports several new command-line options including the ability to specify for which variables data values will be output, to provide brief annotations in the form of CDL comments to identify data values for large multidimensional variables, or to provide full annotations in the form of trailing CDL comments for every data value. The current release also includes a prototype implementation of a C++ interface for the netCDF data access library. It provides all the functionality of the previous C interface, improves type safety by eliminating use of void* pointers, and is somewhat simpler to use than the C interface. With the C++ interface, no IDs are needed for netCDF components, there is no need to specify types when creating attributes, and less indirection is required for dealing with dimensions. However, since this is a prototype interface and implementation, it may be changed before a supported version is released. 5: What is the connection between netCDF and CDF? A: CDF was developed at the NASA Space Science Data Center at Goddard, and is freely available. It was originally a VMS FORTRAN interface for scientific data access. Unidata reimplemented the library from scratch to use XDR for a machine-independent representation, designed the CDL text representation for netCDF data, wrote a User's Guide and made other additions including aggregate data access, single-file implementation, named dimensions, and variable-specific attributes. NetCDF and CDF have evolved independently. CDF now supports many of the same features as netCDF (aggregate data access, XDR representation, single-file representation, variable-specific attributes), but some differences remain (netCDF doesn't support native-mode representation, CDF doesn't support named dimensions). There is no compatibility between data in CDF and netCDF form, and as yet no translation software exists to convert data in one form to data in the other form. 6: What is the connection between netCDF and HDF? A: The National Center for Supercomputing Applications (NCSA) developed the HDF software and makes it freely available. HDF is an extensible data format for self-describing files that was developed independently of netCDF. Applications and utilities based on HDF are available that support raster-image manipulation and display and browsing through multidimensional scientific data. The HDF software includes a package of routines for accessing each HDF data type, as well as a lower-level interface for building packages to support new types. HDF supports both C and Fortran interfaces, and it has been successfully ported to a wide variety of machine architectures and operating systems. HDF emphasizes a single common format for data, on which many interfaces can be built. NCSA has implemented software that provides a netCDF interface to HDF. With this software, it is possible to use the netCDF calling interface to place data into an HDF file. The netCDF calling interface has not changed and netCDF files stored in XDR format are readable, so existing programs and data will still be usable (although programs will need to be relinked to the new library). There is currently no support for the mixing of HDF and netCDF structures. For example, a raster image can exist in the same file as a netCDF object, but you have to use the Raster Image interface to read the image and the netCDF interface to read the netCDF object. The other HDF interfaces are currently being modified to allow multi-file access, closer integration with the netCDF interface will probably be delayed until the end of that project. Eventually, it may be possible to integrate netCDF objects with the rest of the HDF tool suite. Such an integration will then allow tools written for netCDF and tools written for HDF to both interact intelligently with the new data files. 7: Is netCDF available for the Apple Macintosh? A: Unidata doesn't test or maintain a version of netCDF for Macintoshes, but see the files in the directory pub/netcdf/mac/ on unidata.ucar.edu for notes and Macintosh MPW makefiles for porting netCDF to an Apple Macintosh. These were contributed by Chuck Denham, U.S. Geological Survey. 8: What are some references to netCDF? A: Hard-copies of some of these are available from the Unidata Program Center, P.O. Box 3000, Boulder, CO 80307-3000: Rew, R. K., G. P. Davis, and S. Emmerson, NetCDF User's Guide, An Interface for Data Access, Version 2.3, April 1993. (Available from Unidata or by anonymous FTP from unidata.ucar.edu in the file pub/netcdf/guide.ps.Z) Rew, R. K. and G. P. Davis, "NetCDF: An Interface for Scientific Data Access," IEEE Computer Graphics and Applications, Vol. 10, No. 4, pp. 76-82, July 1990. Rew, R. K. and G. P. Davis, "The Unidata netCDF: Software for Scientific Data Access," Sixth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Anaheim, California, American Meteorology Society, February 1990. Jenter, H. L. and R. P. Signell, 1992. "NetCDF: A Freely-Available Software-Solution to Data-Access Problems for Numerical Modelers". Proceedings of the American Society of Civil Engineers Conference on Estuarine and Coastal Modeling. Tampa, Florida. (Also available via anonymous FTP from sparky.er.usgs.gov in the file pub/netcdf.asce.ps) Fulker, D. W., "Unidata Strawman for Storing Earth-Referencing Data," Seventh International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, New Orleans, La., American Meteorology Society, January 1991. Brown, S. A, M. Folk, G. Goucher, and R. Rew, "Software for Portable Scientific Data Management," Computers in Physics, American Institute of Physics, Vol. 7, No. 3, May/June 1993, pp. 304-308. 9: What are the files in pub/netcdf/ on unidata.ucar.edu? A: File Purpose README general information about netCDF FAQ this file of frequently asked questions utilities.txt a list of software packages currently available or under development for manipulating and displaying netCDF data guide.ps.Z a compressed PostScript file of the NetCDF User's Guide. This is included in the netcdf.tar.Z distribution, so you don't need both. ncprogs.ps a draft PostScript document describing an initial set of netCDF operator and utility programs under development ncprogs.txt an ASCII version of ncprogs.ps conventions.txt a draft document of some proposed netCDF conventions cdl/ a directory containing some examples of CDL files (an ASCII representation for netCDF files). msdos/ a directory containing executables and binaries for netCDF under MSDOS 5.0. These can also be built from the sources in netcdf.tar.Z, if you have the necessary Microsoft compilers. mac/ a directory containing notes and Macintosh MPW makefiles for porting netCDF to an Apple Macintosh. These were contributed by Chuck Denham, U.S. Geological Survey. 10: Why do I get XDR error messages when trying to write data? A: Probably due to a write error, perhaps because of exceeding disk quotas, a full device, or permission problems. The netCDF library is usually built on a vendor-supplied XDR library layer. Sometimes errors occur in the XDR layer rather than the netCDF layer, and in this case the error messages from the XDR layer can be cryptic. For example, if the user is writing data and exceeds a disk space quota, this might be detected first in the XDR layer. Although we try to catch and elaborate on errors that occur in the XDR layer, sometimes there is not enough information passed up to calling routines from the XDR layer to make construction of a meaningful error message possible. 11: What is the best way to represent [some particular data] using netCDF? A: There are many ways to represent the same information in any general-purpose data model. Choices left up to the user in the case of netCDF include which information to represent as variables or as variable attributes; what names to choose for variables, dimensions, and attributes; what order to use for the dimensions of multidimensional variables; what variables to include in the same netCDF file; and how to use variable attributes to capture the structure and meaning of data. We provide some guidelines in the NetCDF User's Guide (e.g Section 2.3.2, ``Differences between Attributes and Variables''), but we've found that a little experience helps. Occasionally we have decided it was useful to change the structure of netCDF files after experience with how the data is used. 12: Are there plans to add facilities for data compression to netCDF? A: We have no plans to add data compression to netCDF (although we do hope to eventually add a form of transparent data packing on write and unpacking on read whenever the reserved attributes "_Nbits", "_Scale", and "_Offset" are defined). Hyperslab access and direct access to individual array values conflict with most simple compression schemes. With netCDF, the elements of an array variable can be filled in any order or as cross-sections in any direction. NetCDF permits writing elements in one order and reading them later in different orders. Some compression methods require that all the data to be compressed are known before starting the compression. Techniques like run-length encoding or anything that depends on exploiting similarities in nearby values can't be used if nearby values aren't all known at the time some of the data are to be written. An alternative that can be implemented above the netCDF library is to adopt a convention for compressed data that uses a "compression" attribute to encode the method of compression, e.g. x:compression = "rle" ; for run-length encoding of the data in a variable x. Then when you write the data, compress them into a bland array of bytes and write all the bytes. Note that it would be difficult to define the size of such a variable in advance, since its compressed size depends on its values. You would also have to give up on hyperslab access for such variables, but instead read the compressed array in all at once and uncompress it before using it. 13: What utilities are available for netCDF? A: The only utilities available in the current netCDF distribution from Unidata are ncdump and ncgen, for converting netCDF files to an ASCII human-readable form and for converting from the ASCII human-readable form back to a binary netCDF file or a C or FORTRAN program for generating the netCDF file. In addition, the first release of the netcdf operators package, containing three general-purpose netCDF operators, is available from unidata.ucar.edu in the file ncopers.tar.Z. See the description of the file utilities.txt in the answer to the question above on important files in /pub/netcdf on unidata.ucar.edu for more information. 14: What is the status of implementing additional netCDF operators? A: Work on additional netCDF operators has been suspended until we can get more programmer resources, due to a higher priority recently placed on development of an event-driven network data distribution system. Volunteers to implement one or more of the netCDF operators are hereby solicited. We will coordinate community efforts to avoid duplication of effort, so before you volunteer to work on one of the planned netCDF operators, please contact support@unidata.ucar.edu to find out if someone else is already working on it. 15: Is there a mailing list for netCDF discussions and questions? A: Yes - there are two. The mailing list netcdfgroup@unidata.ucar.edu has over two hundred readers and gateways; you can subscribe or unsubscribe to the mailing list by sending mail to netcdfgroup-adm@unidata.ucar.edu If you would prefer to get only a single daily digest of the postings to the netcdfgroup mailing list, subscribe instead to the netcdfdigest mailing list by sending a request to netcdfdigest-adm@unidata.ucar.edu All the postings to the netcdfgroup mailing list are archived in the file mail-archives/netcdfgroup available via anonymous FTP from unidata.ucar.edu. 16: How do I make a bug report? A: If you find a bug, send a description to support@unidata.ucar.edu This is also a better address to use for questions or discussions about netCDF that you think are not appropriate for the entire netcdfgroup mailing list. 17: Is there any WAIS or gopher access to netCDF information? A: Yes, `netcdf-group.src' is a WAIS source that provides a full-text search of the netcdfgroup mailing list archive and is registered with think.com for general use. Another WAIS source, `unidata-support-netcdf.src', provides a full-text search of the support questions and answers about netCDF provided by Unidata support staff. Both of these WAIS servers and access to other information about Unidata can be accessed most easily through the Unidata gopher server at gopher.unidata.ucar.edu. 18: How widely is netCDF used? A: The netcdfgroup and netcdfdigest mailing lists have 290 addresses (some of which are aliases to more addresses) in fifteen countries. Several groups have adopted netCDF as a standard way to represent some forms of scientific data. The global ocean modeling effort at Los Alamos National Laboratory (LANL), as part of the DOE CHAMMP effort and one of the DOE Grand Challenges, has selected netCDF as the archival format for its computational data. An effort to bring netCDF up on the parallel disks on the CM-5 is planned to begin shortly. The National Center for Supercomputing Applications has incorporated the netCDF 2.3 interfaces into the latest release of their HDF software, permitting HDF tools that use this interface to be applied to netCDF datasets that are either XDR- or HDF-encoded. The Computer Planning Committee of NOAA's Pacific Marine Environmental Laboratory (PMEL) endorsed netCDF as the preferred data format for the Laboratory in early 1993. PMEL has developed the EPIC system for management, display and analysis of oceanographic time series and hydrographic data. EPIC utilizes netCDF as its primary data format. EPIC toolkits for netCDF include a data file I/O library, which is layered on top of the netCDF library, a netCDF calculator (nccalc) linked with a scientific graphics package (PPLUS), and a suite of display and analysis programs for oceanographic data. Lamont-Doherty Earth Observatory of Columbia University has converted all Marine Geophysics data (gravity, magnetics and bathymetry) acquired in the past 40 years by scientists at L-DEO as well as at other institutions to netCDF. A package of programs to access, maintain and display those files has also been completed. The software package is available via FTP (pub/cdf/MGG_CDF.tar.Z on lamont.ldeo.columbia.edu) The Generic Mapping Tools (GMT), a Unix-based set of tools for data manipulation and display using PostScript, make use of netCDF for storage of 2-D gridded data sets. GMT is used worldwide by about 3000 scientists, according to the developers. The Models-3 Project, being cooperatively pursued by the EPA's Atmospheric Research Laboratory and by North Carolina Supercomputing Center, is using an environmental-modeling-specific applications programming interface on top of UCAR's netCDF as the means for persistent storage of both observational and model-output data, as well as for storing sets of data-file-structure definitions and (prototype, so far) data-dependency graphs for scheduling the sets of programs which constitute their environmental models. A group in the Atmospheric Chemistry Division at NCAR that deals with UARS (Upper Atmospheric Research Satellite) data uses netCDF for their binary data format. Output from NCAR's High Altitude Observatory Division Thermospheric General Circulation Model (TGCM) and related models are converted to netCDF files for post-model visualization and diagnostic codes. NCAR's Research Aviation Facility will use netCDF to distribute all aircraft data, if performance tests on writing high rate data are satisfactory. NCAR's Research Data Program uses netCDF as the primary file format for data archived and used in the "zeb" display and analysis system. Quick look data from various projects is distributed by RDP in netCDF. NetCDF is also the file format used by the (zeb-based) Integrated Sounding System. PolyPaint is an interactive 3D visualization package from NCAR. PolyPaint V 3.4 uses netCDF for data sets, and storage of geometry information. PolyPaint+ alpha-version, being developed jointly by NCAR/MMM and LASP with funding from AISRP/NASA uses netCDF and DataHub from JPL. DataHub from JPL, with funding from AISRP/NASA identifies and converts between a variety of data format, CDF, HDF, MMM/netCDF, FITS, PDS, ... Work is under way to support conversion from a variety of NASA data formats to netCDF used by the PolyPaint+ visualization system from NCAR's MMM division. (JPL Contact for DataHub: Tom Handley, thandley@spacemouse.jpl.nasa.gov) A major component of the US Climate and Global Change program is the TOGA-TAO Array in the tropical Pacific, which proposes to maintain approximately 70 moored ATLAS wind and thermistor chain and current meter buoys, spanning the Pacific ocean from 95W to 137E in the equatorial wave guide. The TAO Project Office, at PMEL, has developed distribution and display software for the real-time data from the TAO buoys, in a point-and-click UNIX workstation environment. This software is distributed nationally and internationally. All data is stored and distributed in netCDF format. All graphics displays and animations are produced with the EPIC tools for working with netCDF data files. The US Geological Survey's Branch of Atlantic Marine Geology uses netCDF to access a variety of scientific data sets, including output from circulation and sediment tranport models, sonar imagery, digital elevation models, and environmental sensor data. It is currently investigating the utility of using netCDF for data distribution on CDROM. At the Woods Hole Oceanographic Institution, netCDF is used in several areas. Ships in the UNOLS fleet are recording measurements from the IMET systems in netCDF form. These data include wind, barometer, humidity, air and sea temperature, precipitation, short wave radiation, and GPS navigation. Data sets from these systems taken during the WOCE experiments in the Pacific have been archived recently at NCAR. Also, measurements from a diverse set of instruments deployed on buoys for the Subduction, TOGA/COARE, and several other experiments are translated into netCDF form for processing and archival. Reports that describe the software systems used for these processing activities are available from WHOI. Scripps Institution of Oceanography (SIO) and the University Corporation for Atmospheric Research (UCAR) conducted a multi-platform climate field project during March of this year based in Nadi, Fiji. All data from this experiment will be archived using Unidata's netCDF before release to the scientific community. The Oregon State University Oceanographic research vessel WECOMA uses the netCDF library for primary scientific data logging. This includes navigational, meteorological, and other miscellaneous data. This logging is part of a client/server system for data distribution, display, and management known as XMIDAS. More details of the system are available from oce.orst.edu in the directory /pub/Wecoma, including a sample netCDF created data file from a Wecoma cruise. NOAA's Forecast System Laboratories have adopted netCDF as a data access interface for some of their systems and applications. A researcher at CSIRO Division of Oceanography in Australia is using netCDF for input and output files for a three dimensional coastal and estuarine hydrodynamic model. A general purpose finite element data model (referred to as EXODUS II) utilizing netCDF has been developed at Sandia National Laboratories. It consists of a C and FORTRAN application programming interface (API) to read and write geometry and results (including time varying data) for finite element analyses. For more information, contact Larry Schoof (laschoo@somnet.sandia.gov). NetCDF is the defacto standard for Analytical Data Interchange and Storage Standards (ADISS) for chromatography and mass spectrometry. NetCDF has been adopted by the Analytical Instrument Association (AIA), which includes all major analytical laboratory instrument vendors worldwide. ADISS supplies the Analytical Information Model on which the information content of these standards is based. The main uses of ADISS/netCDF by end users have been data communications and archival. The AIA Standards Committee is currently developing an ADISS Data Dictionary for infrared spectroscopy. The AIA expects to use netCDF for implementing data interchange. Another committee is developing an ADISS Data Dictionary for nuclear magnetic resonance and magnetic resonance imaging spectroscopy. After sufficient performance testing is completed, netCDF may be used for native storage for NMR, as well as data communications. Testing is expected to be completed at the end of summer 1993. Several commercial analysis and data visualization packages have been adapted to access netCDF data, including Spyglass Dicer, IBM Visualization Data Explorer, Research Systems' IDL, and Wavefront's Data Visualizer. SuperComputer Systems Engineering and Services Company (SSESCO) decided that for all future releases of their scientific visualization software, savi3D, netCDF will be the `native' format. SSESCO has implemented a meta-file layer on top of the netCDF library, called MRAF. It handles multiple netCDF files as well as automatic max-min calculations, time-varying gridded, particle, and discrete data, logical groupings for discrete data, and an overall simplified and flexible interface for storing scientific data. MRAF is being used by the DOE at the Hanford-Meterological Site for observational data and will be used for their weather-modelling. SSESCO is also working with meterologists around the US and France who will be moving to netCDF before the end of the year (mostly users of savi3D). The VISAGE visualization system, developed at the General Electric Corporate Research and Development, (Schroeder, WJ et al, "VISAGE: An Object-Oriented Scientific Visualization System", Proceedings of Visualization `92 Conference) uses netCDF as the preferred format. VISAGE is used at GE Corporate R&D, GE Aircraft Engine, GE Canada, GE Power Generation, as well as ETH Zurich, Switzerland, MQS In Chieti, Italy, and Rensselaer Polytechnic Institute in Troy, New York. GE has another application called "Decimate" that does polygon reduction/decimation (Schroeder,WJ et al, "Decimation of Triangle Meshes", Proceedings of SIGGRAPH `92). Again, this application uses netCDF as a preferred format. Decimate is currently licensed to Cyberware, Inc., makers of 3D laser digitizing hardware. Decimate is currently bundled with the scanners, and will soon be available as a commercial product.