Format Atlas

Guides

Checks before handing off scientific arrays

Transfer netCDF, HDF5 or Zarr arrays with their meaning, dependencies and reader requirements. Unlike choosing a table format, this requires checking dimensions and missing-value interpretation.

Which format should you choose?

Choose by the role each file plays in your workflow.

  • Document the array's meaning
  • Assemble the distribution unit
  • Compare values at the destination

Document the array's meaning

Record variable names, types, dimension order, coordinates, units and time conventions. Do not equate missing or fill values with zero. For scale_factor and add_offset, distinguish stored values from decoded values.

Assemble the distribution unit

Check the netCDF variant, HDF5 external-link targets and filters, and Zarr version and codecs. Include required Zarr metadata and chunks; distinguish intentional unstored chunks governed by fill_value from transfer omissions.

Compare values at the destination

Share a version whose writes are complete and read sample slices with the recipient's library. Compare shape, axis order, attributes, missing-value masks and decoded values with the source. Record tolerances for rounding or lossy changes.

Formats to consider

netCDF

Good for: Exchanging dimensions, variables and attributes with compatible scientific tools

Less suitable for: Inferring the internal format from the extension alone

HDF5

Good for: Organizing groups and arrays hierarchically

Less suitable for: Omitting external-link targets or required filters

Zarr

Good for: Managing array chunks in a store

Less suitable for: Copying only metadata and calling the transfer complete

Before you export

  • Retain source data and conversion settings. Matching checksums verifies transfer, not correct units or scientific interpretation.
  • Attributes and coordinates may contain confidential information. Do not embed store credentials in shared files; verify access permissions.

Related comparisons

HDF5 vs netCDFHDF5 is a general container for hierarchical data; netCDF provides a scientific data model and APIs. netCDF-4 uses HDF5 as its storage foundation, but an arbitrary HDF5 file is not necessarily readable as netCDF-4.HDF5 vs ZarrHDF5 organizes groups and datasets hierarchically in a general container. Zarr represents chunked multidimensional arrays and metadata as separate keys in a store, with filesystem and object-store use in mind.netCDF vs ZarrnetCDF provides a scientific data model with dimensions, variables, and attributes across several file formats. Zarr specifies chunked multidimensional arrays stored in key-value systems. The better fit depends on tools, storage, and access patterns.NPZ vs HDF5: array bundles and partial readsNPZ bundles named NPY arrays in ZIP. HDF5 organizes groups, datasets, and attributes hierarchically.

Guides

Choose a format for analytics and data exchangeChoose among columnar analytics, schema-based record exchange, and human-readable tables based on your platform and data types.Checks before handing off GIS dataAgree on layers, coordinate reference systems, attributes and recipient software. Beyond choosing a format, verify positions and retained information after conversion.

References and specifications