Compare
HDF5 vs Zarr
HDF5 organizes groups and datasets hierarchically in a general container. Zarr represents chunked multidimensional arrays and metadata as separate keys in a store, with filesystem and object-store use in mind.
Which format should you choose?
Prefer HDF5 for existing files, heterogeneous hierarchical data, and HDF5-aware tools. Consider Zarr for independent array-chunk access in object storage, and test the Zarr version and codecs in your environment.
Good fit for HDF5
Storing large scientific and technical data hierarchically · Keeping multidimensional arrays and related metadata in one container
- Organizing complex data with groups, datasets, and attributes
- Scientific workflows needing partial I/O or dataset-level compression
Good fit for Zarr
Storing large multidimensional arrays in chunks · Working with scientific data in cloud object storage
- Reading selected regions of large arrays
- Sharing sharded datasets between compatible tools
Basic information
| Field | HDF5 | Zarr |
|---|---|---|
| Extension | .h5, .hdf5 | .zarr |
| MIME type | ||
| Common uses | Storing large scientific and technical data hierarchically · Keeping multidimensional arrays and related metadata in one container | Storing large multidimensional arrays in chunks · Working with scientific data in cloud object storage |
| How to open | Use HDF5-capable visualization or analysis software or a library. Inspect the group hierarchy, dataset shapes and types, attributes, and filters. Consult the creator's documentation because applications may define their own HDF5 conventions. | Use a Zarr-capable library or analysis tool. Check version, array shape, chunk grid, codecs, and attributes. Keep the whole local directory or required object-store keys and consult producer documentation. |
| Export or convert | Before converting to CSV, Parquet, FITS, or another format, decide how to map hierarchy, array dimensions and types, attributes, units, and coordinates. Some structures or metadata may not fit the target; keep the original and compare values and metadata. | When converting from netCDF or HDF5, check dimensions, types, chunks, attributes, and coordinate conventions. Preserve the source and validate values after conversion. |
Choose by use case
Organizing complex data with groups, datasets, and attributes
Storing large scientific and technical data hierarchically · Keeping multidimensional arrays and related metadata in one container
HDF5Scientific workflows needing partial I/O or dataset-level compression
Storing large scientific and technical data hierarchically · Keeping multidimensional arrays and related metadata in one container
HDF5Reading selected regions of large arrays
Storing large multidimensional arrays in chunks · Working with scientific data in cloud object storage
ZarrSharing sharded datasets between compatible tools
Storing large multidimensional arrays in chunks · Working with scientific data in cloud object storage
ZarrKey point 1
HDF5 stores groups, datasets, and attributes inside a file. Zarr maps array or group metadata and chunk data to keys in a store.
More on the differencesKey point 2
HDF5 is a general hierarchical model with optional chunking and compression; Zarr specifies N-dimensional arrays, chunk grids, codecs, and storage keys.
More on the differencesMore on the differences
Choose HDF5 when file-based hierarchy and established HDF5 applications matter. Consider Zarr when workflows benefit from chunk-wise reads and writes on a store. Benchmark with the real access pattern and backend.
- HDF5 stores groups, datasets, and attributes in a file. Zarr stores metadata documents and encoded chunks as separate keys in a store.
- HDF5 offers a general data model and configurable layouts. Zarr specifies chunked arrays, grids, codecs, and key construction.
- Partial-read performance depends on chunk shape, access pattern, caching, and backend. On migration, validate types, missing values, attributes, codecs, chunking, and parallel access.
Frequently asked questions
Is Zarr always faster than HDF5 for partial array reads?
No. Performance depends on chunk shape, workload, caching, implementation, and storage backend; benchmark representative access.