Compare
Parquet vs Zarr
Apache Parquet stores tables in files divided into row groups and column chunks. Zarr stores multidimensional arrays as chunks with array metadata in a store.
Which format should you choose?
Choose Parquet for column-oriented scans of tabular data in analytical queries. Consider Zarr for large multidimensional arrays accessed by chunks or stored in object storage.
Good fit for Parquet
Store structured tables for analytics
- Large datasets read by selected columns
Good fit for Zarr
Storing large multidimensional arrays in chunks · Working with scientific data in cloud object storage
- Reading selected regions of large arrays
- Sharing sharded datasets between compatible tools
Basic information
| Field | Parquet | Zarr |
|---|---|---|
| Extension | .parquet | .zarr |
| MIME type | application/vnd.apache.parquet | |
| Common uses | Store structured tables for analytics | Storing large multidimensional arrays in chunks · Working with scientific data in cloud object storage |
| How to open | Use a Parquet-aware tool to inspect schema, column types, row counts, and values. | Use a Zarr-capable library or analysis tool. Check version, array shape, chunk grid, codecs, and attributes. Keep the whole local directory or required object-store keys and consult producer documentation. |
| Export or convert | From CSV or other formats, check types, nulls, timestamps, encoding, partitions, and row counts. | When converting from netCDF or HDF5, check dimensions, types, chunks, attributes, and coordinate conventions. Preserve the source and validate values after conversion. |
Choose by use case
Large datasets read by selected columns
Store structured tables for analytics
ParquetReading selected regions of large arrays
Storing large multidimensional arrays in chunks · Working with scientific data in cloud object storage
ZarrSharing sharded datasets between compatible tools
Storing large multidimensional arrays in chunks · Working with scientific data in cloud object storage
ZarrKey point 1
Parquet row groups contain column chunks located through file metadata; Zarr stores array metadata such as shape, type, chunk grid, and codecs alongside encoded chunks.
More on the differencesKey point 2
Parquet is organized around table rows and columns; Zarr is organized around N-dimensional array axes and chunks.
More on the differencesMore on the differences
Use Parquet for analytical table workloads that read selected columns. Use Zarr for chunked access to large arrays. Tune row groups or chunks for actual queries and storage, and keep the formats' distinct data models in mind.
- Parquet metadata locates column chunks within row groups. Zarr metadata describes array shape, data type, chunk grid, and codecs, while chunk data is stored as separate keys.
- Parquet maps tabular rows and columns; Zarr maps array dimensions and chunks. Performance depends on layout matching the query or slice pattern.
- When converting, map rows, columns, and dimensions; validate types, nulls or missing values, units, time and coordinate metadata, codecs, and read performance. Transfer all Zarr metadata and chunk keys together.
Frequently asked questions
Can Parquet and Zarr replace each other directly?
No. One models tables and the other multidimensional arrays. A conversion needs an explicit mapping for dimensions, types, missing values, and metadata.