Format Atlas

Guides

Choose a format for analytics and data exchange

Choose among columnar analytics, schema-based record exchange, and human-readable tables based on your platform and data types.

Which format should you choose?

Choose by the role each file plays in your workflow.

  • Columnar analytics and data lakes → Parquet
  • Schema-based record exchange → Avro
  • Columnar analytics in Hadoop ecosystems → ORC
  • Inspection and simple exchange → CSV

Choose a columnar format for analytics

Parquet and ORC are used by analytical workloads that read selected columns. Match the format to your query engine and existing data platform.

Check schemas and conversion results

Avro is used for schema-based record exchange. When converting, verify types, nulls, timestamps, decimals, field names, and compatibility.

Formats to consider

Parquet

Good for: Columnar analysis of large tabular datasets

Less suitable for: Reading content directly as text

Avro

Good for: Exchanging schema-based records

Less suitable for: Workloads focused on column-level aggregation

ORC

Good for: Storing data for analytics platforms using type information and indexes

Less suitable for: Casual sharing as readable text

CSV

Good for: Exchanging tabular data people can inspect

Less suitable for: Preserving complex types or optimizing large-scale analytics

Before you export

  • Check types, nulls, timestamps, and character encoding.
  • Review access controls and sharing scope for personal or confidential columns.
  • Test read performance and compatibility with representative data.

Related comparisons

Parquet vs ORCParquet and ORC are columnar formats for large-scale analytical data. Choose according to the query engine, existing platform, and measured workload.Avro vs ParquetAvro serializes schema-governed records; Parquet stores data by column for large-scale analytical reads. Choose by how the data is written and queried.Avro vs ORCAvro is commonly used to exchange schema-governed records. ORC stores typed data by column for analytical reading.CSV vs JSONCSV represents tabular records; JSON represents structured values such as objects and arrays. Choose based on the shape of the data and the receiving system.

Guides

Choosing a Data Exchange FormatStart with the receiving system's specification. Use JSON for structured API data, XML where an integration requires its schema, and CSV for a single table.Checks before handing off scientific arraysTransfer netCDF, HDF5 or Zarr arrays with their meaning, dependencies and reader requirements. Unlike choosing a table format, this requires checking dimensions and missing-value interpretation.

References and specifications