Data engineering

Partition for the questions people ask

Editorial edition · 16 November 2026

The idea

Partitioning divides a dataset into manageable groups that queries can select. A useful key reduces unnecessary reading. An excessively detailed key can create many tiny fragments and increase metadata and maintenance work instead of improving performance.

Design choice

Start with common filters and expected data volume. Test realistic query patterns before choosing a layout. A high-cardinality identifier is often a poor partition key when most queries scan broader time ranges or categories.

Worked example

A time-oriented analysis stores daily partitions and groups small files within each day. Queries for a month inspect the relevant days rather than millions of customer folders.

Before you ship

  • Test representative query filters
  • Measure file and partition counts
  • Plan compaction and late updates

Examples are illustrative. Adapt the checks to your own environment.

Further reading

Apache Parquet documentation

Search the library