The idea
Partitioning divides a dataset into manageable groups that queries can select. A useful key reduces unnecessary reading. An excessively detailed key can create many tiny fragments and increase metadata and maintenance work instead of improving performance.
Design choice
Start with common filters and expected data volume. Test realistic query patterns before choosing a layout. A high-cardinality identifier is often a poor partition key when most queries scan broader time ranges or categories.
Worked example
A time-oriented analysis stores daily partitions and groups small files within each day. Queries for a month inspect the relevant days rather than millions of customer folders.
Before you ship
- Test representative query filters
- Measure file and partition counts
- Plan compaction and late updates
Examples are illustrative. Adapt the checks to your own environment.