The idea
Average latency can conceal a small group of very slow requests. Percentiles describe positions in the distribution and help teams see the experience near its tail. They must be interpreted with the measurement window and sample size.
Design choice
Keep the unit and aggregation method consistent. Do not average percentiles from separate groups and call the result a global percentile. Use an appropriate shared histogram or combine the underlying observations when feasible.
Worked example
A service reports a median of 120 milliseconds and a much higher 95th percentile. The next investigation focuses on the slow requests rather than optimising the already fast majority.
Before you ship
- State the unit and time window
- Inspect tail sample size
- Avoid averaging percentiles
Examples are illustrative. Adapt the checks to your own environment.