LayerwisePercentile

LayerwisePercentile#

CalibMethod.LayerwisePercentile is an accuracy-oriented calibration method that searches percentile candidates per layer instead of applying one fixed percentile everywhere. It is useful when a model is sensitive to activation range selection and default calibration methods do not produce stable results.

Recent memory improvements#

Recent Quark ONNX pipeline updates significantly reduced the peak RSS memory footprint of LayerwisePercentile, turning it from a method that could previously require very large memory budgets into one that fits standard workstation-class hardware.

The main changes are:

  • Chunked processing during layerwise percentile calibration, replacing bulk loading of all calibration data with small-chunk iteration and buffer reuse.

  • Selective Calibration Propagation (SCP), where distribution-preserving ops such as Reshape and Transpose inherit calibration ranges instead of recomputing them.

  • Improved disk caching that preserves original tensor precision and avoids hidden memory growth caused by dtype promotion on reload.

  • Earlier release of intermediate resources, including inference sessions and other large temporary objects.

When to use it#

  • Try it when MinMax or a single Percentile setting produces noticeable accuracy degradation.

  • It is especially useful for models where different layers need different clipping behavior.

  • With the recent memory reductions, it is now practical to evaluate LayerwisePercentile on standard hardware instead of ruling it out only because of historical memory cost.

  • It can require more calibration work than simpler methods, so expect a runtime trade-off in exchange for better calibration quality.

Key settings#

  • Set calibration_method to CalibMethod.LayerwisePercentile in the tensor config you want to calibrate.

  • Use LWPMetric to choose the metric used to compare percentile candidates.

  • Use PercentileCandidates to control which percentile values are explored during the layerwise search.

  • Use CalibWorkerNum to parallelize calibration when you have enough CPU and memory headroom.

  • Use CalibPassthroughOpTypes to enable Selective Calibration Propagation on distribution-preserving ops. The recommended list is ["Reshape", "Transpose", "MaxPool", "Split", "Slice", "Squeeze", "Unsqueeze", "Gather"].

Memory and disk behavior#

  • The optimal percentile is scored from each tensor’s histogram in a single inference pass.

  • Activations are kept in memory during calibration, minimizing calibration time and disk usage.

  • CalibPassthroughOpTypes reduces redundant calibration work on distribution-preserving ops and can further lower peak RSS.

  • TmpDir lets you move temporary calibration files away from a small default system temp directory when disk space is limited.

Takeaway#

For most models, you can now choose LayerwisePercentile based on accuracy needs rather than assuming it will exceed available memory. Start with the default settings, then tune percentile candidates and worker count as needed.