User Guide

There are several steps to quantize a floating-point model with Quark for PyTorch:

1. Load original float model

2. Set quantization configuration

3. Define dataloader

4. Use the Quark API to perform in-place replacement of the model’s modules with quantized module.

5. (Optional) Export quantized model to other format such as ONNX

More details:

• Quick Start Example

• Configuring Quark for PyTorch

• Adding Calibration Datasets

• Exporting for ONNX & Json-Safetensors(vLLM Adopted)

• Feature Description