File-to-File LLM Quantization

File-to-File LLM Quantization#

Challenges in Quantizing Ultra-Large Models#

For quantization schemes that operate independently on each tensor — such as weight-only quantization (FP8, MXFP4, INT4, etc.) or dynamic activation quantization + weight quantization — standard tools still require loading the entire model into GPU memory. For a 600B+ parameter model this means hundreds of gigabytes, leading to Out of Memory (OOM) failures even on multi-GPU nodes.

This is wasteful: these quantization schemes are per-tensor operations that do not require the full model graph. Each tensor can be quantized independently.

The Solution: File-to-File Quantization, No Full Model Loading#

File-to-File Quantization exploits this independence by reading each .safetensors file, quantizing the weights it contains, and writing the result directly to a new file — without ever loading the full model into memory.

Standard vs. File-to-File quantization#

Item

Standard quantization

File-to-file quantization

What gets loaded

The entire model (all files / full state dict)

One file at a time (plus optional recovery metadata)

Core operation

Per-tensor weight quantization

Per-tensor weight quantization (same math)

Peak memory driver

Full model size (often 100s of GB for 600B+)

Largest single file (typically ~5–10 GB)

Pre-quantized input handling

Typically assumes BF16/FP16 inputs

Can recover then re-quantize (e.g., FP8 or compressed-tensors)

Output

Quantized weights + exported artifacts

Quantized weights + exported artifacts

Both approaches produce the same per-tensor quantization results. File-to-file mode supports weight-only quantization (e.g., FP8, MXFP4, INT4) and dynamic activation quantization + weight quantization, where each tensor can be processed independently — no calibration data, forward pass, or full model graph is required. The only difference is how tensors are loaded: the standard approach loads the entire model at once, while file-to-file reads and writes one .safetensors file at a time.

Supported Input Formats#

Format

Quant Method

Description

Dependencies

BF16 / FP16

—

Standard HuggingFace model, loaded directly

—

FP8

fp8

FP8 weights with _scale_inv tensors; dequantized before re-quantization

Triton

compressed-tensors

compressed-tensors

HuggingFace-style packed weights; decompressed before re-quantization

compressed-tensors

Usage Examples#

Example 1: File-to-File Quantization via Python API#

This approach mirrors the _build_quant_config helper in quantize_quark.py — it uses LLMTemplate to auto-detect the model architecture and build the quantization config from a scheme name, avoiding manual QConfig construction.

import json
from quark.torch import LLMTemplate, ModelQuantizer

model_path = "/path/to/model"
save_path = "/path/to/output"

# Read model type from config.json
with open(f"{model_path}/config.json") as f:
    model_type = json.load(f)["model_type"]

# Build quant config
template = LLMTemplate.get(model_type)
quant_config = template.get_config(
    scheme="mxfp4",                              # or "fp8", "int4_wo_128", etc.
    exclude_layers=["*self_attn*", "*lm_head"],
)

# Run file-to-file quantization
quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=model_path,
    save_path=save_path,
)

# Multi-GPU: pass a list of devices to distribute shards for parallel quantization
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=model_path,
    save_path=save_path,
    device=["cuda:0", "cuda:1", "cuda:2", "cuda:3",
            "cuda:4", "cuda:5", "cuda:6", "cuda:7"],
)

Example 2: File-to-File Mode via quantize_quark.py#

The quantize_quark.py script supports a --quant_flow file2file mode that bypasses model loading and the standard quantization pipeline. It reads config.json to determine the model type, builds quant_config using the same LLMTemplate / --quant_scheme mechanism, and directly quantizes safetensors files in a file-to-file manner.

# Single device
python examples/torch/language_modeling/llm_ptq/quantize_quark.py \
    --model_dir /path/to/model \
    --quant_scheme mxfp4 \
    --exclude_layers "*self_attn*" "*mlp.gate" "*mlp.gate.linear" "*lm_head" \
    --output_dir /path/to/output \
    --quant_flow file2file \
    --skip_evaluation

# Multi-GPU: add --multi_gpu to distribute shards across all visible GPUs
python examples/torch/language_modeling/llm_ptq/quantize_quark.py \
    --model_dir /path/to/model \
    --quant_scheme mxfp4 \
    --output_dir /path/to/output \
    --quant_flow file2file \
    --multi_gpu \
    --skip_evaluation

Note

--file2file_quantization still works but is deprecated; use --quant_flow file2file instead.

API Reference: direct_quantize_checkpoint#

ModelQuantizer.direct_quantize_checkpoint(
    pretrained_model_path,
    save_path,
    keep_excluded_layers_as_original_model_state=False,
    weight_converters=None,
    device=None,
    presharded_weights=None,
)

Parameter

Default

Description

pretrained_model_path

(required)

Path to the pretrained model directory containing .safetensors files.

save_path

(required)

Directory path to save the quantized .safetensors files and all auxiliary files (config.json, model.safetensors.index.json, tokenizer files, etc.).

keep_excluded_layers_as_original_model_state

False

If True, excluded layers that are already quantized in the source checkpoint keep their original model-state format in the output. Useful when the source is a pre-quantized checkpoint and certain layers should pass through unchanged.

weight_converters

None

Optional list of WeightConverter instances to transform tensors after precision recovery and before quantization. See Weight Transformation with WeightConverter below.

device

None

Device(s) for tensor operations. Pass a single device (e.g. "cuda", "cuda:0", "cpu"; defaults to "cuda" when None) for single-device mode, or a list (e.g. ["cuda:0", "cuda:1"]) to distribute shards round-robin across those devices and quantize them in parallel, one worker process per device. With a multi-device list, models with cross-shard weight/scale dependencies fall back to single-device automatically, and FP8 source models are not yet supported (single-device FP8 works).

presharded_weights

None

Optional pre-loaded shard dictionary. When provided, the method uses these tensors directly instead of reading from pretrained_model_path.

Weight Transformation with WeightConverter#

WeightConverter allows post-quantization weight transformations on each .safetensors shard as it passes through the file-to-file pipeline. Converters run after precision recovery (dequantization of pre-quantized inputs) and before Quark quantization.

Use cases:

  • Splitting a fused projection (e.g. gate_up_proj) into separate tensors (gate_proj, up_proj) that Quark quantizes individually.

  • Renaming tensor keys to match a different model variant’s naming convention.

  • Applying custom packing or layout transformations after quantization.

Each converter receives the tensor dictionary for one .safetensors shard and returns a modified dictionary. Converters in the list are applied in order.

Example — split fused gate/up projection:

from quark.torch import LLMTemplate, ModelQuantizer
from quark.torch.quantization.weight_convert import Chunk, WeightConverter

template = LLMTemplate.get("qwen")
quant_config = template.get_config(scheme="mxfp4")

weight_converters = [
    WeightConverter(
        "gate_up_proj.weight",
        ["gate_proj.weight", "up_proj.weight"],
        operations=[Chunk(dim=0)],
    ),
]

quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path="/path/to/model",
    save_path="/path/to/output",
    weight_converters=weight_converters,
)

Note

WeightConverter is suited for post-recovery single-suffix rename or one-source split operations. It is not suited for multi-source merges, cross-shard scale pairing, or dtype conversions such as FP4 → FP8. For those cases, generate a normalization script that pre-processes the checkpoint before running file-to-file quantization.

Progressive (Two-Step) Quantization#

Progressive quantization applies quantization in two sequential stages within a single file-to-file pass:

  1. The weight is first quantized to an intermediate format (e.g. FP8 E4M3 per-tensor).

  2. The intermediate result is then re-quantized to the final target format (e.g. INT4 per-channel).

This produces better accuracy than direct single-step quantization for aggressive targets such as INT4, because the first stage preserves more information than the final format alone.

Built-in progressive scheme — ``int4_fp8``:

The int4_fp8 scheme (W4A8) uses progressive quantization internally:

  • Weight stage 1: FP8 E4M3 per-tensor static quantization.

  • Weight stage 2: INT4 per-channel static quantization of the FP8 result.

  • Activation: FP8 E4M3 per-tensor dynamic quantization.

This matches the AMD-Quark W4A8 recipe used by models such as amd/Kimi-K2-Thinking-W4A8.

import json
from quark.torch import LLMTemplate, ModelQuantizer

model_path = "/path/to/model"
save_path = "/path/to/output"

with open(f"{model_path}/config.json") as f:
    model_type = json.load(f)["model_type"]

template = LLMTemplate.get(model_type)
quant_config = template.get_config(scheme="int4_fp8")   # progressive W4A8

quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=model_path,
    save_path=save_path,
)

python examples/torch/language_modeling/llm_ptq/quantize_quark.py \
    --model_dir /path/to/model \
    --quant_scheme int4_fp8 \
    --quant_flow file2file \
    --output_dir /path/to/output \
    --skip_evaluation

Custom progressive scheme via ``ProgressiveSpec``:

from quark.torch.quantization.config.config import (
    QConfig, QLayerConfig, ProgressiveSpec,
    FP8E4M3PerTensorSpec, Int4PerChannelSpec,
)
from quark.torch import ModelQuantizer

weight_spec = ProgressiveSpec(
    first_stage=FP8E4M3PerTensorSpec(
        observer_method="min_max", scale_type="float32", is_dynamic=False
    ),
    second_stage=Int4PerChannelSpec(
        symmetric=True, scale_type="float32",
        round_method="half_even", ch_axis=0, is_dynamic=False
    ),
).to_quantization_spec()

quant_config = QConfig(global_quant_config=QLayerConfig(weight=weight_spec))
quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path="/path/to/model",
    save_path="/path/to/output",
)

Output tensor naming for progressive quantization:

Progressive quantization writes three tensors per weight:

Tensor key

Contents

<layer>.weight

Packed weight from stage 2 (e.g. INT4 values).

<layer>.weight_scale

Scale from stage 1 (e.g. FP8 per-tensor scale).

<layer>.weight_scale_2

Scale from stage 2 (e.g. INT4 per-channel scale).

Requirements#

  • PyTorch

  • safetensors

  • Triton (optional, only needed for FP8 input format dequantization)

  • compressed-tensors (optional, only needed for compressed-tensors input format)