We identify the missing residual error term ( We name it 'Compensation-aware Error' ) in GPTAQ, which comes the discrepancy between compensated and original weights. Therefore, our method strictly aligns with the original full-precision ouput at each column.

Our codebase is heavily relied on GPTAQ, with simple modifications. Please see fake_quant/gptaq_utils_r.py for details.

Take weight-only quantization as an example:

cd fake_quant bash weight_group_3bit.sh ### per-group quantization bash weight_group_2bit.sh ### Quarot + per-group quantization 

I'm continuously working on improving the stability of ResComp. Since we form a more precise optimization objective, and we only use 128 samples, it may be more sensitive to the quality of calibration data. I warmly welcome further discussions, feel free to contact me (list@zju.edu.cn).

Our codebase is built heavily on previous works, and we would like to acknowledge and thank their awesome contribution:

• GPTAQ: Efficient finetuning-free quantization for asymmetric calibration github
• GPTQ: Accurate post-training quantization for generative pre-trained transformers github
• QuaRot: Outlier-free 4-bit inference in rotated llms github
• SpinQuant: Llm quantization with learned rotations github

If you find our work useful in your research, please kindly cite this paper:

@inproceedings{lirethinking, title={Rethinking Residual Errors in Compensation-based LLM Quantization}, author={Li, Shuaiting and Deng, Juncan and Xu, Kedong and Deng, Rongtao and Gu, Hong and Jiang, Minghan and Shen, Haibin and Huang, Kejie}, booktitle={The Fourteenth International Conference on Learning Representations} } 

Besides, if you are interested in vector quantization, checkout our previous papers: SSVQ [ICCV'25], MVQ [ASPLOS'25], VQ4DiT [AAAI'25], ViM-VQ [ICCV'25]. I'm also seeking collabration opportunity in CUDA kernel optimization to better support SSVQ.