Improved PTQ Method for Diffusion Models

Code base for 2023 Summer Research Internship Programme both in Department of Computer Science at The University of Hong Kong assigned to the AI, Robotics and Visual Computing and School of Computer Science and Engineering at Beihang University assigned to the IRIP Lab.

Reproduce results in PTQ4DM

We found the sample distribution proposed in PTQ4DM does not match the experiment results. The code in PTQ4DM shows that they only quantize the 2D convolutional, linear layer and the activation after these layers. However, U-Net used in improved-diffusion has different structure from traditional ResNet, which indicates function from QDrop can not be applied directly. Therefore, we quantized the diffusion model again, instead of using the partly quantized model. Then, we compared the performance of Normally Distributed Time-step Calibration (NDTC, proposed in PTQ4DM) and Uniformly Distributed Time-step Calibration on both ImageNet64 and CIFAR10. We found normally calibration even worse than uniform calibration, which is same to the result in Q-Diffusion.

A further step based on PTQ4DM

We observed that IS and sFID in 8-bit PTQ4DM can reach or even exceed those in the full-precision model. However, there is always a significant loss on FID after quantization. Therefore, we propose some methods (D2IN) to improve the performance on FID and achieve a lower FID than that of DDIM.

Method	CIFAR10	IS↑	FID↓	sFID↓	ImageNet64	IS↑	FID↓	sFID↓
FP32	DDIM	9.25	10.60	7.41	DDIM	15.20	19.59	9.45
PTQ4DM	Normal	9.38	12.85	7.53	Normal	15.59	22.02	6.62
DI2N (Ours)	Uniform	9.25	10.71	7.38	Uniform	15.30	19.27	6.63

IS loss is caused by the extent of quantization. However, it still outperforms that of DDIM.

Reference

42Shawn/PTQ4DM: Implementation of Post-training Quantization on Diffusion Models (CVPR 2023) (github.com)

Xiuyu-Li/q-diffusion: [ICCV 2023] Q-Diffusion: Quantizing Diffusion Models. (github.com)

CompVis/latent-diffusion: High-Resolution Image Synthesis with Latent Diffusion Models (github.com)

What are Diffusion Models? | Lil'Log (lilianweng.github.io)

Generative Modeling by Estimating Gradients of the Data Distribution | Yang Song (yang-song.net)

openai/guided-diffusion (github.com)

openai/improved-diffusion: Release for Improved Denoising Diffusion Probabilistic Models (github.com)

wimh966/QDrop: The official PyTorch implementation of the ICLR2022 paper, QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization (github.com)

Generative Modeling by Estimating Gradients of the Data Distribution | Yang Song (yang-song.net)

CompVis/stable-diffusion: A latent text-to-image diffusion model (github.com)

yhhhli/BRECQ: Pytorch implementation of BRECQ, ICLR 2021 (github.com)

[2304.08818] Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (arxiv.org)

[2305.10657] PTQD: Accurate Post-Training Quantization for Diffusion Models (arxiv.org)

[1911.07190] Loss Aware Post-training Quantization (arxiv.org)

[2006.11239] Denoising Diffusion Probabilistic Models (arxiv.org)

[2102.09672] Improved Denoising Diffusion Probabilistic Models (arxiv.org)

[2010.02502] Denoising Diffusion Implicit Models (arxiv.org)

[2203.05740] QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization (arxiv.org)

[2105.05233] Diffusion Models Beat GANs on Image Synthesis (arxiv.org)

[1505.04597] U-Net: Convolutional Networks for Biomedical Image Segmentation (arxiv.org)

[2302.04304] Q-Diffusion: Quantizing Diffusion Models (arxiv.org)