Sample images get distorted after a while

[Bs990]

Hi,
When I train with dreambooth my sample images get distorted after ~8000 steps even though I don't get any errors or high loss rate (~0.115). I tried training with xformers 0.16 and latest version of xformers with pytorch 2.0. I had the same issue with both.
1200 steps:

2400 steps:

8640 steps:

Constant with warmup (%10)
LR: 0.0000015
Noise offset 0.1
Xformers & memory efficient attention
Flip augmentation
fp16
AdamW8Bits
6 epoch 12 steps per file