Image-to-image translation with mostly black images

[cristianpjensen]

Hi, I am currently training Palette for a task that involves translating from one bad medical image to a higher quality one. The images are mostly black, since they are kind of like MRI. However, I do not seem to get good outputs. I think the U-net is working pretty well in predicting the noise (MSE = 0.0024584989984181116 at 127654 iters), but the output of the denoising is always just one color (the color is different every time, e.g. blue, green, light-red) with a little bit of noise.

I am just using a small dataset of 48 images to test whether it would even work before I use a lot of computational power on a larger dataset. But, I feel like it should work...

I have three theories on why this does not work:

1. The dataset is too small. (I would just have to train with my larger dataset of a few thousand images.)
2. Diffusion models are not good at predicting mostly black images. (I thought about maybe converting my images to grayscale (since they are mostly black and white, like MRI). Maybe it would be better at this task.)
3. I have to train the model longer. (But, I do not see the MSE of the U-net going down anymore, so I am not so sure about this one)

Is there anyone with any experience in what I am trying to achieve that could validate or invalidate my theories? I am really lost at why it is not working...

Thanks for any help.

Btw, this is the expected output:

And this is what Palette is outputting:

This is the backward diffusion process:

[TumVink]

Maybe try another repo.
I have noticed many people experienced similar problems on this repo

[Ukuer]

Hi, I have the same problem with you. The images in my datasets are specific, and almost in one single color. I agree with your opinions 1 and 2. My datasets is also small, and I also think that diffusion may be not suitable for generating the image with almost single color.

[gguyinging]

那生成黑白两种颜色的图片呢？我感觉也会有这种问题的存在