Anyone successfully implemented colorization?

[Li-En-Good]

Hi I'm wonder if anyone is successful and can provide any insights for easier training.
I trained my data using the default parameters and didn't have much success.
Thanks!

[Badw0lf613]

Hi I'm wonder if anyone is successful and can provide any insights for easier training. I trained my data using the default parameters and didn't have much success. Thanks!

I have the same problem with you. I would be very grateful if someone could sort it out.

[WeiqiZhang1999]

Me too, but have you ever tried to increase the time step(defult=1000) in the test phrase?

[Li-En-Good]

I haven't. Did it help?

[WeiqiZhang1999]

No, I tried to set 2000 time steps but the result doesn't improve anyway.

[Li-En-Good]

Interesting. My loss is also getting higher and not decreasing after ~1000 steps.

[WeiqiZhang1999]

BTW, how many images did you use for training?

[Li-En-Good]

I only used ~50 images but would it be the issue? The results are not even close to what I feed it. And the data I feed it is all in the same category. Something like paint the color of leaves and branches into green and brown.

[WeiqiZhang1999]

Thank you for your reply. Actually, I did another image-to-image translation like style transfer. And I used ~500 images for training. Some results are good but others are completed noise or images with a bit of noise. So I think adding images for training will help you but I think how to get the result like the paper showed is quite important.

[Li-En-Good]

I see. Could you please share some results you got and how many steps you trained using the default parameters? Thank you!

[WeiqiZhang1999]

I'm sorry I can't show you the results because I'm using medical images which are related to the privacy problem.

[Li-En-Good]

No problem! Could you share how many steps though? I'm also using medical images, specifically microscopy images.

[WeiqiZhang1999]

I trained about 100,000 iterations for my dataset and I'm still trying to tune hyperparameters like the learning rate and beta scheduler or even the loss function.

[xenova]

Been having these problems for a while (#37)

@Janspiry any updates? 😅

[Janspiry]

@xenova , I check it and find no problems.

[Janspiry]

Friends, sorry this problem has not been solved yet and I don't have time to experiment with it. The color shift problem also exists on some other tasks. The issue in my other repository shows that GroupNorm may be the cause [https://github.com/Janspiry/Image-Super-Resolution-via-Iterative-Refinement/issues/69#issuecomment-1345372952], have any of you tried the relevant changes?

[urimerhav]

@Janspiry

I have some insights on this, sharing in case someone wants to test that it also helps on the Colorization task.

Following Janspiry's link, I've decided to simply replace all instances of GroupNorm with BatchNorm2D. In particular, in guided_diffusion_models/nn.py we have this class GroupNorm32 which if I replace with an equivalent BatchNorm32.

Updating its normalize function to return the new class, the model throws an exception in a middle layer in Unet!

AttentionBlock has the following implementation.

It seems to reshape all spatial dimensions into a single dimension, making the image 1D - this fails on batchnorm2d, but I can't make any sense of doing groupnorm in groups of 32 dimensions along the new spatial dimensions intra-sample. I think it's a case where the code was silently agreeing to do something that doesn't particular sense - not sure why this is very harmful in colorization but not so in uncropping.

For my task of image denoising, it solved the off-color issue.

[kkamankun]

@urimerhav
Thank you for your help. I have experimented with GroupNorm, BatchNorm, InstanceNorm, and no normalization on my dataset, but I did not get any improved results with any of them. Specifically, when I experimented with BatchNorm, only one color appeared. In the future, I plan to conduct experiments with larger batch size and adjusted learning rate.

[sjg918]

@developerTae
Hello. Thanks for sharing your experience.
I performed colorization experiments using batchnorm on my diffusion network.
As you mentioned, I ran into a situation where only 1 color appeared.
Have you ever solved this problem?

Does anyone have any insight on this issue???

[kkamankun]

@sjg918
No, I have not solved the problem yet, but I am still working on it.

[duxiangcheng]

@developerTae Hello. Thanks for your sharing. Have you solved the problem of only 1 color appeared?

[duxiangcheng]

@Li-En-Good @Badw0lf613 hello, have you successfully implemented colorization? can you give me some advice？In my experiments, the result of coloring is very strange, and the overall bias is towards one color

[fremk]

Hey guys, I'm using the colorization config to train the model for noisy, blurred and darkened gray scale image -> RGB image translation. I'm training using 10k images and using the default parameters.
At first I faced the same problem where the model just converged to a single color, but out of curiosity I let the model train for a longer period of time (1000 epochs, ~4 days on 2 RTX3090) and the results are way better. (Not perfect in any way, but considerably better than at first)

[rutuja1409]

Hey guys, I trained for 150 epochs for colorization task and I got the output as below.
Here is the validation data that is provided

And here is the output

Can someone help me @urimerhav @duxiangcheng @xenova @Janspiry @developerTae why the output are darkened and not generate white background image like in input validation dataset?

[xchwan]

Hey guys, I'm using the colorization config to train the model for noisy, blurred and darkened gray scale image -> RGB image translation. I'm training using 10k images and using the default parameters. At first I faced the same problem where the model just converged to a single color, but out of curiosity I let the model train for a longer period of time (1000 epochs, ~4 days on 2 RTX3090) and the results are way better. (Not perfect in any way, but considerably better than at first)

@fremk Could you share your result? I'm confusing how many epoch need?

[TumVink]

@kkamankun
Heyy friend, are you making some progress on this task?

[TumVink]

Hey guys, I'm using the colorization config to train the model for noisy, blurred and darkened gray scale image -> RGB image translation. I'm training using 10k images and using the default parameters. At first I faced the same problem where the model just converged to a single color, but out of curiosity I let the model train for a longer period of time (1000 epochs, ~4 days on 2 RTX3090) and the results are way better. (Not perfect in any way, but considerably better than at first)

@fremk
Congratulations!
Could you maybe share the loss curves? Does the training loss goes down during the training?

[fremk]

Hey guys, I'm using the colorization config to train the model for noisy, blurred and darkened gray scale image -> RGB image translation. I'm training using 10k images and using the default parameters. At first I faced the same problem where the model just converged to a single color, but out of curiosity I let the model train for a longer period of time (1000 epochs, ~4 days on 2 RTX3090) and the results are way better. (Not perfect in any way, but considerably better than at first)

@fremk Could you share your result? I'm confusing how many epoch need?

Hello,
I'm sorry due to privacy related issues I can't share the results. As this image translation task isn't really something crucial in my internship I didnt spend too much time on it after the results I got. Just try and let it train for as much time as you have available and check the results (1000 epochs maybe).
I'm just speculating as I still think the model is not training right and there's definitely something wrong with it.

[fremk]

Hey guys, I'm using the colorization config to train the model for noisy, blurred and darkened gray scale image -> RGB image translation. I'm training using 10k images and using the default parameters. At first I faced the same problem where the model just converged to a single color, but out of curiosity I let the model train for a longer period of time (1000 epochs, ~4 days on 2 RTX3090) and the results are way better. (Not perfect in any way, but considerably better than at first)

@fremk Congratulations! Could you maybe share the loss curves? Does the training loss goes down during the training?

The loss curves are hectic, they do not converge and they're not decreasing in a stable manner either. They're just all over the place. Similar to the ones posted earlier ^.
As I said in my previous response, as this colorization step is not an important one for me at the moment, I did not recheck it. I just had some initial results to work with. Far from the best and they're not consistent by all means, but they were something :')

[TumVink]

@fremk thanks for your reply!
I had the same thought that this model has something wrong inside it but I could not find it out.
Anyway I will try another week and I will post it here if I find something.
Best,
Jingsong