Colorization training isn't working #37
Comments
|
I'm also experiencing the same issues. Are your results also very unsaturated? |
|
I'm not sure if you have tried this, but what about setting "clip_denoised" to False (instead of True, which is the default)? It might result in more saturated results. ^ I will try this for my task and let you know how it goes |
|
Thanks @xenova , waiting for your update |
|
After training for another 3 hours with clip_denoised=False, I haven't seen any improvement. Perhaps @Janspiry can provide some extra assistance. |
|
@ksunho9508 @xenova I am still unable to obtain reliable results. In my opinion, the flicker dataset does not contain enough data to generalize this task via diffusion based methods. The authors of the original paper applied their method to the ImageNet dataset, which contains much more training data. |
|
@Janspiry I've also tried on my custom dataset (with millions of images), and I get the same results :/ ... I'm really not sure how this is the only task that is facing these issues; all other tasks seem to work fine. |
|
@xenova I'll make sure there are no bugs in the coloration part of the code |
|
@Janspiry Thank you. And can you add config file of super resolution too? |
|
I also found this problem. I used my own small-scale data set to train it, but still failed to get results after many epoch。 @Janspiry |
|
@omerb01 |
|
I experienced the same problem. ------------------------------Validation Start------------------------------ INFO: train/mse_loss: 0.011829124199711352 ------------------------------Validation Start------------------------------ |
没有，扩散模型的损失函数计算是计算噪声和预测噪声间的mse_loss，详见：#26 (comment) -1282232897。而且扩散模型的推理也存在很大的随机性，出现这种情况很正常 23-09-03 04:09:21.974 - INFO: train/mse_loss: 0.004320403648868778 ------------------------------Validation Start------------------------------ 23-09-03 04:23:16.320 - INFO: train/mse_loss: 0.004661682129078468 ------------------------------Validation Start------------------------------ 23-09-03 04:37:05.177 - INFO: train/mse_loss: 0.004233014806692961 ------------------------------Validation Start------------------------------ 23-09-03 04:50:55.078 - INFO: train/mse_loss: 0.004784488215476157 ------------------------------Validation Start------------------------------ |
|
您好，我目前的问题貌似就是过拟合，我的数据集10k张图，在两块3090上训练了12个小时，然后val上只能生成噪声了。val loss也一直0.7几 |
|
For me, it happens as well. The training loss decreases very quickly and drops to 0.02 after 5 epochs, but the validation result is bad as hell. |
I think it is very normal actually, that the val loss is much larger than the training loss, considering that the loss value is calculated differently during inference than training. |
|
我最开始训练效果也不好，用默认的mse损失函数训练1200epoch着色的图片会偏色严重，后来换了一个损失函数好一些，但是没有碰到val上只能生成噪声的情况。附带一些效果不好的val图片
我的训练集只有1500张，但是更换损失函数之后在测试集上的效果还可以，10k张图应该没那么容易过拟合，可以试试用训练的图片跑试试，可能连训练集上都没办法取得很好的着色效果，建议换一下损失函数试试，我目前的训练效果还可以
------------------ 原始邮件 ------------------
发件人: "Janspiry/Palette-Image-to-Image-Diffusion-Models" ***@***.***>;
发送时间: 2023年10月11日(星期三) 凌晨2:44
***@***.***>;
***@***.******@***.***>;
主题: Re: [Janspiry/Palette-Image-to-Image-Diffusion-Models] Colorization training isn't working (Issue #37)
您好，我目前的问题貌似就是过拟合，我的数据集10k张图，在两块3090上训练了12个小时，然后val上只能生成噪声了。val loss也一直0.7几
I think it is very normal actually, that the val loss is much larger than the training loss, considering that the loss value is calculated differently during inference than training.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Heyy Glad to hear it! At least it proves the correctness of this repo. BW, |
|
The hybrid loss function you mentioned is a mixture of true variational lower bound and BCE.
Am i correct?
…________________________________
Von: 1228967342 ***@***.***>
Gesendet: Montag, 23. Oktober 2023 17:10:26
An: Janspiry/Palette-Image-to-Image-Diffusion-Models
Cc: Jingsong Liu; Comment
Betreff: Re: [Janspiry/Palette-Image-to-Image-Diffusion-Models] Colorization training isn't working (Issue #37)
混合损失函数
―
Reply to this email directly, view it on GitHub<#37 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARFVZL5XXPK77A2XZTUCMZTYA2CGFAVCNFSM6AAAAAAQCBIWN6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZVGQZDINRYGU>.
You are receiving this because you commented.Message ID: ***@***.***>
|
不是，只是很简单的混合 |
|
@1228967342 你好，我也遇到了相同的问题，请问可以分享一下你的混合损失设计吗 |
I downloaded the flicker25k dataset, preprocessed it and train a model with these modifications in the config file:
The rest of the configurations remained as in the current config file.
Even after 1000 training epochs, the model still produces bad results.
Is there anything I'm missing? Thanks.
The text was updated successfully, but these errors were encountered: