Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

it occurs 'nan' in training_stage_1 #109

Open
lwmming opened this issue Apr 10, 2024 · 1 comment
Open

it occurs 'nan' in training_stage_1 #109

lwmming opened this issue Apr 10, 2024 · 1 comment

Comments

@lwmming
Copy link

lwmming commented Apr 10, 2024

When we run 'train_stage_1.py' on CelebV-HQ dataset, we find that the training loss will be 'nan'.
Then we try to locate this issue and find that after running line76 in train_stage_1.py, i.e., 'pose_fea = self.pose_guider(pose_cond_tensor, ref_pose_tensor)' , 'nan' will occur in pose_fea[1].
Have you encountered similar problems and how did you solve them? Thank you~

@Zejun-Yang
Copy link
Owner

We recommend you to use fp16.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants