Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] ConstantGradScaler and loss-scale argument not match #776

Open
BeingGod opened this issue Apr 12, 2024 · 0 comments
Open

[BUG] ConstantGradScaler and loss-scale argument not match #776

BeingGod opened this issue Apr 12, 2024 · 0 comments

Comments

@BeingGod
Copy link

BeingGod commented Apr 12, 2024

Describe the bug
The usage and description of loss-scale is inconsistent. The argument of loss-scale expect to get a number of positive power of 2 but ConstantGradScaler set loss-scale to real scale directly rather than 2**loss-scale.

Argument Description:
image

Argument Usage:
image

To Reproduce
Steps to reproduce the behavior. The easier it is to reproduce the faster it will get maintainer attention.

Expected behavior
A clear and concise description of what you expected to happen.

Stack trace/logs
If applicable, add the stack trace or logs from the time of the error.

Environment (please complete the following information):

  • Megatron-LM commit ID
  • PyTorch version
  • CUDA version
  • NCCL version

Proposed fix
If you have a proposal for how to fix the issue state it here or link to a PR.

Additional context
Add any other context about the problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant