Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix model error #243

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

fix model error #243

wants to merge 1 commit into from

Conversation

f-hy
Copy link
Contributor

@f-hy f-hy commented Feb 18, 2024

fix the following error:

Traceback (most recent call last):
  File "e:\data\frid\python\codes\adlpi\labml_nn\transformers\rope\__init__.py", line 231, in <module>
    _test_rotary()
  File "e:\data\frid\python\codes\adlpi\labml_nn\transformers\rope\__init__.py", line 227, in _test_rotary
    inspect(rotary_pe(x))
  File "E:\data\frid\python\p121\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
  File "e:\data\frid\python\codes\adlpi\labml_nn\transformers\rope\__init__.py", line 188, in forward
    x_rope = (x_rope * self.cos_cached[:x.shape[0]]) + (neg_half_x * self.sin_cached[:x.shape[0]])
RuntimeError: The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 3

the reason is that the dimensions of two multiplied vectors do not match.

fix the following error:
```
Traceback (most recent call last):
  File "e:\data\frid\python\codes\adlpi\labml_nn\transformers\rope\__init__.py", line 231, in <module>
    _test_rotary()
  File "e:\data\frid\python\codes\adlpi\labml_nn\transformers\rope\__init__.py", line 227, in _test_rotary
    inspect(rotary_pe(x))
  File "E:\data\frid\python\p121\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
  File "e:\data\frid\python\codes\adlpi\labml_nn\transformers\rope\__init__.py", line 188, in forward
    x_rope = (x_rope * self.cos_cached[:x.shape[0]]) + (neg_half_x * self.sin_cached[:x.shape[0]])
RuntimeError: The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 3
```
the reason is that the dimensions of two multiplied vectors do not match.
@vpj
Copy link
Member

vpj commented Mar 2, 2024

In cached sin and cos the first dimension is the sequence length. So it should be cos_cached[:x.shape[0]]

@chen-xin-94
Copy link

In cached sin and cos the first dimension is the sequence length. So it should be cos_cached[:x.shape[0]]

That's right. I think the fix should be like this: #249

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants