Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AttributeError in MPFuture #552

Open
borzunov opened this issue Feb 5, 2023 · 2 comments
Open

AttributeError in MPFuture #552

borzunov opened this issue Feb 5, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@borzunov
Copy link
Member

borzunov commented Feb 5, 2023

This happens when a Petals server is ran on AMD GPUs:

  File "/path/hivemind/utils/mpfuture.py", line 300, in __del__
    MPFuture._active_futures.pop(self._uid, None)
AttributeError: 'NoneType' object has no attribute 'pop'
@borzunov borzunov added the bug Something isn't working label Feb 5, 2023
@justheuristic
Copy link
Member

To the best of my knowledge, the only way this can occur is if two threads race for creating MPFuture. The first thread checked that it needs to initialize and began doing so; while it was at it, the second thread created MPFuture and, say, failed - and tried to delete it - but could not, because the first hasn't finished initializing yet.

@justheuristic
Copy link
Member

Still, i think that the attempt to run with AMD GPUs failed elsewhere, and this is only a symptom of the real problem

borzunov pushed a commit that referenced this issue Feb 14, 2023
This PR addresses the bug reported in #552 - or, at least, it should, since we cannot reproduce the problem locally.
mryab pushed a commit that referenced this issue Feb 15, 2023
This PR addresses the bug reported in #552 - or, at least, it should, since we cannot reproduce the problem locally.

(cherry picked from commit 8c98caa)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants