Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for M1 Mac, or non-cuda devices #11

Open
mattkanwisher opened this issue Mar 12, 2024 · 3 comments · May be fixed by #22
Open

Support for M1 Mac, or non-cuda devices #11

mattkanwisher opened this issue Mar 12, 2024 · 3 comments · May be fixed by #22
Labels
enhancement New feature or request

Comments

@mattkanwisher
Copy link

Seems like parts of the code explicitly call cuda functions, so you can't just switch it to MPS for Macs. Any roadmap for supporting Macs?

Io.py

def load_pretrained_model(model_path: str):
    vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
    tokenizer = vl_chat_processor.tokenizer

    vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
        model_path, trust_remote_code=True
    )
    vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

error

  File "/Users/user/projects/ai/DeepSeek-VL/deepseek_vl/utils/io.py", line 37, in load_pretrained_model
    vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()
  File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2528, in cuda
    return super().cuda(*args, **kwargs)
  File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 911, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 802, in _apply
    module._apply(fn)
  File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 802, in _apply
    module._apply(fn)
  File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 802, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 825, in _apply
    param_applied = fn(param)
  File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 911, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/Users/user/miniconda3/envs/py39/lib/python3.9/site-packages/torch/cuda/__init__.py", line 293, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
@mattkanwisher mattkanwisher changed the title Support for M1 Mac Support for M1 Mac, or non-cuda devices Mar 12, 2024
@soloice
Copy link

soloice commented Mar 13, 2024

We don't run it on Mac internally, so nobody is doing this. But I guess this should be easy and just need a little bit coding. We would appreciate it if you can do this and make a pull request.

@soloice soloice added the enhancement New feature or request label Mar 13, 2024
@AbeEstrada
Copy link

This PR #22

@Benjamin-eecs Benjamin-eecs linked a pull request Mar 15, 2024 that will close this issue
feat: add mps support #22
Open
@mattkanwisher
Copy link
Author

confirmed #22 makes it run now, albeit slowly, some of the ops are still unsupported by mps by PyTorch but fall back to cpu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: add mps support Fodark/DeepSeek-VL
3 participants
@mattkanwisher @AbeEstrada @soloice