Meta-Llama-3-70B-Instruct running out of memory on 8 A100-40GB

[whatdhack]

Describe the bug

Out of memory. Tried to allocate X.XX GiB .....

Minimal reproducible example

I guess any A100 system with 8+ GPUs

python example_chat_completion.py

Output

<Remember to wrap the output in ```triple-quotes blocks```>

Out of memory. Tried to allocate X.XX GiB  .....

Runtime Environment

• Model: Meta-Llama-3-70B-Instruct
• Using via huggingface?: no
• OS: Linux
• GPU VRAM: 40 GB
• Number of GPUs: 8
• GPU Make: Nvidia

Additional context
Is there a way to reduce the memory requirement ? Most obvious trick, reducing batch size, did not prevent OOM.

[whatdhack]

What is the best way to adapt the 8 checkpoints for A100-80GB/H100 for the 70B model to say 16 A100-40GB ?

[subramen]

Please see this thread: #157 (comment)