How to speed up the model?

[VlaTal1]

I'm using the TheBloke/WizardCoder-15B-1.0-GPTQ model and the whole model can fit into the graphics card (3090TI 24GB if that matters), but the model works very slow. A request can be processed for about a minute, although the exact same request is processed by TheBloke/WizardLM-13B-V1.1-GGML model for about 30 seconds. But difference between them not very big to work so slowly.

Is there any way to speed up the model?

[TheBloke]

That's definitely not normal on a 3090Ti. You should be getting 20-30 tokens/s at least.

My first thought is that maybe your AutoGPTQ CUDA extension is not compiled.

Can you please show the full output you see when doing inference, including all messages printed by AutoGPTQ.

[pascalc]

I've set device_map=0 which I believe puts everything on GPU 0 and it still happens.

RAM usage first goes up by ~9GB then drops, then VRAM usage goes up and stays at ~9GB, which I believe is the model passing through RAM to the GPU.

Then CPU usage stays at 100% for a long time, after which the output is coherent, so everything works, but it feels like some CPU operation is bottlenecking the whole thing.

My environment:

Ubuntu 22.04
Nvidia 3090
Python 3.10.12
torch 2.0.1+cu118
triton 2.0.0

AutoGPTQ: I installed this wheel: https://github.com/PanQiWei/AutoGPTQ/releases/download/v0.3.0/auto_gptq-0.3.0+cu118-cp310-cp310-linux_x86_64.whl

And ran the code:

from transformers import AutoTokenizer, pipeline, logging
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_name_or_path = "../llm/gptq/wizardcoder-guanaco-15b"
model_basename = "gptq_model-4bit-128g"

use_triton = True

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)

model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
        model_basename=model_basename,
        use_safetensors=True,
        device_map=0,
        trust_remote_code=False,
        use_triton=use_triton,
        quantize_config=None)

prompt = "Tell me about AI"
prompt_template=f'''Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction: {prompt}

### Response:
'''

print("\n\n*** Generate:")
input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
print(tokenizer.decode(output[0]))

[VlaTal1]

I am nor an expert, but this is from HuggingFace docs:
You can let 🤗 Accelerate handle the device map computation by setting device_map to one of the supported options ("auto", "balanced", "balanced_low_0", "sequential") or create one yourself if you want more control over where each layer should go.

So maybe the problem with your device_map=0?

I had broblems beacause I use use_triton=True and because CUDA extension was not installed. So just try to run it without triton. Maybe you would have some errors

[pascalc]

You were right, I had to both compile from source and disable triton, then it worked! If you have a mismatch between the cuda version from the auto_gptq wheel and your system it seems to silently "fail" and run everything on your cpu, but compiling from source fixes this and I can add that it works against cuda 12.1.

Thanks for your help!

[VlaTal1]

Just in case, make sure you have PyTorch that supports CUDA 12.1. It also can make problems with cuda extention installation

And if you want to use triton, you have to install autoGPTQ like pip install .[triton] in case of souce installation. I have never run with triton, but personally I'm satisfied with the speed without it

[pascalc]

Okay good to know, thanks!

[RichardScottOZ]

I am seeing something along these lines - using a ubuntu deep learning EC2 - some other models ok, now trying TheBloke/Llama-2-13B-chat-GPTQ.

Generation seemed slow, so then I noticed the CUDA extension not installed error here. Don't get it for a vanilla Falcon 7B or Llama 7B for example. Have cuda toolkit etc. 11.7. Then I noticed via nvidia-smi

Is it possible that this being 11.6 is a problem ? ````bas
nvidia-smi
Sun Jul 30 03:35:13 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 |
| 0% 27C P8 16W / 300W | 0MiB / 23028MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

[VlaTal1]

As I know, version here is not matter.Try nvcc --version. there is the "right" version

[TheBloke]

Yes 11.6 is likely causing a problem because AutoGPTQ currently only installs pre-built wheels on 11.7 and 11.8.

Please try building AutoGPTQ from source:

pip3 uninstall -y auto-gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
pip3 install .

[RichardScottOZ]

GPU usage is around 13GB and says volatile util is 100% when I checked while it was generating, so a little confusing.

[RichardScottOZ]

python3 -c 'import torch ; import auto_gptq ; import autogptq_cuda'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'autogptq_cuda'

``

[TheBloke]

Please build AutoGPTQ from source - see my comment to Vlatal above - EDIT: just saw you already did this, good

[RichardScottOZ]

Started another newer machine that has CUDA up to 12.2 and started from scratch

Successfully built auto-gptq
Installing collected packages: auto-gptq
Successfully installed auto-gptq-0.3.2+cu118
(transormers) ubuntu@test:~/AutoGPTQ$ python3 -c 'import torch ; import auto_gptq ; import autogptq_cuda'
(transormers) ubuntu@test:~/AutoGPTQ$

[TheBloke]

Ah good, yes this is all correct.

[RichardScottOZ]

Yes, the 11.6 issue would have been delving into C++ type compilation fun. Quicker to set u a new machine.

[RichardScottOZ]

I had to install from scratch cloning the repo though, a pip install brand new didn't work

[TheBloke]

Yeah this is a problem at the moment. Unless you have CUDA 11.7 or 11.8, you must git clone and install from source

And for some reason, some users have to install from source even if they do have CUDA 11.7 or 11.8

@PanQiWei has said he will look into it as soon as possible

[PanQiWei]

I'm working on integrate xformers into auto-gptq(and paged-attention in a very soon future) so the speed will be improved hopefully, also in the next version there should have more pre-build wheels

[RichardScottOZ]

I went with 11.8 FYI.