I would like to know about the behavior of the server's Parallel option. Doesn't processing in parallel = increase GPU usage?

[Taikono-Himazin]

I would like to know about the behavior of the server's Parallel option. Doesn't processing in parallel = increase GPU usage?

Suppose that the behavior without the parallel option is as follows.

I thought that adding the parallel option would work as follows.

But in reality it seems to work like this.

Even with the parallel option, GPU usage does not increase. Throughput has also increased only slightly. (On my server, GPU usage stays at around 10%)

Is my understanding correct?

[Taikono-Himazin]

Is it possible that parallel only works in streaming mode?

[phymbert]

No it does not increase the VRAM as each sequence uses a part of the unified KV cache. i.e n_slot_ctx = n_ctx / parallel.

It works well also with streaming enabled.

Have you faced any issue ?

[Taikono-Himazin]

I would like to increase throughput.
Current GPU usage is around 10%.
So, I tried to perform 10 parallel processes and get 10 times the current throughput. For this purpose, I have set the following options:
--Parallel 10 -cb
But that hasn't actually happened. Why?

[phymbert]

Please share how the steps you have followed, from HuggingFace to the server config and http requests you are sending.

You can use the following discussion as a reference, on how to configure the server for performances

• LLM inference server performances comparison llama.cpp / TGI / vLLM #6730

[Taikono-Himazin]

Thank you for the information! That helps a lot!
I'll check it now.
Does this mean that the llama.cpp server is not suitable if performance is required?

[phymbert]

not at all, performances are really good on small devices, or on metal, cpu backends. But the CUDA backend is currently in performance improvment devlopment, see for example:

• ggml : add Flash Attention #5021
• New optimization from NVIDIA to use CUDA Graphs in llama.cpp #6763

The server is kind of production ready:

• `server` production readiness #6398