Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metric: tgi_request_total increments by 2 upon every request #1901

Open
thenu97 opened this issue May 15, 2024 · 0 comments
Open

metric: tgi_request_total increments by 2 upon every request #1901

thenu97 opened this issue May 15, 2024 · 0 comments

Comments

@thenu97
Copy link

thenu97 commented May 15, 2024

System Info

2024-05-15T16:10:20.104786Z  INFO text_generation_launcher: Runtime environment:
Target: x86_64-unknown-linux-gnu
Cargo version: 1.75.0
Commit sha: bf700e7eef4771f280c19dbc7270c8c7c20efbbc
Docker label: N/A
nvidia-smi:
Wed May 15 16:10:19 2024       
   +---------------------------------------------------------------------------------------+
   | NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
   |-----------------------------------------+----------------------+----------------------+
   | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
   |                                         |                      |               MIG M. |
   |=========================================+======================+======================|
   |   0  NVIDIA A10G                    On  | 00000000:00:1E.0 Off |                    0 |
   |  0%   34C    P0              60W / 300W |  19020MiB / 23028MiB |      0%      Default |
   |                                         |                      |                  N/A |
   +-----------------------------------------+----------------------+----------------------+
                                                                                            
   +---------------------------------------------------------------------------------------+
   | Processes:                                                                            |
   |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
   |        ID   ID                                                             Usage      |
   |=======================================================================================|
   |    0   N/A  N/A      4235      C   ...untu/miniconda3/envs/tgi/bin/python    19008MiB |
   +---------------------------------------------------------------------------------------+

### Information

- [ ] Docker
- [ ] The CLI directly

### Tasks

- [X] An officially supported command
- [ ] My own modifications

### Reproduction

1. send a request to `chat_completions()`
2. increments request_total_count by 2

### Expected behavior

`chat_completions()` itself should increment the counter by 1. its subsequent func calls (within chat_completions()) shouldn't increment the counter
`chat_completions()` calls functions `generate` or `generate_stream_internal` depending on if steam is enabled and both these functions further increments the counter, hence incrementing the counter by 2 for every request 
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant