Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support mistral and llava_mistral in turbomind #1579

Merged
merged 8 commits into from
May 27, 2024
Merged

support mistral and llava_mistral in turbomind #1579

merged 8 commits into from
May 27, 2024

Conversation

lvhan028
Copy link
Collaborator

@lvhan028 lvhan028 commented May 10, 2024 •
edited

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

resolve #1573

Note that slide window attention is not supported yet.

  • support llava-mistral

Tests

from lmdeploy import pipeline, TurbomindEngineConfig
from lmdeploy.vl import load_image

pipe = pipeline('/workspace/lmdeploy/workspace/llava-v1.6-mistral-7b',
                backend_config=TurbomindEngineConfig(session_len=8192, tp=1),
                log_level='INFO')

image_urls=['workspace/Chongqing_Small.jpeg', 'workspace/Shanghai_Small.jpeg']

images = [load_image(img_url) for img_url in image_urls]
response = pipe(('describe the two pictures in detail respectively', images))
print(response)

@irexyc irexyc mentioned this pull request May 10, 2024
Closed
2 tasks
@lvhan028 lvhan028 removed the WIP label May 11, 2024
@lvhan028 lvhan028 changed the title support mistral in turbomind engine support mistral and llava_mistral May 11, 2024
@lvhan028 lvhan028 changed the title support mistral and llava_mistral support mistral and llava_mistral in turbomind May 11, 2024
@lvhan028 lvhan028 requested review from irexyc and AllentDan May 11, 2024 09:34
@lvhan028 lvhan028 mentioned this pull request May 13, 2024
Closed
AllentDan
AllentDan approved these changes May 16, 2024
View reviewed changes
Copy link
Collaborator

@AllentDan AllentDan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested OK with pipeline.

@AllentDan
Copy link
Collaborator

May resolve conflicts.

@AllentDan
Copy link
Collaborator

What is the merge plan? since it has conflicts with the other two VL PRs

@lvhan028
Copy link
Collaborator Author

After other PRs merged

@AllentDan
Copy link
Collaborator

Auto backend UT failed.

irexyc
irexyc reviewed May 27, 2024
View reviewed changes
lmdeploy/vl/model/llava.py Outdated Show resolved Hide resolved
@irexyc
Copy link
Collaborator

irexyc commented May 27, 2024

lmdeploy/lmdeploy/model.py

Lines 294 to 296 in 7758151

path = model_path.lower()
if 'llava' in path and 'v1' in path and 'v1.6-34b' not in path:
return 'llava-v1'

should add mistral not in path

@irexyc
Copy link
Collaborator

irexyc commented May 27, 2024

多轮对话时，模版跟llava有一些不一样，不确定是否重要。

lmdeploy:
[INST] <IMAGE_TOKEN>\nq1 [/INST]a1</s>[INST] q2 [/INST]
llava:
[INST] <IMAGE_TOKEN>\nq1 [/INST] a1 </s>[INST] q2 [/INST]

https://github.com/haotian-liu/LLaVA/blob/main/llava/conversation.py#L350-L359
https://github.com/haotian-liu/LLaVA/blob/main/llava/conversation.py#L90

@lvhan028
Copy link
Collaborator Author

多轮对话时，模版跟llava有一些不一样，不确定是否重要。

lmdeploy: [INST] <IMAGE_TOKEN>\nq1 [/INST]a1</s>[INST] q2 [/INST] llava: [INST] <IMAGE_TOKEN>\nq1 [/INST] a1 </s>[INST] q2 [/INST]

https://github.com/haotian-liu/LLaVA/blob/main/llava/conversation.py#L350-L359 https://github.com/haotian-liu/LLaVA/blob/main/llava/conversation.py#L90

I found in the interactive inference mode, the model generate " " at the end of its answer.
So, I think we can use lmdeploy's chat template if users don't strip the blanks of the answer.

@lvhan028 lvhan028 merged commit e6468e7 into InternLM:main May 27, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@irexyc irexyc irexyc approved these changes

@AllentDan AllentDan AllentDan approved these changes

Assignees
No one assigned
Labels
improvement
Projects
None yet
Milestone
No milestone
3 participants
@lvhan028 @AllentDan @irexyc