The transformation of the model Blip2ForConditionalGeneration to BetterTransformer failed

[garyzhang99]

System Info

optimum == 1.19.1, python == 3.8.17, centos

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)

import transformers
from transformers import AutoConfig

config = AutoConfig.from_pretrained('Salesforce/blip2-opt-2.7b')
class_name = config.architectures[0]
model_type = config.model_type
model_class = getattr(transformers, class_name)
model  = model_class.from_pretrained("Salesforce/blip2-opt-2.7b")

from optimum.bettertransformer.models import BetterTransformerManager
if BetterTransformerManager.supports(model_type):
    model = model.to_bettertransformer()

File ~/miniconda3/envs/dj/lib/python3.8/site-packages/transformers/modeling_utils.py:4132, in PreTrainedModel.to_bettertransformer(self)
   4126     raise ImportError(
   4127         f"Please install optimum>=1.7.0 to use Better Transformer. The version {optimum_version} was found."
   4128     )
   4130 from optimum.bettertransformer import BetterTransformer
-> 4132 return BetterTransformer.transform(self)

File ~/miniconda3/envs/dj/lib/python3.8/contextlib.py:75, in ContextDecorator.__call__.<locals>.inner(*args, **kwds)
     72 @wraps(func)
     73 def inner(*args, **kwds):
     74     with self._recreate_cm():
---> 75         return func(*args, **kwds)

File ~/miniconda3/envs/dj/lib/python3.8/site-packages/optimum/bettertransformer/transformation.py:270, in BetterTransformer.transform(model, keep_original_model, max_memory, offload_dir, **kwargs)
    267     model = None
    269 if BetterTransformerManager.requires_nested_tensor(model_fast.config.model_type):
--> 270     set_last_layer(model_fast)
    272 # Add a class arguments, we might need to identify whether the model
    273 # has been correctly converted to its `BetterTransformer` version.
    274 setattr(model_fast, "use_bettertransformer", True)

File ~/miniconda3/envs/dj/lib/python3.8/site-packages/optimum/bettertransformer/transformation.py:166, in set_last_layer(model)
    163             setattr(dict_named_module[key][-1], "is_last_layer", True)
    164             return
--> 166 raise Exception(
    167     f"The transformation of the model {model.__class__.__name__} to BetterTransformer failed while it should not. Please fill"
    168     " a bug report or open a PR to support this model at https://github.com/huggingface/optimum/"
    169 )

Exception: The transformation of the model Blip2ForConditionalGeneration to BetterTransformer failed while it should not. Please fill a bug report or open a PR to support this model at https://github.com/huggingface/optimum/

Expected behavior

Since the model_type is 'blip-2' and is supported in #1125, I expected that the model could be converted. But the Exception Occured:

File ~/miniconda3/envs/dj/lib/python3.8/site-packages/transformers/modeling_utils.py:4132, in PreTrainedModel.to_bettertransformer(self)
   4126     raise ImportError(
   4127         f"Please install optimum>=1.7.0 to use Better Transformer. The version {optimum_version} was found."
   4128     )
   4130 from optimum.bettertransformer import BetterTransformer
-> 4132 return BetterTransformer.transform(self)

File ~/miniconda3/envs/dj/lib/python3.8/contextlib.py:75, in ContextDecorator.__call__.<locals>.inner(*args, **kwds)
     72 @wraps(func)
     73 def inner(*args, **kwds):
     74     with self._recreate_cm():
---> 75         return func(*args, **kwds)

File ~/miniconda3/envs/dj/lib/python3.8/site-packages/optimum/bettertransformer/transformation.py:270, in BetterTransformer.transform(model, keep_original_model, max_memory, offload_dir, **kwargs)
    267     model = None
    269 if BetterTransformerManager.requires_nested_tensor(model_fast.config.model_type):
--> 270     set_last_layer(model_fast)
    272 # Add a class arguments, we might need to identify whether the model
    273 # has been correctly converted to its `BetterTransformer` version.
    274 setattr(model_fast, "use_bettertransformer", True)

File ~/miniconda3/envs/dj/lib/python3.8/site-packages/optimum/bettertransformer/transformation.py:166, in set_last_layer(model)
    163             setattr(dict_named_module[key][-1], "is_last_layer", True)
    164             return
--> 166 raise Exception(
    167     f"The transformation of the model {model.__class__.__name__} to BetterTransformer failed while it should not. Please fill"
    168     " a bug report or open a PR to support this model at https://github.com/huggingface/optimum/"
    169 )

Exception: The transformation of the model Blip2ForConditionalGeneration to BetterTransformer failed while it should not. Please fill a bug report or open a PR to support this model at https://github.com/huggingface/optimum/