cuda增加，直到溢出报错

[sevenandseven]

Hello, I am using the officially provided method of loading the reranker to perform similarity calculations. During the calculation process, I found that after the cache stabilizes for a period of time, it gradually increases until there is not enough video memory left.
How can I solve this problem?
I tried using （torch.cuda.empty_cache()）this method, but it didn’t work very well, and there was still not enough video memory.

[staoxiao]

@sevenandseven , which reranker do you use?

[sevenandseven]

@sevenandseven , which reranker do you use?

bge-reranker-large、bge-reranker-base、bge-reranker-v2-m3、bge-reranker-v2-gemma、bge-reranker-v2-minicpm-layerwise。
The above models all exhibit this behavior.

[staoxiao]

You can reduce the batch size and max_length to reduce memory cost.

[sevenandseven]

You can reduce the batch size and max_length to reduce memory cost.

I encountered this situation while inference, without the above parameters.

[staoxiao]

@sevenandseven , you can pass batch size and max_length to compute_score(batch size=?, max_length=?) function: https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/flag_reranker.py#L194

[sevenandseven]

@sevenandseven , you can pass batch size and max_length to compute_score(batch size=?, max_length=?) function: https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/flag_reranker.py#L194

ok,thanks。