How to make predictions deterministic?

[abdulmeral]

Hello everyone,
how to prevent it from changing new predictions after each training?
Of course it is a LLM, but it is problem that changing every time. for example; my mape value is changing rapidly, bad or better..
have any recommend?

thank you already now.
abdül

[lostella]

@abdulmeral since the models are based on PyTorch, you can refer to PyTorch documentation about reproducibility.

In particular, setting the random number generation seed with

import torch
torch.manual_seed(0) # or some other number

before doing predictions, should make the behavior deterministic.

Another option to stabilize predictions (at least the "central tendencies" like mean and median) is to increase the number of samples by passing in num_samples when calling .predict (default is 20): this however will come at the cost of slower predictions.

[abdulfatir]

@lostella I think it maybe better to use transformers.set_seed for seeding. It seeds everything under the sun, so you will have consistent results.

[abdulmeral]

thank you very much guys. both ideas are working.
and also my colleague recommends that making predictions with different seed numbers so we can get more generic results.

[abdulfatir]

@abdulmeral glad that it helped. On another note, you should probably stop using MAPE because it's not a good metric. Checkout something like MASE, if you're working with point forecasts.

[lostella]

@lostella I think it maybe better to use transformers.set_seed for seeding. It seeds everything under the sun, so you will have consistent results.

TIL