I don't think this is quite the right approach because I don't think it allows for stopping/restarting which is part of the point of checkpointing. Instead, the checkpointer should store the LoRA'd model, not the pretrained model. Ideally it would even only store the LoRA weights (and any updated weights) and not the whole model, making for very small checkpoints and faster saving. Example:

add_adapters: !name:speechbrain.lobes.models.Adapters.add_adapters_to_linear_in_model
  adapter_class: !name:speechbrain.lobes.models.Adapters.HoulsbyAdapterLinear
  projection_size: 32

pretrainer: !new:speechbrain....Pretrainer
  loadables:
    transformer: !ref <Transformer>

# Load the pretrained model.
run_on_main(hparams["pretrainer"].collect_files)
hparams["pretrainer"].load_collected()
hparams["add_adapters"](hparams["Transformer"])

asr_brain = ASR(
    modules=hparams["modules"],
    opt_class=hparams["Adam"],
    hparams=hparams,
    run_opts=run_opts,
    checkpointer=hparams["checkpointer"], # Checkpointer loads LoRA weights only and applies them
)

asr_brain.fit(
    asr_brain.hparams.epoch_counter, train_data, valid_data,
)

It does allow for stop and restart because you are altering the object i.e. the checkpointer keeps track of it! The only problem is indeed that you store the whole model, however, I don't think it's an issue because in-fine you may simply don't know where to put the pre-trained adapters in the model if they are not applied to every linear layer for instance. I'd be happy to see a functionnal example of something else though.

Tbh I think PEFT handles this perfectly, perhaps we should lift their code wholesale.

You mean depend on another Huggingface library?

My opinion is we should just add it as a dependency, but I understand the objections to it. So instead we could just copy the parts of the code that make sense into speechbrain.

If you could give me a neat example of an integration of PEFT, I could be convinced.

The problem with peft is when we want to load the model from the speechbrain checkpoint. It is a mess to make it work and also it could cause the problem when using different version of peft. But maybe we could find a way to do this in a cleaner way.