LoRA: Extract small function

[madroidmaq]

Extract the load_with_lora function for easy use in other files.

[mzbac]

Maybe we should consider moving the function out of utils.py. It has becomed too large now, it might be better to create a new file named lora_utils.py and place it there?

[madroidmaq]

@mzbac Great suggestion. Initially, I tried placing the load_with_lora function in both tuner/lora.py and tuner/utils.py, but encountered a circular dependency issue. When I attempted to resolve this issue, I found that it required significant changes. So, I ended up putting it in the utils.py.

I will reassess the extent of changes needed to solve the circular dependency problem. The worst-case scenario would be creating a new file named tuner/lora_utils.py, which I will try my best to avoid.

[madroidmaq]

The main reason for the circular dependency is the reliance on the load() function in utils.py, which in turn depends on the linear_to_lora_layers function in tuner/utils.py.

I have now removed the logic of actively loading models within functions and instead accept an already loaded model. This can avoid issues with sequential dependencies. Meanwhile, it will be necessary to first load the model and then call the prepare_for_training function when using it, which I think is acceptable.

[awni]

@madroidmaq sorry for the delay! I think we can merge this. But could you first rebase to resolve conflicts?

[madroidmaq]

@madroidmaq sorry for the delay! I think we can merge this. But could you first rebase to resolve conflicts?

@awni It's DONE.

[awni]

Thanks!