You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
so giving a background -
I am using Mistral 7b along with HF trainer for finetuning on domain specific data.
Where the task is CausalLM ie next word prediction.
Using datacollatorfor Causal LM for data prep using context size is 1000 tokens per data point and I have 9k total dataset. which includes 5-10% of Wiki data for mixing it with Domain data for avoiding Catastrophic Forgetting.
Test data is a part of train to make it learn on the specific data
I am utilizing the DynamicLayerActivationCallback from LMFlow in my trainer as Training Callbacks.
lisa_activated_layers- 2 , lisa_interval_steps - 50 epoch 10
for both of the runs the loss starts around 8 ad goes around 5-6 but it goes into plateau . and doesnt come below 5 .
I find it little strange, maybe need other experimentation on -
changing lisa_activated_layers
changing interval steps (I think this can be important factor too)
Also would like to get the idea, whats the ideal or recommended hyperparams for such type of finetuning with around 10K datapoints.
Thanks in Advance
The text was updated successfully, but these errors were encountered:
Thanks for your interest in LMFlow! We have fixed several bugs of the LISA implementation in LMFlow, it would be nice if you could check whether the implementation matches our latest version.
If the implementation is correct, it is worth trying:
A smaller lisa_interval_steps, like 5 instead of 50, since more frequent sampling allows more layers to be covered.
If that doesn't work, you may try larger lisa_activated_layers. We have observed that in some cases, such as llama-2-70b, deeper structure requires larger lisa_activated_layers. This may be also the case when data distribution is not that easy to learn.
Hope this information can be helpful. Please feel free to let us know if further problems are encountered 😄
Hi ,
so giving a background -
I am using Mistral 7b along with HF trainer for finetuning on domain specific data.
Where the task is CausalLM ie next word prediction.
Using datacollatorfor Causal LM for data prep using context size is 1000 tokens per data point and I have 9k total dataset. which includes 5-10% of Wiki data for mixing it with Domain data for avoiding Catastrophic Forgetting.
Test data is a part of train to make it learn on the specific data
I am utilizing the
DynamicLayerActivationCallback
from LMFlow in my trainer asTraining Callbacks.
I tried multiple experiments with -
for both of the runs the loss starts around 8 ad goes around 5-6 but it goes into plateau . and doesnt come below 5 .
I find it little strange, maybe need other experimentation on -
Also would like to get the idea, whats the ideal or recommended hyperparams for such type of finetuning with around 10K datapoints.
Thanks in Advance
The text was updated successfully, but these errors were encountered: