Causal LM finetuning

[harry7171]

Hey,

Great to see LISA implemented here.

As for the background, I am trying to finetune models with LORA other techniques on domain data but the Task i am doing is Causal LM is Next word Prediction.

Earlier I have been using DataCollator from Hugging face to convert the data into required format and use Trainer to train the model using LORA.

I have been quite curious for LISA , and wanted to do the same with LISA.
I am not getting the right process to convert my text into CausalLM data type and then use that to Finetune with LISA technique.

Any assistance/examples or references would be of great help.

[research4pan]

Thanks for your interest in LMFlow! You may try text-only dataset format (https://optimalscale.github.io/LMFlow/examples/DATASETS.html#textonly), which is natively supported in LMFlow. Intuitively it is doing next-word prediction for the whole sequence, and LMFlow will automatically check the type and do the causal LM next word prediction.

Other code will stay the same as examples provided in README. If you encountered any further problems, please feel free to let us know 😄

[harry7171]

@research4pan Thanks for the quick reply. I would definitely try it out and let know in case of any issues.

BTW just out of curiosity , I havent checked the code yet but are you using Data collator for CausalLM (DataCollatorForLanguageModeling where mlm is False) for data prep as its pretty good for trainer and Causal LM data prep.

Thanks again for the replies

[research4pan]

Thanks for your valuable suggestion! We use default data collator for huggingface Trainer:

• https://github.com/OptimalScale/LMFlow/blob/main/src/lmflow/pipeline/finetuner.py#L367
• -> https://github.com/OptimalScale/LMFlow/blob/main/src/lmflow/pipeline/finetuner.py#L298
• -> https://github.com/huggingface/transformers/blob/main/src/transformers/data/data_collator.py#L74
• -> https://github.com/huggingface/transformers/blob/main/src/transformers/data/data_collator.py#L127

But I think this is indeed a great suggestion! We welcome any contributions that help improves LMFlow. If it is convenient on your end, you may create a PR to improve the default behavior of LMFlow. We will quickly review it once it passed the test.

Or we can schedule the support this feature in our roadmaps and implement it on our end. Thanks for your advice 😄