Small dataset training

[raymond-infinitecode]

Hi Community,

I've a small sets of data e.g. 10 items in the json dataset.
The training completed very fast,. However, I couldn't get anything reflected.

set CUDA_VISIBLE_DEVICES=0
set USE_MODELSCOPE_HUB=0
python src/train_bash.py --stage sft --do_train --model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 --dataset myown_data_en --template default --finetuning_type lora --lora_target q_proj,v_proj --output_dir m_checkpoints --overwrite_cache --per_device_train_batch_size 1 --gradient_accumulation_steps 4 --lr_scheduler_type cosine --logging_steps 10 --save_steps 1000 --learning_rate 5e-5 --num_train_epochs 3.0 --plot_loss --overwrite_output_dir --bf16

Is anything I did wrong in the arguments ?

[hiyouga]

increase your training steps by using larger num_epoches, until the training loss is lower than 0.1

[raymond-infinitecode]

Really grateful for the answer, it is exactly the case.
python src/train_bash.py --stage sft --do_train --model_name_or_path mistralai/Mistral-7B-Instruct-v0.2 --dataset myown_data_en --template default --finetuning_type lora --lora_target q_proj,v_proj --output_dir m_checkpoints --overwrite_cache --per_device_train_batch_size 1 --gradient_accumulation_steps 4 --lr_scheduler_type cosine --logging_steps 10 --save_steps 1000 --learning_rate 5e-5 --num_train_epochs 50.0 --plot_loss --overwrite_output_dir --bf16

Revised epoc to 50 to make the training loss lower than 0.1