Large datasets and QLoRA

[Katehuuh]

In QLoRA Training with large datasets~300GB, and given dataset i use

--lora_rank 256 --lora_alpha 512

--lora_target all --learning_rate 2e-05 --num_train_epochs 1.0 --max_samples 100000 --warmup_steps 1000  --lora_dropout 0.15 --visual_inputs True 

The log loss ranges from 1.0 to 1.5. Is that reasonable LR + arg for the datasets?

[hiyouga]

The learning ability of QLoRA is limited, increasing lora rank may not work, consider use ReLoRA or init multiple adapters

[Katehuuh]

ReLoRA

Dataset is not pretraining but QA-sft and Is QLoRA + ReLoRA possible? if not, propably OOM 😅

[hiyouga]

I think it is compatible

[Katehuuh]

But LLaMA-Factory does not include ReLoRA? on 24GB only QLoRa can train llava-13B, (If it was 7B I could of use BAdam/GaLore)

[Katehuuh]

QLoRa did not work But given this LoRA https://huggingface.co/liuhaotian/llava-v1.5-13b-lora it look like possible?

[Katehuuh]

@hiyouga If QLoRA is limited, Can BAdam Could handle such Large dataset?