fsdp-qlora yi-34B-chat throw error " ValueError: Cannot flatten integer dtype tensors"

[hellostronger]

Reminder

• I have read the README and searched the existing issues.

Reproduction

CUDA_VISIBLE_DEVICES=0,1 accelerate launch
--config_file config.yaml
src/train_bash.py
--stage sft
--do_train
--model_name_or_path /workspace/models/Yi-34B-Chat
--dataset law_with_basis
--dataset_dir data
--template default
--finetuning_type lora
--lora_target q_proj,v_proj
--output_dir /workspace/ckpt/Yi-34B-Chat-sft
--overwrite_cache
--overwrite_output_dir
--cutoff_len 1024
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--gradient_accumulation_steps 8
--lr_scheduler_type cosine
--logging_steps 10
--save_steps 100
--eval_steps 100
--evaluation_strategy steps
--load_best_model_at_end
--learning_rate 5e-5
--num_train_epochs 3.0
--max_samples 3000
--val_size 0.1
--quantization_bit 4
--plot_loss
--fp16

config.yaml

compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch: BACKWARD_PRE
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: false
  fsdp_offload_params: true
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: false
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

Expected behavior

fsdp qlora yi-34B-chat

System Info

transformers 4.39.3 torch 2.1.2 cuda 121 python3.8

Others

[hellostronger]

have seen exist issue written in March,but i cannot get any useful info to find out why this error came，hoping your suggestion

[hiyouga]

please provide your version of accelerate and bitsandbytes

[hellostronger]

@hiyouga accelerate==0.28.0 bitsandbytes==0.43.0 ,Do these versions have any problems?hoping your suggestion

[hiyouga]

did you use the latest code?

[etemiz]

While I am trying to train https://huggingface.co/cognitivecomputations/dolphin-2.9-llama3-70b I am getting the same error "ValueError: Cannot flatten integer dtype tensors".
The error seems to be resolved when I reinstalled LLaMA-Factory again. These are the versions:

accelerate 0.29.3
bitsandbytes 0.43.1

[hellostronger]

@hiyouga sorry，my answer is so late this case, using newest llama_factory code, it work currently right now