Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

求大佬帮忙看看,为什么社区的微调代码刚执行到保存了一个checkpoint就报错 #308

Open
bapu3212 opened this issue Mar 21, 2024 · 1 comment

Comments

@bapu3212
Copy link

Traceback (most recent call last):
File "/root/autodl-tmp/Llama-Chinese/train/sft/finetune_clm_lora.py", line 692, in
main()
File "/root/autodl-tmp/Llama-Chinese/train/sft/finetune_clm_lora.py", line 653, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 1624, in train
return inner_training_loop(
File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2029, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2423, in _maybe_log_save_evaluate
self._save_checkpoint(model, trial, metrics=metrics)
File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2525, in _save_checkpoint
self.state.save_to_json(os.path.join(staging_output_dir, TRAINER_STATE_NAME))
File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer_callback.py", line 113, in save_to_json
json_string = json.dumps(dataclasses.asdict(self), indent=2, sort_keys=True) + "\n"
File "/root/miniconda3/lib/python3.10/json/init.py", line 238, in dumps
**kw).encode(obj)
File "/root/miniconda3/lib/python3.10/json/encoder.py", line 201, in encode
chunks = list(chunks)
File "/root/miniconda3/lib/python3.10/json/encoder.py", line 431, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "/root/miniconda3/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/root/miniconda3/lib/python3.10/json/encoder.py", line 325, in _iterencode_list
yield from chunks
File "/root/miniconda3/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/root/miniconda3/lib/python3.10/json/encoder.py", line 438, in _iterencode
o = _default(o)
File "/root/miniconda3/lib/python3.10/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.class.name} '
TypeError: Object of type Tensor is not JSON serializable

@t0111
Copy link

t0111 commented Mar 28, 2024

#294 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants