Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ChatGLM3训练时报错TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType #127

Open
AILWQ opened this issue Jan 6, 2024 · 14 comments
Open

ChatGLM3训练时报错TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType #127

AILWQ opened this issue Jan 6, 2024 · 14 comments

Comments

@AILWQ
Copy link

AILWQ commented Jan 6, 2024

运行脚本如下：

deepspeed --include localhost:6 --master_port 520 train.py \
                --train_path data/spo_0.json \
                --model_name_or_path ChatGLM3-6B \
                --per_device_train_batch_size 1 \
                --max_len 1560 \
                --max_src_len 1024 \
                --learning_rate 1e-4 \
                --weight_decay 0.1 \
                --num_train_epochs 2 \
                --gradient_accumulation_steps 4 \
                --warmup_ratio 0.1 \
                --mode glm3 \
                --lora_dim 16 \
                --lora_alpha 64 \
                --lora_dropout 0.1 \
                --lora_module_name "query_key_value,dense_h_to_4h,dense_4h_to_h,dense" \
                --seed 1234 \
                --ds_file ds_zero2_no_offload.json \
                --gradient_checkpointing \
                --show_loss_step 10 \
                --output_dir ./output-glm3

报错如下：

[2024-01-07 00:44:07,189] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-07 00:44:10,270] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [6]}
[2024-01-07 00:44:10,271] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=1, node_rank=0
[2024-01-07 00:44:10,271] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2024-01-07 00:44:10,271] [INFO] [launch.py:163:main] dist_world_size=1
[2024-01-07 00:44:10,271] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=6
[2024-01-07 00:44:12,382] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
  File "/home/wangyingli/liwenqiang/ChatGLM-Finetuning/train.py", line 22, in <module>
    from utils import print_trainable_parameters, print_rank_0, to_device, set_random_seed, save_model
  File "/home/wangyingli/liwenqiang/ChatGLM-Finetuning/utils.py", line 15, in <module>
    from transformers import set_seed
  File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 16, in <module>
    from .utils.versions import require_version, require_version_core
  File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/utils/__init__.py", line 18, in <module>
    from huggingface_hub import get_full_repo_name  # for backward compatibility
  File "<frozen importlib._bootstrap>", line 1055, in _handle_fromlist
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/__init__.py", line 357, in __getattr__
    submod = importlib.import_module(submod_path)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 49, in <module>
    from huggingface_hub.utils import (
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/__init__.py", line 45, in <module>
    from ._headers import build_hf_headers, get_token_to_send, LocalTokenNotFoundError
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_headers.py", line 20, in <module>
    from ._runtime import (
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_runtime.py", line 62, in <module>
    _package_versions[candidate_name] = importlib.metadata.version(name)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/metadata.py", line 569, in version
    return distribution(distribution_name).version
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/metadata.py", line 542, in distribution
    return Distribution.from_name(distribution_name)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/metadata.py", line 192, in from_name
    dist = next(iter(dists), None)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 739, in <genexpr>
    path.search(prepared) for path in map(FastPath, paths)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 631, in search
    return self.lookup(self.mtime).search(name)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 636, in mtime
    return os.stat(self.root).st_mtime
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

麻烦聪哥看下这是什么问题导致的，弄一晚上了没解决
@liucongg
Copy link
Owner

liucongg commented Jan 7, 2024

你应该是模型目录（ChatGLM3-6B）里缺东西

@liucongg
Copy link
Owner

liucongg commented Jan 7, 2024

确保ChatGLM3-6B文件夹中，模型文件、tokenizer文件都是全的

@AILWQ
Copy link
Author

AILWQ commented Jan 7, 2024

确保ChatGLM3-6B文件夹中，模型文件、tokenizer文件都是全的

我重新下载了一遍huggingface上的文件，应该是没问题的，还是报原来的错误
image

@liucongg
Copy link
Owner

liucongg commented Jan 7, 2024

你的transformer版本是多少

@AILWQ
Copy link
Author

AILWQ commented Jan 7, 2024

你的transformer版本是多少

4.36.2

@liucongg
Copy link
Owner

liucongg commented Jan 7, 2024

确保ChatGLM3-6B文件夹中，模型文件、tokenizer文件都是全的

我重新下载了一遍huggingface上的文件，应该是没问题的，还是报原来的错误 image

按照你的截图，貌似你的执行命令不对，应该--model_name_or_path models/ChatGLM3-6B

@liucongg
Copy link
Owner

liucongg commented Jan 7, 2024

你的transformer版本是多少

4.36.2

可以切到4.33试一下

@AILWQ
Copy link
Author

AILWQ commented Jan 7, 2024

确保ChatGLM3-6B文件夹中，模型文件、tokenizer文件都是全的

我重新下载了一遍huggingface上的文件，应该是没问题的，还是报原来的错误 image

按照你的截图，貌似你的执行命令不对，应该--model_name_or_path models/ChatGLM3-6B

原来那个命令的路径是之前模型的路径，我刚才在models里又下载了一份新的哈哈，路径应该没问题的

@AILWQ
Copy link
Author

AILWQ commented Jan 7, 2024

你的transformer版本是多少

4.36.2

可以切到4.33试一下

好的我试下

@AILWQ
Copy link
Author

AILWQ commented Jan 7, 2024

你的transformer版本是多少

4.36.2

可以切到4.33试一下

切到4.33还是一样的问题，太奇怪了哈哈

@AILWQ
Copy link
Author

AILWQ commented Jan 7, 2024

你的transformer版本是多少

4.36.2

可以切到4.33试一下

感觉不是路径的问题，我用绝对路径也报错

@liucongg
Copy link
Owner

liucongg commented Jan 7, 2024

你的transformer版本是多少

4.36.2

可以切到4.33试一下

切到4.33还是一样的问题，太奇怪了哈哈

你的环境配置是多少，可以按照requestments配置一下

@liucongg
Copy link
Owner

liucongg commented Jan 7, 2024

运行脚本如下：

deepspeed --include localhost:6 --master_port 520 train.py \
                --train_path data/spo_0.json \
                --model_name_or_path ChatGLM3-6B \
                --per_device_train_batch_size 1 \
                --max_len 1560 \
                --max_src_len 1024 \
                --learning_rate 1e-4 \
                --weight_decay 0.1 \
                --num_train_epochs 2 \
                --gradient_accumulation_steps 4 \
                --warmup_ratio 0.1 \
                --mode glm3 \
                --lora_dim 16 \
                --lora_alpha 64 \
                --lora_dropout 0.1 \
                --lora_module_name "query_key_value,dense_h_to_4h,dense_4h_to_h,dense" \
                --seed 1234 \
                --ds_file ds_zero2_no_offload.json \
                --gradient_checkpointing \
                --show_loss_step 10 \
                --output_dir ./output-glm3

报错如下：

[2024-01-07 00:44:07,189] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-07 00:44:10,270] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [6]}
[2024-01-07 00:44:10,271] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=1, node_rank=0
[2024-01-07 00:44:10,271] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2024-01-07 00:44:10,271] [INFO] [launch.py:163:main] dist_world_size=1
[2024-01-07 00:44:10,271] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=6
[2024-01-07 00:44:12,382] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
  File "/home/wangyingli/liwenqiang/ChatGLM-Finetuning/train.py", line 22, in <module>
    from utils import print_trainable_parameters, print_rank_0, to_device, set_random_seed, save_model
  File "/home/wangyingli/liwenqiang/ChatGLM-Finetuning/utils.py", line 15, in <module>
    from transformers import set_seed
  File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 16, in <module>
    from .utils.versions import require_version, require_version_core
  File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/utils/__init__.py", line 18, in <module>
    from huggingface_hub import get_full_repo_name  # for backward compatibility
  File "<frozen importlib._bootstrap>", line 1055, in _handle_fromlist
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/__init__.py", line 357, in __getattr__
    submod = importlib.import_module(submod_path)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 49, in <module>
    from huggingface_hub.utils import (
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/__init__.py", line 45, in <module>
    from ._headers import build_hf_headers, get_token_to_send, LocalTokenNotFoundError
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_headers.py", line 20, in <module>
    from ._runtime import (
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_runtime.py", line 62, in <module>
    _package_versions[candidate_name] = importlib.metadata.version(name)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/metadata.py", line 569, in version
    return distribution(distribution_name).version
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/metadata.py", line 542, in distribution
    return Distribution.from_name(distribution_name)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/metadata.py", line 192, in from_name
    dist = next(iter(dists), None)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 739, in <genexpr>
    path.search(prepared) for path in map(FastPath, paths)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 631, in search
    return self.lookup(self.mtime).search(name)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 636, in mtime
    return os.stat(self.root).st_mtime
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

麻烦聪哥看下这是什么问题导致的，弄一晚上了没解决

File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/utils/init.py", line 18, in
from huggingface_hub import get_full_repo_name # for backward compatibility
File "", line 1055, in _handle_fromlist
File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/init.py", line 357, in getattr
submod = importlib.import_module(submod_path)
File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 49, in
from huggingface_hub.utils import (
File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/init.py", line 45, in
from ._headers import build_hf_headers, get_token_to_send, LocalTokenNotFoundError
File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_headers.py", line 20, in
from ._runtime import (
File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_runtime.py", line 62, in
_package_versions[candidate_name] = importlib.metadata.version(name)

通过这几行的报错，感觉就是transformers通过repo_id再找模型

@AILWQ
Copy link
Author

AILWQ commented Jan 8, 2024

运行脚本如下：

deepspeed --include localhost:6 --master_port 520 train.py \
                --train_path data/spo_0.json \
                --model_name_or_path ChatGLM3-6B \
                --per_device_train_batch_size 1 \
                --max_len 1560 \
                --max_src_len 1024 \
                --learning_rate 1e-4 \
                --weight_decay 0.1 \
                --num_train_epochs 2 \
                --gradient_accumulation_steps 4 \
                --warmup_ratio 0.1 \
                --mode glm3 \
                --lora_dim 16 \
                --lora_alpha 64 \
                --lora_dropout 0.1 \
                --lora_module_name "query_key_value,dense_h_to_4h,dense_4h_to_h,dense" \
                --seed 1234 \
                --ds_file ds_zero2_no_offload.json \
                --gradient_checkpointing \
                --show_loss_step 10 \
                --output_dir ./output-glm3

报错如下：

[2024-01-07 00:44:07,189] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-07 00:44:10,270] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [6]}
[2024-01-07 00:44:10,271] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=1, node_rank=0
[2024-01-07 00:44:10,271] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2024-01-07 00:44:10,271] [INFO] [launch.py:163:main] dist_world_size=1
[2024-01-07 00:44:10,271] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=6
[2024-01-07 00:44:12,382] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
  File "/home/wangyingli/liwenqiang/ChatGLM-Finetuning/train.py", line 22, in <module>
    from utils import print_trainable_parameters, print_rank_0, to_device, set_random_seed, save_model
  File "/home/wangyingli/liwenqiang/ChatGLM-Finetuning/utils.py", line 15, in <module>
    from transformers import set_seed
  File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 16, in <module>
    from .utils.versions import require_version, require_version_core
  File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/utils/__init__.py", line 18, in <module>
    from huggingface_hub import get_full_repo_name  # for backward compatibility
  File "<frozen importlib._bootstrap>", line 1055, in _handle_fromlist
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/__init__.py", line 357, in __getattr__
    submod = importlib.import_module(submod_path)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 49, in <module>
    from huggingface_hub.utils import (
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/__init__.py", line 45, in <module>
    from ._headers import build_hf_headers, get_token_to_send, LocalTokenNotFoundError
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_headers.py", line 20, in <module>
    from ._runtime import (
  File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_runtime.py", line 62, in <module>
    _package_versions[candidate_name] = importlib.metadata.version(name)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/metadata.py", line 569, in version
    return distribution(distribution_name).version
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/metadata.py", line 542, in distribution
    return Distribution.from_name(distribution_name)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/metadata.py", line 192, in from_name
    dist = next(iter(dists), None)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 739, in <genexpr>
    path.search(prepared) for path in map(FastPath, paths)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 631, in search
    return self.lookup(self.mtime).search(name)
  File "/home/wangyingli/.conda/envs/competition/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 636, in mtime
    return os.stat(self.root).st_mtime
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

麻烦聪哥看下这是什么问题导致的，弄一晚上了没解决

File "/home/wangyingli/.local/lib/python3.9/site-packages/transformers/utils/init.py", line 18, in from huggingface_hub import get_full_repo_name # for backward compatibility File "", line 1055, in _handle_fromlist File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/init.py", line 357, in getattr submod = importlib.import_module(submod_path) File "/home/wangyingli/.conda/envs/competition/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 49, in from huggingface_hub.utils import ( File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/init.py", line 45, in from ._headers import build_hf_headers, get_token_to_send, LocalTokenNotFoundError File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_headers.py", line 20, in from ._runtime import ( File "/home/wangyingli/.local/lib/python3.9/site-packages/huggingface_hub/utils/_runtime.py", line 62, in _package_versions[candidate_name] = importlib.metadata.version(name)

通过这几行的报错，感觉就是transformers通过repo_id再找模型

环境按照requirments.txt重新装了一遍，并且换成GLM2也还是不行，报同样的错误。。。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@liucongg @AILWQ