Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

학습이 되지 않습니다. #219

Open
Seoyoung-Jo opened this issue Feb 1, 2024 · 1 comment
Open

학습이 되지 않습니다. #219

Seoyoung-Jo opened this issue Feb 1, 2024 · 1 comment

Comments

@Seoyoung-Jo
Copy link

❓ Questions & Help

가상 환경 setup 완료 후, 학습을 시작했는데 학습이 제대로 되지 않습니다.
Librispeech, Kospeech에 대해 character, subword, grapheme 전부 시도해보았는데 loss가 떨어지지 않습니다.
오류가 발생하거나 하지 않아 원인을 모르겠습니다.
아래에 실행 코드 및 환경 정보를 적어두겠습니다.
감사합니다.

Details

실행 코드

ASR_DATASET=ksponspeech
DATASET_PATH="DB_folder"
TEST_DATASET_PATH="TestDB_folder"
TEST_MANIFEST_DIR="Scripts_folder"
ASR_TOKENIZER=kspon_subword
MANIFEST_FILE_PATH="${DATASET_PATH}/train_sub.txt"
SP_MODEL_PATH="${DATASET_PATH}/labels_sub.model"

CUDA_VISIBLE_DEVICES=1 HYDRA_FULL_ERROR=1 python3 ./openspeech_cli/hydra_train.py \
    dataset=${ASR_DATASET} \
    +dataset.dataset_download=False \
    dataset.dataset_path=${DATASET_PATH} \
    dataset.manifest_file_path=${MANIFEST_FILE_PATH} \
    dataset.test_dataset_path=${TEST_DATASET_PATH} \
    dataset.test_manifest_dir=${TEST_MANIFEST_DIR} \
    tokenizer=${ASR_TOKENIZER} \
    model=${ASR_MODEL} \
    audio=${ASR_AUDIO} \
    lr_scheduler=warmup_reduce_lr_on_plateau \
    trainer=gpu \
    trainer.batch_size=4 \
    trainer.auto_scale_batch_size=False \
    +save_checkpoint_n_steps=10000 \
    criterion=ctc \
    tokenizer.sp_model_path=${SP_MODEL_PATH}
    # tokenizer.vocab_path=${VOCAB_FILE_PATH}

모델 정보

audio:
  name: fbank
  sample_rate: 16000
  frame_length: 20.0
  frame_shift: 10.0
  del_silence: false
  num_mels: 80
  apply_spec_augment: true
  apply_noise_augment: false
  apply_time_stretch_augment: false
  apply_joining_augment: false
augment:
  apply_spec_augment: false
  apply_noise_augment: false
  apply_joining_augment: false
  apply_time_stretch_augment: false
  freq_mask_para: 27
  freq_mask_num: 2
  time_mask_num: 4
  noise_dataset_dir: None
  noise_level: 0.7
  time_stretch_min_rate: 0.7
  time_stretch_max_rate: 1.4
dataset:
  dataset: ksponspeech
  dataset_path: /ssd1/DB/KsponSpeech
  test_dataset_path: /ssd1/DB
  manifest_file_path: /ssd1/DB/KsponSpeech/train_sub.txt
  test_manifest_dir: /ssd1/DB/KsponSpeech_scripts
  preprocess_mode: phonetic
  dataset_download: false
criterion:
  criterion_name: ctc
  reduction: mean
  zero_infinity: true
lr_scheduler:
  lr: 0.0001
  scheduler_name: warmup_reduce_lr_on_plateau
  lr_patience: 1
  lr_factor: 0.3
  peak_lr: 0.0001
  init_lr: 1.0e-10
  warmup_steps: 4000
model:
  model_name: conformer
  encoder_dim: 512
  num_encoder_layers: 17
  num_attention_heads: 8
  feed_forward_expansion_factor: 4
  conv_expansion_factor: 2
  input_dropout_p: 0.1
  feed_forward_dropout_p: 0.1
  attention_dropout_p: 0.1
  conv_dropout_p: 0.1
  conv_kernel_size: 31
  half_step_residual: true
  optimizer: adam
trainer:
  seed: 1
  accelerator: dp
  accumulate_grad_batches: 1
  num_workers: 4
  batch_size: 4
  check_val_every_n_epoch: 1
  gradient_clip_val: 5.0
  logger: wandb
  max_epochs: 20
  save_checkpoint_n_steps: 10000
  auto_scale_batch_size: 'False'
  sampler: else
  name: gpu
  device: gpu
  use_cuda: true
  auto_select_gpus: true
tokenizer:
  sos_token: <s>
  eos_token: </s>
  pad_token: <pad>
  blank_token: <blank>
  encoding: utf-8
  unit: kspon_subword
  sp_model_path: /ssd1/DB/KsponSpeech/labels_sub.model
  vocab_size: 3200
save_checkpoint_n_steps: 10000

Global seed set to 1
[2024-01-31 14:01:57,940][openspeech.utils][INFO] - audio:
  name: fbank
  sample_rate: 16000
  frame_length: 20.0
  frame_shift: 10.0
  del_silence: false
  num_mels: 80
  apply_spec_augment: true
  apply_noise_augment: false
  apply_time_stretch_augment: false
  apply_joining_augment: false
augment:
  apply_spec_augment: false
  apply_noise_augment: false
  apply_joining_augment: false
  apply_time_stretch_augment: false
  freq_mask_para: 27
  freq_mask_num: 2
  time_mask_num: 4
  noise_dataset_dir: None
  noise_level: 0.7
  time_stretch_min_rate: 0.7
  time_stretch_max_rate: 1.4
dataset:
  dataset: ksponspeech
  dataset_path: /ssd1/DB/KsponSpeech
  test_dataset_path: /ssd1/DB
  manifest_file_path: /ssd1/DB/KsponSpeech/train_sub.txt
  test_manifest_dir: /ssd1/DB/KsponSpeech_scripts
  preprocess_mode: phonetic
  dataset_download: false
criterion:
  criterion_name: ctc
  reduction: mean
  zero_infinity: true
lr_scheduler:
  lr: 0.0001
  scheduler_name: warmup_reduce_lr_on_plateau
  lr_patience: 1
  lr_factor: 0.3
  peak_lr: 0.0001
  init_lr: 1.0e-10
  warmup_steps: 4000
model:
  model_name: conformer
  encoder_dim: 512
  num_encoder_layers: 17
  num_attention_heads: 8
  feed_forward_expansion_factor: 4
  conv_expansion_factor: 2
  input_dropout_p: 0.1
  feed_forward_dropout_p: 0.1
  attention_dropout_p: 0.1
  conv_dropout_p: 0.1
  conv_kernel_size: 31
  half_step_residual: true
  optimizer: adam
trainer:
  seed: 1
  accelerator: dp
  accumulate_grad_batches: 1
  num_workers: 4
  batch_size: 4
  check_val_every_n_epoch: 1
  gradient_clip_val: 5.0
  logger: wandb
  max_epochs: 20
  save_checkpoint_n_steps: 10000
  auto_scale_batch_size: 'False'
  sampler: else
  name: gpu
  device: gpu
  use_cuda: true
  auto_select_gpus: true
tokenizer:
  sos_token: <s>
  eos_token: </s>
  pad_token: <pad>
  blank_token: <blank>
  encoding: utf-8
  unit: kspon_subword
  sp_model_path: /ssd1/DB/KsponSpeech/labels_sub.model
  vocab_size: 3200
save_checkpoint_n_steps: 10000

[2024-01-31 14:01:57,957][openspeech.utils][INFO] - Operating System : Linux 6.5.0-15-generic
[2024-01-31 14:01:57,957][openspeech.utils][INFO] - Processor : x86_64
[2024-01-31 14:01:57,957][openspeech.utils][INFO] - device : NVIDIA GeForce RTX 4090
[2024-01-31 14:01:57,958][openspeech.utils][INFO] - CUDA is available : True
[2024-01-31 14:01:57,958][openspeech.utils][INFO] - CUDA version : 11.1
[2024-01-31 14:01:57,958][openspeech.utils][INFO] - PyTorch version : 1.8.1+cu111
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs

conda list

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   2.1.0                    pypi_0    pypi
aiohttp                   3.9.1                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
antlr4-python3-runtime    4.9.3                    pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
astropy                   5.2.2                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
audioread                 3.0.1                    pypi_0    pypi
ca-certificates           2023.11.17           hbcca054_0    conda-forge
cachetools                5.3.2                    pypi_0    pypi
certifi                   2023.11.17               pypi_0    pypi
cffi                      1.16.0                   pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
ctcdecode                 1.0.3                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
docker-pycreds            0.4.0                    pypi_0    pypi
filelock                  3.13.1                   pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2023.12.2                pypi_0    pypi
future                    0.18.3                   pypi_0    pypi
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.41                   pypi_0    pypi
google-auth               2.27.0                   pypi_0    pypi
google-auth-oauthlib      1.0.0                    pypi_0    pypi
grpcio                    1.60.0                   pypi_0    pypi
hydra-core                1.3.2                    pypi_0    pypi
idna                      3.6                      pypi_0    pypi
importlib-metadata        7.0.1                    pypi_0    pypi
importlib-resources       6.1.1                    pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
joblib                    1.3.2                    pypi_0    pypi
lazy-loader               0.3                      pypi_0    pypi
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
levenshtein               0.23.0                   pypi_0    pypi
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
librosa                   0.9.2                    pypi_0    pypi
libsqlite                 3.44.2               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lightning-utilities       0.10.1                   pypi_0    pypi
llvmlite                  0.41.1                   pypi_0    pypi
markdown                  3.5.2                    pypi_0    pypi
markupsafe                2.1.4                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
msgpack                   1.0.7                    pypi_0    pypi
multidict                 6.0.4                    pypi_0    pypi
ncurses                   6.4                  h59595ed_2    conda-forge
networkx                  3.1                      pypi_0    pypi
numba                     0.58.1                   pypi_0    pypi
numpy                     1.24.4                   pypi_0    pypi
nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
nvidia-cudnn-cu12         8.9.2.26                 pypi_0    pypi
nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
nvidia-nccl-cu12          2.18.1                   pypi_0    pypi
nvidia-nvjitlink-cu12     12.3.101                 pypi_0    pypi
nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
omegaconf                 2.3.0                    pypi_0    pypi
openspeech-core           0.4.0                     dev_0    <develop>
openssl                   1.1.1w               hd590300_0    conda-forge
packaging                 23.2                     pypi_0    pypi
pandas                    2.0.3                    pypi_0    pypi
pillow                    10.2.0                   pypi_0    pypi
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
platformdirs              4.1.0                    pypi_0    pypi
pooch                     1.8.0                    pypi_0    pypi
protobuf                  4.25.2                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
pyasn1                    0.5.1                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pybind11                  2.11.1                   pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pydeprecate               0.3.1                    pypi_0    pypi
pyerfa                    2.0.0.3                  pypi_0    pypi
python                    3.8.0                h357f687_5    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
python-levenshtein        0.23.0                   pypi_0    pypi
pytorch-lightning         1.4.0                    pypi_0    pypi
pytz                      2023.3.post1             pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
rapidfuzz                 3.6.1                    pypi_0    pypi
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
resampy                   0.4.2                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-learn              1.3.2                    pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
sentencepiece             0.1.99                   pypi_0    pypi
sentry-sdk                1.39.2                   pypi_0    pypi
setproctitle              1.3.3                    pypi_0    pypi
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
six                       1.16.0                   pypi_0    pypi
smmap                     5.0.1                    pypi_0    pypi
soundfile                 0.12.1                   pypi_0    pypi
soxr                      0.3.7                    pypi_0    pypi
sqlite                    3.44.2               h2c6b66d_0    conda-forge
sympy                     1.12                     pypi_0    pypi
tensorboard               2.14.0                   pypi_0    pypi
tensorboard-data-server   0.7.2                    pypi_0    pypi
threadpoolctl             3.2.0                    pypi_0    pypi
tk                        8.6.13          noxft_h4845f30_101    conda-forge
torch                     1.8.1+cu111              pypi_0    pypi
torchaudio                0.8.1                    pypi_0    pypi
torchmetrics              0.4.0                    pypi_0    pypi
torchvision               0.9.1+cu111              pypi_0    pypi
tqdm                      4.66.1                   pypi_0    pypi
triton                    2.1.0                    pypi_0    pypi
typing-extensions         4.9.0                    pypi_0    pypi
tzdata                    2023.4                   pypi_0    pypi
urllib3                   2.1.0                    pypi_0    pypi
wandb                     0.16.2                   pypi_0    pypi
warp-rnnt                 0.4.0                    pypi_0    pypi
werkzeug                  3.0.1                    pypi_0    pypi
wget                      3.2                      pypi_0    pypi
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yarl                      1.9.4                    pypi_0    pypi
zipp                      3.17.0                   pypi_0    pypi
zlib                      1.2.13               hd590300_5    conda-forge

Dataset structure

dataset.dataset_path: $BASE_PATH/KsponSpeech
$BASE_PATH/KsponSpeech
├── KsponSpeech_01
├── KsponSpeech_02
├── KsponSpeech_03
├── KsponSpeech_04
├── KsponSpeech_05
└── etc... (cvs, model, txt...)
dataset.test_dataset_path: $BASE_PATH/KsponSpeech_eval
$BASE_PATH/KsponSpeech_eval
├── eval_clean
└── eval_other
dataset.test_manifest_dir: $BASE_PATH/KsponSpeech_scripts
$BASE_PATH/KsponSpeech_scripts
├── eval_clean.trn
└── eval_other.trn

감사합니다.
@upskyy
Copy link
Member

upskyy commented Feb 1, 2024

@Seoyoung-Jo
확인해보겠습니다.
감사합니다.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@upskyy @Seoyoung-Jo