Add filepaths to pretty_errors' stacktraces; pretty_errors is installed by PL

[akoumpa]

PL installs pretty_errors which alters sys.excepthook from sys.__excepthook__ to a text formatting function the pretty_errors library defines. However, the format used is incomplete as it misses filepaths, which hinders debugging.

This PR includes pretty_errors and configures it so that we get filepaths in the stacktraces, for example:

Before:

-------------------------------------------------------------------------------
megatron_gpt_finetuning.py 18 <module>
from nemo.utils import logging

__init__.py 31 <module>
from nemo.utils.lightning_logger_patch import add_memory_handlers_to_pl_logger

lightning_logger_patch.py 19 <module>
q()

NameError:
name 'q' is not defined

After:

*******************************************************************************
18 <module> ...examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py"/opt/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py", line 18
> from nemo.utils import logging

31 <module> /opt/NeMo/nemo/utils/__init__.py
"/opt/NeMo/nemo/utils/__init__.py", line 31
> from nemo.utils.lightning_logger_patch import add_memory_handlers_to_pl_logger...

19 <module> /opt/NeMo/nemo/utils/lightning_logger_patch.py
"/opt/NeMo/nemo/utils/lightning_logger_patch.py", line 19

  import logging as _logging
  from logging.handlers import MemoryHandler

  import pytorch_lightning as pl
> q()

  HANDLERS = {}

_logging: <module 'logging' from '/usr/lib/python3.10/logging/__init__.py'>
MemoryHandler: <class 'logging.handlers.MemoryHandler'>
pl: <module 'pytorch_lightning' from '/usr/local/lib/python3.10/dist-p... [105]
NameError:
name 'q' is not defined

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

• Add specific line by line info of high level changes in this PR.

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this 

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[akoumpa]

Example list of synonym files in NeMo

akoumparouli@1604ab7-lcedt:~/fix_ci/NeMo$ find ./ | grep -i '.py' | rev | cut -d'/' -f1 | rev | sort | uniq -c | sort -nr | head -n 20
235 init.py
20 utils.py
7 get_data.py
6 base.py
4 util.py
4 transformer.py
4 rnnt.py
4 helpers.py
4 features.py
4 dataset.py
4 ctc.py
3 waveglow.py
3 transformer_utils.py
3 tokenizer_utils.py
3 tacotron2.py
3 ssl_tts.py
3 spectrogram_enhancer.py
3 radtts.py
3 mixer_tts.py
3 import_datasets.py

[ericharper]

LGTM. Thanks!

[titu1994]

Might be nicer to move this to nemo.utils.logging. doing stuff inside init which usually doesn't get checked often is not a good pattern.

We almost always import logging so you can do a global var check if you already set it once and if so ignore otherwise do this setup in side the logging.py file

[titu1994]

Also, it's a core change so whether here or logging, it needs sign-offs from @nithinraok @ericharper @blisc @yaoyu-33

[akoumpa]

@titu1994 no problem, I will move this to nemo/utils/logging.

[titu1994]

Cool thanks ! Also for some reason the speech text we disabled failed so rebase latest main changes.

[akoumpa]

Ok, false alarm everyone.

On the container I tested on Friday it used torchmetrics v1.4.0, which was installing pretty-errors (https://github.com/Lightning-AI/torchmetrics/blob/v1.4.0/requirements/base.txt#L9)

Ten hours ago, it was moved to a debug requirement https://github.com/Lightning-AI/torchmetrics/pull/2527/files and now the default stack trace is used.

As a result, this is no longer required and I will be closing this PR.

Apologies for pulling everyone to review.

If any take-away from this PR, I would keep perhaps that adding a test to confirm software dependencies altering global behaviors would be a good idea to avoid having them go unnoticed (TODO).