Resolving Cython and youtokentome dependencies

[oakieoaktree]

Trying and failing to install nemo_toolkit['all'].

First installed Cython as described (pip install Cython)

However when running pip install nemo_toolkit['all'] the build step Collecting youtokentome>=1.0.5 (from nemo_toolkit[all]) fails with ModuleNotFoundError: No module named 'Cython'

Any ideas? No luck on the youtokentome github pages. Any known combinations of versions to use here?

Environment:
MacBookPro 2021 M1
python --version # 3.10.8
pip --version # 23.3.2
pip list cython # Cython 3.0.8

[titu1994]

@AlexGrinch @okuchaiev we've kept yttm for long enough, let's remove it it's an outdated library

[Dev-iL]

In the meantime, the workaround is to pip install wheel first.

[genagurbanguliyev]

Do you use conda venv ?

[taras-sereda]

it there any real need to keep youtokentom in the NeMo?
I only found one tokenizer that is based on it, but It's not even used for training any of the models.

It will be great to get rid off one extra dependency!

[titu1994]

As of #8322, yttm has been removed from NeMo.