Huggingface Spaces Demo UI: Whisper-youtube-crosslingual-subtitles

[R4ZZ3]

Hi,

As part of Huggingface whisper finetuning event I created a demo where you can:

Download youtube video with a given URL

2. Watch downloaded video in the first video component

3. Run automatic speech recognition on the video using Whisper models using models from this

4. Translate the recognized transcriptions to 26 languages supported by deepL

5. Download generated subtitle files in .srt and .vtt formats

6. Watch the video in another video component with added subtitles
   You can test it from here
   --> https://huggingface.co/spaces/RASMUS/Whisper-youtube-crosslingual-subtitles <--

[trbielec]

This is really user friendly, thank you! Could you add a .txt output option as well? And maybe an option to skip translation?

[R4ZZ3]

Sure thing. What kind of .txt are you looking for? Just the whole text concatenated together or with timestamps?

For skipping would you like to see something like a button to "use original transcription for subtitles" or just add logic if transcription language == selected_translation_language as default?

[trbielec]

Sorry for the delay. Yes concatenated together without timestamps would be fantastic

[iansquenet]

I really only seem to be able to get the sound effects by using this version only (like taps, clapping etc), but I'd like to run it locally. How could I go about doing that?

[ulatekh]

I've gotten decent results with whisperX. The output still needs editing...the timestamps are very accurate when the background audio is quiet, but tend to get out of sync when there's a lot of overlapping non-voice activity. But whisperX generates a very good first draft that I then hand-edit. I'm testing it on a handful of videos for which there have never been subtitles, and that I have never fully understood (due to mumbling, thick accents, and/or background audio that masks the dialog).