Oryx: Support subtitle 608/708 format by FFmpeg

[winlinvip]

Use Scenario

SyncWords AI Media both companies have this feature. AWS also integrates SyncWords tools to their service, see this post.

The common methods to embed closed captions into live streams include embedding captions into EIA-608 and 708 standards. Or using RTMP onCaptionInfo to embed base-64 encoded 608/708 caption data.

The other challenge is that 608/708 embedded captions do not support more than two languages, especially for languages outside of the seven languages supported by 608. This means that if there is a requirement to stream with three or more languages with support for Unicode character sets like Chinese, Japanese, and Korean than the traditional 608/708 method of encoding is a non-starter.

If i want to send live caption to youtube or vimeo i need to use this format. any company who wants to use this for broadcasting will wnat it in that format.

Note that besides 608/708 format, HLS also support Web Video Text Tracks (WebVTT), which is supported by Oryx, please see Transcription.

Goal

Patch FFmpeg to support mux video and subtitle to mpegts file, for example:

ffmpeg -i input.mp4 -i subtitle.srt ...... -f mpegts output.ts

Using this command, FFmepg mux video input.mp4 and subtitle subtitle.srt to output.ts with 608 subtitle.

Research

Use VLC or MPlayer as test palyer for 608/708 captions. to valdidate you can use samples by CCExtrator.

FFmpeg only supports decoding of 608 subtitles and does not implement encoding. If encoding is required, it needs to be implemented independently.