Better Co-training Images and Video Method In imagen-video Paper

[lxj616]

As you request in README, I'm here to open a issue about "how this is done", you must have seen the paper already

We follow Ho et al. (2022b) in jointly training all the models in the Imagen Video pipeline on images
and videos. During training, individual images are treated as single frame videos. We achieve this
by packing individual independent images into a sequence of the same length as a video, and bypass
the temporal convolution residual blocks by masking out their computation path. Similarly, we
disable cross-frame temporal attention by applying masking to the temporal attention maps. This
strategy allows us to use to train our video models on image-text datasets that are significantly larger
and more diverse than available video-text datasets.

https://imagen.research.google/video/paper.pdf