Set of scripts to run monotextor-like pipeline under slurm HPCs

Scripts for parallelized extraction of plain texts from WARC archieves. Aiming at common and reproducible extraction approach.

Information and pipelines on WP4: language models training

Data Analytics Tool

tf/idf-based document aligner from Bitextor

This contains the configuration and scripts for HPLT MT model releases.

Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca

OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.

Curriculum training