❓Get stats (e.g. counts) about the merged pairs

[pietrolesci]

Hi there,

I was wondering whether there is an easy way to ask the tokeniser trainer to return the counts (or frequency) of the pair in the moment the merge decision is made. The ideal output would be having a new column with the frequency of the pair in the merges.txt file.

Tagging the line below as it seems the relevant part of the code that has the info (i.e., pair_counts)

tokenizers/tokenizers/src/models/bpe/trainer.rs

Line 465 in 25aee8b

let (mut pair_counts, mut where_to_update) = self.count_pairs(&words, &counts, &progress);

Thanks a lot for your help!