Skip to content

Machine translated multilingual STS benchmark dataset.

License

Notifications You must be signed in to change notification settings

WhereIsAI/stsb-multi-mt

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

39 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

STSb Multi MT

Machine translated multilingual STS benchmark dataset.

These are different multilingual translations and the English original of the STSbenchmark dataset. Translation has been done with deepl.com.

  • Available languages are: de, en, es, fr, it, ja, nl, pl, pt, ru, zh
  • Dataset splits are called: train, dev, test

It can be used to train sentence embeddings like T-Systems-onsite/cross-en-de-roberta-sentence-transformer.

Please open an issue if you have questions or want to report problems.

This dataset provides pairs of sentences and a score of their similarity.

score 2 example sentences explanation
5 The bird is bathing in the sink.
Birdie is washing itself in the water basin.
The two sentences are completely equivalent, as they mean the same thing.
4 Two boys on a couch are playing video games.
Two boys are playing a video game.
The two sentences are mostly equivalent, but some unimportant details differ.
3 John said he is considered a witness but not a suspect.
“He is not a suspect anymore.” John said.
The two sentences are roughly equivalent, but some important information differs/missing.
2 They flew out of the nest in groups.
They flew into the nest together.
The two sentences are not equivalent, but share some details.
1 The woman is playing the violin.
The young lady enjoys listening to the guitar.
The two sentences are not equivalent, but are on the same topic.
0 The black dog is running through the snow.
A race car driver is driving his car through the mud.
The two sentences are completely dissimilar.

Content

  • folder raw-data: the raw data how it was convertet with deepl.com
  • folder data: the data: sentence1, sentence2, similarity_score
  • convert.py: script to convert data from raw-data to data

Examples of Use

import csv

with open(filepath, newline="", encoding="utf-8") as csvfile:
    csv_dict_reader = csv.DictReader(
        csvfile,
        dialect='excel',
        fieldnames=["sentence1", "sentence2", "similarity_score"],
    )
    for row in csv_dict_reader:
        print(row)

Known Issues

none

Manual Testing of Datasets

Language 1st train 1000st train last train 1st dev 1000st dev last dev 1st test 1000st test last test
de ok ok ok ok ok ok ok ok ok
en ok ok ok ok ok ok ok ok ok
es
fr
it
ja
nl ok ok partially English ok ok ok ok ok poor grammar
pl
pt
ru
zh

About

Machine translated multilingual STS benchmark dataset.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%