Incomparable results in the WMT 2014 EN-DE table for machine translation

[rihardsk]

I noticed that some of the results reported in the WMT 2014 EN-DE table are obtained by models trained on data from newer WMT datasets (but they report results on newstest2014), e.g, Edunov et al. (2018) uses WMT’18 and Wu et al. (2019) uses WMT’16 for training.

The few results on WMT 2014 EN-FR that i checked were fine though. Here are the papers i checked

Paper	en-de data	en-fr data
Transformer (Vaswani et al., 2017)	WMT’2014	WMT’2014
AdvSoft + Transformer Big (Wang et al., 2019)	WMT’2014
MUSE (Zhao et al., 2019)	WMT’2014	WMT’2014
DynamicConv (Wu et al., 2019)	WMT’2016	WMT’2014
Transformer Big + BT (Edunov et al., 2018)	WMT’2018	WMT’2014

[sebastianruder]

Thanks for this observation. Do you think you could an asterisk (*) to the models that use newer data for training?