We proposed a new AMR-based logic-driven data augmentation for contrastive learning intermediate training and then we conduct the downstream tasks require logical reasoning including logical reasoning reading comprehension tasks (ReClor and LogiQA) and natural language inference tasks (MNLI, MRPC, RTE, QNLI and QQP). Our AMR-LDA
model (AMR-LDA Prompt Augmentation+GPT4) and AMR-LDA (DeBERTa-v2-xxlarge-AMR-LDA-Cont)
lead the ReClor leaderboard and we are the first group scored above 90% on the hidden test set around the world. Our paper has been accepted by the Findings of ACL-24.
- Install the all required packages from the requirements_latest.txt
pip install -r requirements_latest.txt
- You can run
logical_equivalence_synthetic_dataset.py
to automatically generate sentences which is ready for the stage-1 finetuning. - All code about logical equivalence data augmentation can be found in logical_equivalence_functions.py. You can run the script by
python logical_equivalence_functions.py
- To adjust the porprotion of positive and negative samples in the stage-1 finetuning, you can run the
negative_sample_extention.py
.
You can follow the running script script_running_notes.txt
and use the training commands to conduct stage-1 finetuning and stage-2 finetuning. Please remember you need to conduct the stage-1 finetuning firstly and then conduct the stage-2 finetuning. The main function code is in BERT/run_glue_no_trainer.py
.
Here is an example of stage-1 finetuning.
python run_glue_no_trainer.py \
--seed 2021 \
--model_name_or_path roberta-large \
--train_file ../output_result/Synthetic_xfm_t5wtense_logical_equivalence_train_v4.csv \
--validation_file ../output_result/Synthetic_xfm_t5wtense_logical_equivalence_validation_v4.csv \
--max_length 256 \
--per_device_train_batch_size 32 \
--learning_rate 2e-5 \
--num_train_epochs 10 \
--output_dir Transformers/roberta-large-our-model-v4/
Here is an example of stage-2 finetuning on MRPC.
python run_glue_no_trainer.py \
--seed 42 \
--model_name_or_path Transformers/roberta-large-our-model-v4/ \
--task_name mrpc \
--max_length 256 \
--per_device_train_batch_size 32 \
--learning_rate 2e-5 \
--num_train_epochs 10 \
--output_dir Transformers/mrpc/synthetic-logical-equivalence-finetuned-roberta-large-v4/
For the stage-2 finetuning on ReClor and LogiQA, you need to run the commands under the BERT/scripts
.
Here is an example of stage-2 finetuning for ReClor.
export RECLOR_DIR=reclor_data
export TASK_NAME=reclor
export MODEL_NAME=microsoft/deberta-v2-xxlarge
export OUTPUT_NAME=deberta-v2-xxlarge
CUDA_VISIBLE_DEVICES=3 python run_multiple_choice.py \
--model_type debertav2 \
--model_name_or_path $MODEL_NAME \
--task_name $TASK_NAME \
--do_train \
--evaluate_during_training \
--do_test \
--do_lower_case \
--data_dir $RECLOR_DIR \
--max_seq_length 256 \
--per_gpu_eval_batch_size 4 \
--per_gpu_train_batch_size 4 \
--gradient_accumulation_steps 24 \
--learning_rate 1e-05 \
--num_train_epochs 10.0 \
--output_dir Checkpoints/$TASK_NAME/${OUTPUT_NAME} \
--logging_steps 200 \
--save_steps 200 \
--adam_betas "(0.9, 0.98)" \
--adam_epsilon 1e-6 \
--no_clip_grad_norm \
--warmup_proportion 0.1 \
--weight_decay 0.01
Here is an example of stage-2 finetuning for LogiQA.
export RECLOR_DIR=logiqa_data
export TASK_NAME=logiqa
export MODEL_NAME=microsoft/deberta-v2-xxlarge
export OUTPUT_NAME=deberta-v2-xxlarge
CUDA_VISIBLE_DEVICES=3 python run_multiple_choice.py \
--model_type debertav2 \
--model_name_or_path $MODEL_NAME \
--task_name $TASK_NAME \
--do_train \
--evaluate_during_training \
--do_test \
--do_lower_case \
--data_dir $RECLOR_DIR \
--max_seq_length 256 \
--per_gpu_eval_batch_size 4 \
--per_gpu_train_batch_size 4 \
--gradient_accumulation_steps 24 \
--learning_rate 1e-05 \
--num_train_epochs 10.0 \
--output_dir Checkpoints/$TASK_NAME/${OUTPUT_NAME} \
--logging_steps 200 \
--save_steps 200 \
--adam_betas "(0.9, 0.98)" \
--adam_epsilon 1e-6 \
--no_clip_grad_norm \
--warmup_proportion 0.1 \
--weight_decay 0.01
If the paper and code are helpful, please kindly cite our paper:
@inproceedings{Bao24amrlda,
author = {Qiming Bao and
Alex Yuxuan Peng and
Zhenyun Deng and
Wanjun Zhong and
Gaël Gendron and
Neşet Tan and
Nathan Young and
Yang Chen and
Yonghua Zhu and
Michael Witbrock and
Jiamou Liu},
title = {Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning},
booktitle = {Findings of ACL},
publisher = {{ACL}},
year = {2024}
}