Skip to content

B.Sc. Final Project: LXMERT Model Compression for Visual Question Answering.

License

Notifications You must be signed in to change notification settings

ghazaleh-mahmoodi/lxmert_compression

Repository files navigation

LXMERT Model Compression for Visual Question Answering

arXiv PWC PWC

This project implementation is built on the great repo of LXMERT and PyTorch code for the EMNLP 2019 paper "LXMERT: Learning Cross-Modality Encoder Representations from Transformers" on VQA v2.0.

See the complete report here (Latex Template at overleaf).

Slides of project representation are available here (Google Docs).

Abstract Paper Accepted in WeCNLP is available here (paper, poster, video).

Visual Question Answering Usage

Medical Visual Question Answering

"VQA-Med: Overview of the Medical Visual Question Answering Task at ImageCLEF 2019" alt text

Answering Visual Questions from Blind People

"VizWiz Grand Challenge: Answering Visual Questions from Blind People" alt text

Summary

Large-scale pretrained models such as LXMERT are becoming popular for learning cross-modal representations on text-image pairs for vision-language tasks. According to the lottery ticket hypothesis, NLP and computer vision models contain smaller subnetworks capable of being trained in isolation to full performance. In this project, we combine these observations to evaluate whether such trainable subnetworks exist in LXMERT when fine-tuned on the VQA task. In addition, we perform a model size cost-benefit analysis by investigating how much pruning can be done without significant loss in accuracy.

Run

Install the required packages

pip3 install -r requirements.txt

Run All Experiment

to run all experiment ,in lxmert folder run following command:

bash run/vqa_run.bash

Results

  • The plots are available in lxmert/result directory.
  • The trained models are available in lxmert/models directory.
  • The logs are available in lxmert/logs directory.

Plots

Low Magnitude Pruning Subnetwork

alt text


Random Pruning Subnetwork

alt text


High Magnitude Pruning Subnetwork

alt text


All Result Based on Pruning Sparcity

alt text


All Result Based on Pruning mode

alt text

Citation

@misc{hashemi2023lxmert,
      title={LXMERT Model Compression for Visual Question Answering}, 
      author={Maryam Hashemi and Ghazaleh Mahmoudi and Sara Kodeiri and Hadi Sheikhi and Sauleh Eetemadi},
      year={2023},
      eprint={2310.15325},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

About

B.Sc. Final Project: LXMERT Model Compression for Visual Question Answering.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published