Skip to content

Official implementation of the NAACL 2024 paper "Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learning"

License

MIT, MIT licenses found

Licenses found

MIT
LICENSE
MIT
LICENSE_R2REnvDrop
Notifications You must be signed in to change notification settings

eric-ai-lab/Naivgation-as-wish

Repository files navigation

Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learning

University of California, Santa Cruz, USA 

Accepted to the Main Conference of NAACL 2024

We release the reproducible code here.

Environment Installation

Python version: Need python 3.8

pip install -r python_requirements.txt

Please refer to this link to install Matterport3D simulator:

Pre-Computed Features

ImageNet ResNet152

Download image features for environments for Envdrop model:

mkdir img_features
wget https://www.dropbox.com/s/o57kxh2mn5rkx4o/ResNet-152-imagenet.zip -P img_features/
cd img_features
unzip ResNet-152-imagenet.zip

CLIP Features

Please download the CLIP-ViT features for CLIP-ViL models with this link:

wget https://nlp.cs.unc.edu/data/vln_clip/features/CLIP-ViT-B-32-views.tsv -P img_features

Training and Testing On RxR

Data

Please download the pre-processed data with link:

wget https://nlp.cs.unc.edu/data/vln_clip/RxR.zip -P tasks
unzip tasks/RxR.zip -d tasks/

Testing NAW and PBA

For testing the performance of PBA on RxR dataset with model FedEnvDrop, please run

bash run/agent_rxr_envdrop_attack.bash

For testing the performance of PBA on RxR dataset with model FedCLIP-ViL, please run

bash run/agent_rxr_clip_vit_attack.bash

If you want to simply test NAW without any defense, please change the param defense_method in the bash file from PBA to mean.

Training and Testing on R2R

Download the Data

Download Room-to-Room navigation data:

bash ./tasks/R2R/data/download.sh

Testing NAW and PBA

For testing the performance of PBA on R2R dataset with model FedEnvDrop, please run

bash run/agent_envdrop_attack.bash

For testing the performance of PBA on R2R dataset with model FedCLIP-ViL, please run

bash run/agent_clip_vit_attack.bash

If you want to simply test NAW without any defense, please change the param defense_method in the bash file from PBA to mean.

Related Links

Reference

If you use our work in your research or wish to refer to the baseline results published here, please use the following BibTeX entry.

@inproceedings{zhang-etal-2024-NAW,
    title = "Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learning",
    author = "Zhang, Yunchao and Di, Zonglin and Zhou, Kaiwen and Xie, Cihang and Wang, Xin Eric",
    booktitle = "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)",
    year = "2024",
}

About

Official implementation of the NAACL 2024 paper "Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learning"

Topics

Resources

License

MIT, MIT licenses found

Licenses found

MIT
LICENSE
MIT
LICENSE_R2REnvDrop

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published