Skip to content

EasyRobust: an Easy-to-use library for state-of-the-art Robust Computer Vision Research with PyTorch.

License

Notifications You must be signed in to change notification settings

alibaba/easyrobust

Repository files navigation

EasyRobust

license open issues GitHub pull-requests GitHub latest release

What's New

Our Research Project

  • [T-IFS 2024] Revisiting and Exploring Efficient Fast Adversarial Training via LAW: Lipschitz Regularization and Auto Weight Averaging [Paper, Code]
  • [ICCV 2023] COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts [Paper, COCO-O dataset]
  • [INTERSPEECH 2023] Robust Automatic Speech Recognition via WavAugment Guided Phoneme Adversarial Training [Paper, Code]
  • [CVPR 2023] ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing [Paper, Image editing toolkit, ImageNet-E dataset]
  • [ICLR 2023] Inequality phenomenon in $l_\infty$-adversarial training, and its unrealized threats [Paper, Code]
  • [ICASSP 2023] TransAudio: Towards the Transferable Adversarial Audio Attack via Learning Contextualized Perturbations [Paper, Code]
  • [TIP 2022] Towards Understanding and Boosting Adversarial Transferability from a Distribution Perspective [Paper, Code]
  • [NeurIPS 2022] Boosting Out-of-distribution Detection with Typical Features [Paper, Code]
  • [NeurIPS 2022] Enhance the Visual Representation via Discrete Adversarial Training [Paper, Code]
  • [CVPR 2022] Towards Robust Vision Transformer [Paper, Code]

Introduction

EasyRobust is an Easy-to-use library for state-of-the-art Robust Computer Vision Research with PyTorch. EasyRobust aims to accelerate research cycle in robust vision, by collecting comprehensive robust training techniques and benchmarking them with various robustness metrics. The key features includes:

Technical Articles

We have a series of technical articles on the functionalities of EasyRobust.

Installation

Install from Source:

$ git clone https://github.com/alibaba/easyrobust.git
$ cd easyrobust
$ pip install -e .

Install from PyPI:

$ pip install easyrobust

download the ImageNet dataset and place into /path/to/imagenet. Specify $ImageNetDataDir as ImageNet path by:

$ export ImageNetDataDir=/path/to/imagenet

[Optional]: If you use EasyRobust to evaluate the model robustness, download the benchmark dataset by:

$ sh download_data.sh

[Optional]: If you use analysis tools in tools/, install extra requirements by:

$ pip install -r requirements/optional.txt

Docker

We have provided a runnable environment in docker/Dockerfile for users who do not want to install by pip. To use it, please confirm that docker and nvidia-docker have installed. Then run the following command:

docker build -t alibaba/easyrobust:v1 -f docker/Dockerfile . 

Getting Started

EasyRobust focuses on the basic usages of: (1) Evaluate and benchmark the robustness of a pretrained models and (2) Train your own robust models or reproduce the results of previous SOTA methods.

1. How to evaluate and benchmark the robustness of given models?

It only requires a few lines to evaluate the robustness of a model using EasyRobust. We give a minimalist example in benchmarks/resnet50_example.py:

#############################################################
#         Define your model
#############################################################
model = torchvision.models.resnet50(pretrained=True)
model = model.eval()
if torch.cuda.is_available(): model = model.cuda()

#############################################################
#         Start Evaluation
#############################################################

# ood
evaluate_imagenet_val(model, 'benchmarks/data/imagenet-val')
evaluate_imagenet_a(model, 'benchmarks/data/imagenet-a')
evaluate_imagenet_r(model, 'benchmarks/data/imagenet-r')
evaluate_imagenet_sketch(model, 'benchmarks/data/imagenet-sketch')
evaluate_imagenet_v2(model, 'benchmarks/data/imagenetv2')
evaluate_stylized_imagenet(model, 'benchmarks/data/imagenet-style')
evaluate_imagenet_c(model, 'benchmarks/data/imagenet-c')
# objectnet is optional since it spends a lot of disk storage. we skip it here. 
# evaluate_objectnet(model, 'benchmarks/data/ObjectNet/images')

# adversarial
evaluate_imagenet_autoattack(model, 'benchmarks/data/imagenet-val')

You can do evaluation by simply running the command: python benchmarks/resnet50_example.py. After running is completed, your will get the following output:

Top1 Accuracy on the ImageNet-Val: 76.1%
Top1 Accuracy on the ImageNet-A: 0.0%
Top1 Accuracy on the ImageNet-R: 36.2%
Top1 Accuracy on the ImageNet-Sketch: 24.1%
Top1 Accuracy on the ImageNet-V2: 63.2%
Top1 Accuracy on the Stylized-ImageNet: 7.4%
Top1 accuracy 39.2%, mCE: 76.7 on the ImageNet-C
Top1 Accuracy on the AutoAttack: 0.0%

2. How to use EasyRobust to train my own robust models?

We implement most robust training methods in the folder examples/imageclassification/imagenet/. All of them are based on a basic training script: examples/imageclassification/imagenet/base_training_script.py. By comparing the difference, you can clearly see where and which hyperparameters of basic training are modified to create a robust training example. Below we present the tutorials of some classic methods:

Analytical Tools

see tools/README.md

Model Zoo and Baselines

Submit your models

We provide a tool benchmarks/benchmark.py to help users directly benchmark their models:

Usage: 
    python benchmarks/benchmark.py [OPTIONS...]

OPTIONS:
    --model [ARCH in timm]
    --data_dir [PATH of the bencmark datasets]
    --ckpt_path [URL or PATH of the model weights]

If you are willing to submit the model to our benchmarks, you can prepare a python script similar to benchmarks/benchmark.py and weights file xxx.pth, zip all the files. Then open an issue with the "Submit Model" template and provide a json storing submit information. Below is a submission template in adversarial robustness benchmark of image classification:

## Submit Json Information

{"date": "19/06/2017", 
 "extra_data": "no", 
 "model": "<b>Adversarial Training</b>", 
 "institution": "MIT", 
 "paper_link": "https://arxiv.org/abs/1706.06083", 
 "code_link": "", 
 "architecture": "swin-b", 
 "training framework": "easyrobust (v1)", 
 "ImageNet-val": 75.05, 
 "autoattack": 47.42, 
 "files": "<a href=http://alisec-competition.oss-cn-shanghai.aliyuncs.com/xiaofeng/imagenet_pretrained_models/advtrain_models/advtrain_swin_base_patch4_window7_224_ep4.pth >download</a>", 
 "advrob_imgcls_leaderboard": true, 
 "oodrob_imgcls_leaderboard": false, 
 "advrob_objdet_leaderboard": false, 
 "oodrob_objdet_leaderboard": false}

We will check the result and present your result into the benchmark if there is no problem. For submission template of other benchmarks, check submit-model.md.

Below is the model zoo and benchmark of the EasyRobust. All the results are runned by benchmarks/adv_robust_bench.sh and benchmarks/non_adv_robust_bench.sh.

Adversarial Robust Benchmark (sorted by AutoAttack)

Training Framework Method Model ImageNet-Val AutoAttack Files
EasyRobust (V1) Adversarial Training Swin-B 75.05% 47.42% ckpt
EasyRobust (V1) Adversarial Training Swin-S 73.41% 46.76% ckpt
EasyRobust (V1) Adversarial Training ViT-B/16 70.64% 43.04% ckpt
EasyRobust (V1) Adversarial Training EfficientNet-B3 67.65% 41.72% ckpt
EasyRobust (V1) Adversarial Training ResNet101 69.51% 41.04% ckpt
EasyRobust (V1) Adversarial Training ViT-S/16 66.43% 39.20% ckpt
EasyRobust (V1) Adversarial Training EfficientNet-B2 64.75% 38.54% ckpt
EasyRobust (V1) Adversarial Training ResNeSt50d 70.03% 38.52% ckpt
EasyRobust (V1) Adversarial Training ViT-B/32 65.58% 37.38% ckpt
EasyRobust (V1) Adversarial Training EfficientNet-B1 63.99% 37.20% ckpt
EasyRobust (V1) Adversarial Training SEResNet101 71.11% 37.18% ckpt
EasyRobust (V1) Adversarial Training ResNeXt50_32x4d 67.39% 36.42% ckpt
EasyRobust (V1) Adversarial Training EfficientNet-B0 61.83% 35.06% ckpt
robustness Adversarial Training ResNet50 64.02% 34.96% ckpt
EasyRobust (Ours) Adversarial Training ResNet50 65.1% 34.9% ckpt/args/logs
EasyRobust (V1) Adversarial Training SEResNet50 66.68% 33.56% ckpt
EasyRobust (V1) Adversarial Training DenseNet121 60.90% 29.78% ckpt
Official Free AT ResNet50 59.96% 28.58% ckpt
Official FGSM AT ResNet50 55.62% 26.24% ckpt
EasyRobust (V1) Adversarial Training VGG16 59.96% 25.92% ckpt

Non-Adversarial Robust Benchmark (sorted by ImageNet-C)

Training Framework Method Model Files ImageNet-Val V2 C (mCE↓) R A Sketch Stylized ObjectNet
EasyRobust (Ours) DAT ViT-B/16 ckpt/args/logs 81.38% 69.99% 45.59 49.64% 24.61% 36.46% 24.84% 20.12%
EasyRobust (Ours) - RVT-S* ckpt/args/logs 82.10% 71.40% 48.22 47.84% 26.93% 35.34% 20.71% 23.24%
Official - RVT-S* ckpt 81.82% 71.05% 49.42 47.33% 26.53% 34.22% 20.48% 23.11%
EasyRobust (Ours) - DrViT-S ckpt/args/logs 80.66% 69.62% 49.96 43.68% 20.79% 31.13% 17.89% 20.50%
- - DrViT-S - 77.03% 64.49% 56.89 39.02% 11.85% 28.78% 14.22% 26.49%
Official PRIME ResNet50 ckpt 76.91% 65.42% 57.49 42.20% 2.21% 29.82% 13.94% 16.59%
EasyRobust (Ours) PRIME ResNet50 ckpt/args/logs 76.64% 64.37% 57.62 41.95% 2.07% 29.63% 13.56% 16.28%
EasyRobust (Ours) DeepAugment ResNet50 ckpt/args/logs 76.58% 64.77% 60.27 42.80% 3.62% 29.65% 14.88% 16.88%
Official DeepAugment ResNet50 ckpt 76.66% 65.24% 60.37 42.17% 3.46% 29.50% 14.68% 17.13%
EasyRobust (Ours) Augmix ResNet50 ckpt/args/logs 77.81% 65.60% 64.14 43.34% 4.04% 29.81% 12.33% 17.21%
EasyRobust (Ours) APR ResNet50 ckpt/args/logs 76.28% 64.78% 64.89 42.17% 4.18% 28.90% 13.03% 16.78%
Official Augmix ResNet50 ckpt 77.54% 65.42% 65.27 41.04% 3.78% 28.48% 11.24% 17.54%
Official APR ResNet50 ckpt 75.61% 64.24% 65.56 41.35% 3.20% 28.37% 13.01% 16.61%
Official S&T Debiased ResNet50 ckpt 76.91% 65.04% 67.55 40.81% 3.50% 28.41% 17.40% 17.38%
EasyRobust (Ours) SIN+IN ResNet50 ckpt/args/logs 75.46% 63.50% 67.73 42.34% 2.47% 31.39% 59.37% 16.17%
Official SIN+IN ResNet50 ckpt 74.59% 62.43% 69.32 41.45% 1.95% 29.69% 57.38% 15.93%
Non-Official AdvProp ResNet50 ckpt 77.04% 65.27% 70.81 40.13% 3.45% 25.95% 10.01% 18.23%
EasyRobust (Ours) S&T Debiased ResNet50 ckpt/args/logs 77.21% 65.10% 70.98 38.59% 3.28% 26.09% 14.59% 16.99%
EasyRobust (Ours) AdvProp ResNet50 ckpt/args/logs 76.64% 64.35% 77.64 37.43% 2.83% 24.71% 7.33% 16.82%

Credits

EasyRobust concretizes previous excellent works by many different authors. We'd like to thank, in particular, the following implementations which have helped us in our development:

  • timm @rwightman and the training script.
  • robustness @MadryLab and autoattack @fra31 for attack implementation.
  • modelvshuman @bethgelab for model analysis.
  • AdaIN @naoto0804 for style trnsfer and VQGAN @CompVis for image discretization.
  • All the authors and implementations of the robustness research work we refer in this library.

Citing EasyRobust

We provide a BibTeX entry for users who apply EasyRobust to help their research:

@misc{mao2022easyrobust,
  author =       {Xiaofeng Mao and Yuefeng Chen and Xiaodan Li and Gege Qi and Ranjie Duan and Rong Zhang and Hui Xue},
  title =        {EasyRobust: A Comprehensive and Easy-to-use Toolkit for Robust Computer Vision},
  howpublished = {\url{https://github.com/alibaba/easyrobust}},
  year =         {2022}
}