Skip to content

Long Range Graph Benchmark, NeurIPS 2022 Track on D&B

License

Notifications You must be signed in to change notification settings

vijaydwivedi75/lrgb

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

LRGB: Long Range Graph Benchmark

arXiv

We present the Long Range Graph Benchmark (LRGB) with 5 graph learning datasets that arguably require long-range reasoning to achieve strong performance in a given task.

  • PascalVOC-SP
  • COCO-SP
  • PCQM-Contact
  • Peptides-func
  • Peptides-struct

In this repo, we provide the source code to load the proposed datasets and run baseline experiments. The repo is based on GraphGPS which is built using PyG and GraphGym from PyG2.

Update: Reassessment of LRGB

For a reassessment of the baselines on which LRGB were initially evaluated, we refer to this paper and thank @toenshoff for the PR on PCQM-Contact's evaluation metric.

Overview of Datasets

Dataset Domain Task Node Feat. (dim) Edge Feat. (dim) Perf. Metric
PascalVOC-SP Computer Vision Node Prediction Pixel + Coord (14) Edge Weight (1 or 2) macro F1
COCO-SP Computer Vision Node Prediction Pixel + Coord (14) Edge Weight (1 or 2) macro F1
PCQM-Contact Quantum Chemistry Link Prediction Atom Encoder (9) Bond Encoder (3) Hits@K, MRR
Peptides-func Chemistry Graph Classification Atom Encoder (9) Bond Encoder (3) AP
Peptides-struct Chemistry Graph Regression Atom Encoder (9) Bond Encoder (3) MAE

Statistics of Datasets

Dataset # Graphs # Nodes μ Nodes μ Deg. # Edges μ Edges μ Short. Path μ Diameter
PascalVOC-SP 11,355 5,443,545 479.40 5.65 30,777,444 2,710.48 10.74±0.51 27.62±2.13
COCO-SP 123,286 58,793,216 476.88 5.65 332,091,902 2,693.67 10.66±0.55 27.39±2.14
PCQM-Contact 529,434 15,955,687 30.14 2.03 32,341,644 61.09 4.63±0.63 9.86±1.79
Peptides-func 15,535 2,344,859 150.94 2.04 4,773,974 307.30 20.89±9.79 56.99±28.72
Peptides-struct 15,535 2,344,859 150.94 2.04 4,773,974 307.30 20.89±9.79 56.99±28.72

Python environment setup with Conda

conda create -n lrgb python=3.9
conda activate lrgb

conda install pytorch=1.9 torchvision torchaudio -c pytorch -c nvidia
conda install pyg=2.0.2 -c pyg -c conda-forge
conda install pandas scikit-learn

# RDKit is required for OGB-LSC PCQM4Mv2 and datasets derived from it.  
conda install openbabel fsspec rdkit -c conda-forge

# Check https://www.dgl.ai/pages/start.html to install DGL based on your CUDA requirements
pip install dgl-cu111 dglgo -f https://data.dgl.ai/wheels/repo.html

pip install performer-pytorch
pip install torchmetrics==0.7.2
pip install ogb
pip install wandb

conda clean --all

Running GraphGPS

conda activate lrgb

# Running GCN baseline for Peptides-func.
python main.py --cfg configs/GCN/peptides-func-GCN.yaml  wandb.use False

# Running SAN baseline for PascalVOC-SP.
python main.py --cfg configs/SAN/vocsuperpixels-SAN.yaml  wandb.use False

The scripts for all experiments are located in run directory.

W&B logging

To use W&B logging, set wandb.use True and have a gtransformers entity set-up in your W&B account (or change it to whatever else you like by setting wandb.entity).

License Information

Dataset Derived from Original License LRGB Release License
PascalVOC-SP Pascal VOC 2011 Custom* Custom*
COCO-SP MS COCO CC BY 4.0 CC BY 4.0
PCQM-Contact PCQM4Mv2 CC BY 4.0 CC BY 4.0
Peptides-func SATPdb CC BY-NC 4.0 CC BY-NC 4.0
Peptides-struct SATPdb CC BY-NC 4.0 CC BY-NC 4.0

*Custom License for Pascal VOC 2011 (respecting Flickr terms of use)

Leaderboards

The leaderboards of various models' performance on the datasets in LRGB are at paperswithcode.

Currently reported results (last update on Aug 10th, 2023)

PascalVOC-SP (Node Classification)
Model Test F1 (higher is better) Reference #params
Exphormer 0.3975±0.0037 Shirzad, Velingker, Venkatachalam, et al, ICML 2023 509k
GraphGPS 0.3748±0.0109 Rampášek et al, NeurIPS 2022 510k
Cache-GNN+LapPE 0.3462±0.0085 Ma et al, KDD 2023 500k
DRew-GatedGCN+LapPE 0.3314±0.0024 Gutteridge et al, ICML 2023 502k
SAN+LapPE 0.3230±0.0039 Dwivedi et al, NeurIPS 2022 531k
SAN+RWSE 0.3216±0.0027 Dwivedi et al, NeurIPS 2022 468k
GatedGCN+LapPE+virtual node 0.3103±0.0068 Cai et al, ICML 2023 502k
GatedGCN 0.2873±0.0219 Dwivedi et al, NeurIPS 2022 502k
GatedGCN+LapPE 0.2860±0.0085 Dwivedi et al, NeurIPS 2022 502k
Transformer+LapPE 0.2694±0.0098 Dwivedi et al, NeurIPS 2022 501k
GCNII 0.1698±0.0080 Dwivedi et al, NeurIPS 2022 492k
GCN 0.1268±0.0060 Dwivedi et al, NeurIPS 2022 496k
GINE 0.1265±0.0076 Dwivedi et al, NeurIPS 2022 505k
COCO-SP (Node Classification)
Model Test F1 (higher is better) Reference #params
Exphormer 0.3455±0.0009 Shirzad, Velingker, Venkatachalam, et al, ICML 2023 499k
GraphGPS 0.3412±0.0044 Rampášek et al, NeurIPS 2022 516k
Cache-GNN+LapPE 0.2793±0.0033 Ma et al, KDD 2023 500k
GatedGCN 0.2641±0.0045 Dwivedi et al, NeurIPS 2022 509k
Transformer+LapPE 0.2618±0.0031 Dwivedi et al, NeurIPS 2022 508k
SAN+LapPE 0.2592±0.0158 Dwivedi et al, NeurIPS 2022 536k
GatedGCN+LapPE 0.2574±0.0034 Dwivedi et al, NeurIPS 2022 509k
SAN+RWSE 0.2434±0.0156 Dwivedi et al, NeurIPS 2022 474k
GCNII 0.1404±0.0011 Dwivedi et al, NeurIPS 2022 505k
GINE 0.1339±0.0044 Dwivedi et al, NeurIPS 2022 515k
GCN 0.0841±0.0010 Dwivedi et al, NeurIPS 2022 509k
Peptides-func (Graph Classification)
Model Test AP (higher is better) Reference #params
DRew-GCN+LapPE 0.7150±0.0044 Gutteridge et al, ICML 2023 502k
GRIT 0.6988±0.0082 Ma, Lin, et al, ICML 2023 443k
GraphMLP-Mixer 0.6970±0.0080 He et al, ICML 2023 397k
Graph ViT 0.6942±0.0075 He et al, ICML 2023 692k
MGT+WavePE 0.6817±0.0064 Ngo, Hy, et al, 2023 499k
PathNN 0.6816±0.0026 Michel, Nikolentzos et al, ICML 2023 510k
GatedGCN+RWSE+virtual node 0.6685±0.0062 Cai et al, ICML 2023 506k
Cache-GNN+LapPE 0.6671±0.0056 Ma et al, KDD 2023 500k
Graph Diffuser 0.6651±0.0010 Glickman & Yahav, 2023 509k
CIN++ 0.6569±0.0117 Giusti et al, 2023 ~500k
GraphGPS 0.6535±0.0041 Rampášek et al, NeurIPS 2022 504k
Exphormer 0.6527±0.0043 Shirzad, Velingker, Venkatachalam, et al, ICML 2023 446k
SAN+RWSE 0.6439±0.0075 Dwivedi et al, NeurIPS 2022 500k
SAN+LapPE 0.6384±0.0121 Dwivedi et al, NeurIPS 2022 493k
Transformer+LapPE 0.6326±0.0126 Dwivedi et al, NeurIPS 2022 488k
GatedGCN+RWSE 0.6069±0.0035 Dwivedi et al, NeurIPS 2022 506k
GCN 0.5930±0.0023 Dwivedi et al, NeurIPS 2022 508k
GatedGCN 0.5864±0.0077 Dwivedi et al, NeurIPS 2022 509k
GCNII 0.5543±0.0078 Dwivedi et al, NeurIPS 2022 505k
GINE 0.5498±0.0079 Dwivedi et al, NeurIPS 2022 476k
Peptides-struct (Graph Regression)
Model Test MAE (lower is better) Reference #params
Cache-GNN+LapPE 0.2358±0.0013 Ma et al, KDD 2023 500k
Graph ViT 0.2449±0.0016 He et al, ICML 2023 561k
MGT+WavePE 0.2453±0.0025 Ngo, Hy, et al, 2023 499k
GRIT 0.2460±0.0012 Ma, Lin, et al, ICML 2023 439k
Graph Diffuser 0.2461±0.0010 Glickman & Yahav, 2023 509k
Exphormer 0.2481±0.0007 Shirzad, Velingker, Venkatachalam, et al, ICML 2023 426k
GCN+virtual node 0.2488±0.0021 Cai et al, ICML 2023 508k
Graph MLP-Mixer 0.2494±0.0007 He et al, ICML 2023 397k
GraphGPS 0.2500±0.0005 Rampášek et al, NeurIPS 2022 504k
CIN++ 0.2523±0.0013 Giusti et al, 2023 ~500k
Transformer+LapPE 0.2529±0.0016 Dwivedi et al, NeurIPS 2022 488k
DRew-GCN+LapPE 0.2536±0.0015 Gutteridge et al, ICML 2023 495k
SAN+RWSE 0.2545±0.0012 Dwivedi et al, NeurIPS 2022 500k
PathNN 0.2545±0.0032 Michel, Nikolentzos et al, ICML 2023 469k
NPQ+GATv2 0.2589±0.0031 Jain et al, KLR Workshop at ICML, 2023 NA
SAN+LapPE 0.2683±0.0043 Dwivedi et al, NeurIPS 2022 493k
GatedGCN+RWSE 0.3357±0.0006 Dwivedi et al, NeurIPS 2022 506k
GatedGCN 0.3420±0.0013 Dwivedi et al, NeurIPS 2022 509k
GCNII 0.3471±0.0010 Dwivedi et al, NeurIPS 2022 505k
GCN 0.3496±0.0013 Dwivedi et al, NeurIPS 2022 508k
GINE 0.3547±0.0045 Dwivedi et al, NeurIPS 2022 476k
PCQM-Contact (Link Prediction)
Model Test MRR (higher is better) Test Hits@1 Test Hits@3 Test Hits@10 Reference #params
Exphormer 0.3637±0.0020 Shirzad, Velingker, Venkatachalam, et al, ICML 2023 396k
Cache-GNN+RWSE 0.3488±0.0008 0.1463±0.0011 0.4102±0.0008 0.8693±0.0008 Ma et al, KDD 2023 500k
DRew-GCN 0.3444±0.0017 Gutteridge et al, ICML 2023 515k
Graph Diffuser 0.3388±0.0011 0.1369±0.0012 0.4053±0.0011 0.8592±0.0007 Glickman & Yahav, 2023 521k
SAN+LapPE 0.3350±0.0003 0.1355±0.0017 0.4004±0.0021 0.8478±0.0044 Dwivedi et al, NeurIPS 2022 499k
SAN+RWSE 0.3341±0.0006 0.1312±0.0016 0.4030±0.0008 0.8550±0.0024 Dwivedi et al, NeurIPS 2022 509k
GraphGPS 0.3337±0.0006 Rampášek et al, NeurIPS 2022 513k
GatedGCN+RWSE 0.3242±0.0008 0.1288±0.0013 0.3808±0.0006 0.8517±0.0005 Dwivedi et al, NeurIPS 2022 524k
GCN 0.3234±0.0006 0.1321±0.0007 0.3791±0.0004 0.8256±0.0006 Dwivedi et al, NeurIPS 2022 504k
GatedGCN 0.3218±0.0011 0.1279±0.0018 0.3783±0.0004 0.8433±0.0011 Dwivedi et al, NeurIPS 2022 527k
GINE 0.3180±0.0027 0.1337±0.0013 0.3642±0.0043 0.8147±0.0062 Dwivedi et al, NeurIPS 2022 517k
Transformer+LapPE 0.3174±0.0020 0.1221±0.0011 0.3679±0.0033 0.8517±0.0039 Dwivedi et al, NeurIPS 2022 502k
GCNII 0.3161±0.0004 0.1325±0.0009 0.3607±0.0003 0.8116±0.0009 Dwivedi et al, NeurIPS 2022 501k

Citation

If you find this work useful, please cite our paper:

@inproceedings{dwivedi2022LRGB,
  title={Long Range Graph Benchmark}, 
  author={Dwivedi, Vijay Prakash and Rampášek, Ladislav and Galkin, Mikhail and Parviz, Ali and Wolf, Guy and Luu, Anh Tuan and Beaini, Dominique},
  booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2022},
  url={https://openreview.net/forum?id=in7XC5RcjEn}
}