This repo aims to provide a comprehensive survey of recent papers on Neural Network Pruning at Initialization (PaI). Feel free to add any paper you think is pertinent!
- Survey accepted in IJCAI'22 Survey Track:
- Code Base
2018
- 2018-ICLR-Deep rewiring: Training very sparse deep networks
- 2018-ICML-Stronger generalization bounds for deep nets via a compression approach
- 2018-Nature Communications-Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science
2019
- 2019-ICLR-Snip: Single-shot network pruning based on connection sensitivity
- 2019-ICLR-The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks [Best Paper!] [Code 1] [Code 2]
- 2019-ICML-Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
- 2019-NIPS-Deconstructing lottery tickets: Zeros, signs, and the supermask
- 2019-NIPS-One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers [Code, Code Report]
- 2019.7-Sparse networks from scratch: Faster training without losing performance [ICLR'20 rejected version] [Code]
2020
- 2020-ICLR-Picking Winning Tickets Before Training By Preserving Gradient Flow [Code]
- 2020-ICLR-Playing the lottery with rewards and multiple languages: Lottery tickets in RL and NLP
- 2020-ICLR-Drawing early-bird tickets: Towards more efficient training of deep networks (Spotlight)
- 2020-ICLR-The Early Phase of Neural Network Training
- 2020-ICLR-A signal propagation perspective for pruning neural networks at initialization
- 2020-ICLR-Comparing Rewinding and Fine-tuning in Neural Network Pruning [Code] (Oral)
- 2020-ICLR-Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
- 2020-ICLR reject-Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win
- 2020-CVPR-What's hidden in a randomly weighted neural network?
- 2020-ICML-Proving the Lottery Ticket Hypothesis: Pruning is All You Need
- 2020-ICML-Rigging the Lottery: Making All Tickets Winners [Code]
- 2020-ICML-Linear Mode Connectivity and the Lottery Ticket Hypothesis
- 2020-ICML-Finding trainable sparse networks through neural tangent transfer
- 2020-ICML-Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection [Code]
- 2020-NIPS-Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
- 2020-NIPS-Logarithmic Pruning is All You Need
- 2020-NIPS-Winning the Lottery with Continuous Sparsification
- 2020-NIPS-Movement Pruning: Adaptive Sparsity by Fine-Tuning
- 2020-NIPS-The Lottery Ticket Hypothesis for Pre-trained BERT Networks
- 2020-NIPS-Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
- 2020-Neural Computing and Applications-Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware
- 2020-NIPS-Train-by-Reconnect: Decoupling Locations of Weights from their Values
- 2020-NIPS-Pruning neural networks without any data by iteratively conserving synaptic flow
- 2020.2-Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
- 2020.9-The Hardware Lottery
- 2020.10-The Sooner The Better: Investigating Structure of Early Winning Lottery Tickets
2021
- 2021-ICLR-Pruning Neural Networks at Initialization: Why Are We Missing the Mark?
- 2021-ICLR-Long Live the Lottery- The Existence of Winning Tickets in Lifelong Learning
- 2021-ICLR-Robust Pruning at Initialization
- 2021-ICLR-Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network [PyTorch Code]
- 2021-ICLR-Layer-adaptive Sparsity for the Magnitude-based Pruning [Code]
- 2021-ICLR-Progressive skeletonization: Trimming more fat from a network at initialization
- 2021-CVPR-The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
- 2021-ICML-Lottery Ticket Implies Accuracy Degradation, Is It a Desirable Phenomenon?
- 2021-ICML-A Unified Lottery Ticket Hypothesis for Graph Neural Networks
- 2021-ICML-Do we actually need dense over-parameterization? in-time over-parameterization in sparse training
- 2021-ICML-Efficient Lottery Ticket Finding: Less Data is More
- 2021-NIPS-The Elastic Lottery Ticket Hypothesis [Code]
- 2021-NIPS-Training Neural Networks with Fixed Sparse Masks [Code]
- 2021-NIPS-Validating the Lottery Ticket Hypothesis with Inertial Manifold Theory [Code] (Code link seems invalid now...)
- 2021-NIPS-Sparse Training via Boosting Pruning Plasticity with Neuroregeneration [Code]
- 2021-NIPS-Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
- 2021-ACL-Parameter-Efficient Transfer Learning with Diff Pruning
- 2021.6-Why is Pruning at Initialization Immune to Reinitializing and Shuffling
- 2021.7-Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity
2022
- 2022-ICLR-Dual Lottery Ticket Hypothesis
- 2022-ICLR-The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training
- 2022-ICLR-Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
- 2022-ICLR-Pixelated butterfly: Simple and efficient sparse training for neural network models [Code]
- 2022-ICLR-Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients [Code]
- 2022-AISTATS-Finding Everything within Random Binary Networks
- 2022-NIPS-Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training [Code]
- 2022-NIPS-Advancing Model Pruning via Bi-level Optimization [Code]
- 2022-NIPS-Parameter-Efficient Masking Network [Code]
- 2022-NIPS-Rare Gems: Finding Lottery Tickets at Initialization
- 2022-NIPS-Pruning’s Effect on Generalization Through the Lens of Training and Regularization
- 2022-NIPS-Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks
- 2022-NIPS-Robust Binary Models by Pruning Randomly-initialized Networks
- 2022-NIPS-Most Activation Functions Can Win the Lottery Without Excessive Depth
- 2022-NIPS-Sparse Winning Tickets are Data-Efficient Image Recognizers
- 2022-NIPS-Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspective
- 2022-NIPS-Advancing Model Pruning via Bi-level Optimization
- 2022-NIPS-Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
- 2022-NIPS-Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training
- 2022-NIPS-Exposing and Exploiting Fine-Grained Block Structures for Fast and Accurate Sparse Training
- 2022-NIPS-Where to Pay Attention in Sparse Training for Feature Selection?
- 2022-NIPS-Get More at Once: Alternating Sparse Training with Gradient Correction
- 2022-NIPS-Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks
- 2022-NIPS-SparCL: Sparse Continual Learning on the Edge