Gaussian Process Self-Distillation (GPSD)

Self-distillation for Gaussian Process Regression and Gaussian Process Classification.

Read the Paper»

About The Project

This is the official implementation of the paper Self-Distillation for Gaussian Processes by Kenneth Borup and Lars N. Andersen.

@misc{borup2023selfdistillation,
      title={Self-Distillation for Gaussian Process Regression and Classification}, 
      author={Kenneth Borup and Lars Nørvang Andersen},
      year={2023},
      eprint={2304.02641},
      archivePrefix={arXiv}
}

The package is designed to be used in a similar way to scikit-learn. The package includes two classes for Gaussian Process Regression and two classes for Gaussian Process Classification.

Getting Started

To get a local copy up and running follow the simple example under Installation. For usage examples, see Usage and be careful of training times for different methods (see Training speed). For more details on the methods, see the paper.

Dependencies

Currently the implementation is reliant on the following dependencies:

• scikit-learn
• torch
• numpy
• scipy

They are all installed by default when installing the package.

Installation

The package can be installed using pip from the github repository or by cloning the repository and installing the package locally.

Directly from github using pip

pip install git+https://github.com/kennethborup/gaussian_process_self_distillation.git

or clone the repository and install locally

git clone https://github.com/kennethborup/gaussian_process_self_distillation.git
cd gaussian_process_self_distillation
pip install .

Verify the installation by running the following command in your terminal window:

python -c "import gpsd; print(f'{gpsd.__version__} by {gpsd.__author__}')"

Usage

The package is designed to be used in a similar way to scikit-learn. The package includes four classes:

• DataCentricGPR for Data-Centric Self-Distillation for Gaussian Process Regression
• DistributionCentricGPR for Distribution-Centric Self-Distillation for Gaussian Process Regression
• DataCentricGPC for Data-Centric Self-Distillation for Gaussian Process Classification
• DistributionCentricGPC for Distribution-Centric Self-Distillation for Gaussian Process Classification

Each method has .fit, and .predict methods that are similar to the scikit-learn API - see the paper for more details on each method.

Examples

Given some data X and y, the following examples fit different GPSD models using a predefined kernel from sklearn and predict on the same data. I.e. first run

import GPSD
from sklearn.gaussian_process.kernels import RBF

X, y = # some data
kernel = 1.0 * RBF()

then run one of the following examples.

Data-Centric GP Regression

model = gpsd.DataCentricGPR(
    kernel=kernel,
    num_distillations=5,
    alphas=1e-2, # If a single value is given, it is used for all distillation steps
    optimize_mode="first", # only perform hyperparameter optimization in the first distillation step
    fit_mode="efficient", # use efficient computation of distillation steps (alternative is "naive")
)
model.fit(X, y)
y_pred = model.predict(X)

Distribution-Centric GP Regression

model = gpsd.DistributionCentricGPR(
    kernel=kernel,
    alphas=[1e-2]*5, # Must be a list of the amount of distillation steps
)
model.fit(X, y)
y_pred = model.predict(X)

Data-Centric GP Classification

model = gpsd.DataCentricGPC(
    kernel=kernel,
    num_distillations=5,
    optimize_mode="first", # only perform hyperparameter optimization in the first distillation step
)
model.fit(X, y)
y_pred = model.predict(X)

Distribution-Centric GP Classification

model = gpsd.DistributionCentricGPC(
    kernel=kernel,
    num_distillations=5,
    fit_mode="approx", # use approximate computation of distillation steps (alternative is "exact")
)
model.fit(X, y)
y_pred = model.predict(X)

Training speed

Note, the training speed of different methods vary a lot (especially for large number of distillation steps). The following plots are training times for each of the methods on a simulated dataset, when fitted on a Mac M1 Pro CPU. The training time is presented relative to an ordinary fit of a GPR/GPC model using sklearn . The training time is measured across 30 replications, and the mean and 10%/90% quantiles are plotted.

Citation

If you use this code in your research, please cite the following paper:

@misc{borup2023selfdistillation,
      title={Self-Distillation for Gaussian Process Regression and Classification}, 
      author={Kenneth Borup and Lars Nørvang Andersen},
      year={2023},
      eprint={2304.02641},
      archivePrefix={arXiv}
}

License

Distributed under the MIT License. See LICENSE.txt for more information.