coral-implementation-recipe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CORAL Implementation Recipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook provides a brief overview of the relevant parts of CORAL that distinguish from a \"regular\" deep neural network for classification. The purpose of this overview is to provide a succinct resource that may help other researchers in porting the CORAL framework to other, non-PyTorch code bases (such as TensorFlow, Keras, MXnet, etc.). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Implement a function that converts class labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In contrast to regular cross entropy-based classification approaches, CORAL operates on binary tasks rather than integer class labels (or one-hot encoding representations thereof); hence, we have to convert class labels into the respective representation, which is illustrated here. \n",
    "- To provide an example, suppose you have a dataset consisting of 5 classes; consequently, the class labels are 0, 1, 2, 3, and 4.\n",
    "- The following `label_to_levels` function converts class labels into the binary task representation required by CORAL, we call them \"levels:\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "def label_to_levels(label, num_classes):\n",
    "    levels = [1]*label + [0]*(num_classes - 1 - label)\n",
    "    levels = torch.tensor(levels, dtype=torch.float32)\n",
    "    return levels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To continue with the example, assume we have a dataset of 3 training examples with class labels 2, 1, 4\n",
    "- The converted labels would look as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 0., 0.],\n",
       "        [1., 0., 0., 0.],\n",
       "        [1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NUM_CLASSES = 5\n",
    "\n",
    "levels = []\n",
    "class_labels = [2, 1, 4]\n",
    "\n",
    "for label in class_labels:\n",
    "    levels_from_label = label_to_levels(label, num_classes=NUM_CLASSES)\n",
    "    levels.append(levels_from_label)\n",
    "\n",
    "levels = torch.stack(levels)\n",
    "levels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Implement the loss function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As outlined in the paper, the loss function is defined as\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "L(\\mathbf{W}, \\mathbf{b})=& \\\\\n",
    "&-\\sum_{i=1}^{N} \\sum_{k=1}^{K-1} \\lambda^{(k)}\\left[\\log \\left(s\\left(g\\left(\\mathbf{x}_{i}, \\mathbf{W}\\right)+b_{k}\\right)\\right) y_{i}^{(k)}\\right.\\\\\n",
    "&\\left.+\\log \\left(1-s\\left(g\\left(\\mathbf{x}_{i}, \\mathbf{W}\\right)+b_{k}\\right)\\right)\\left(1-y_{i}^{(k)}\\right)\\right]\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- In the paper, we used a uniform task importance weight $\\lambda$ (this means, all binary tasks were treated equally); this can be achieved by using a vectors of 1's as the task importance weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "importance_weights = torch.ones(NUM_CLASSES-1, dtype=torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The loss function itself, based on the equation above, can be implemented as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "\n",
    "def loss_fn1(logits, levels, imp):\n",
    "    val =  -torch.sum((torch.log(torch.sigmoid(logits))*levels + \n",
    "             torch.log(1 - torch.sigmoid(logits))*(1-levels))*imp,\n",
    "           dim=1)\n",
    "    return torch.mean(val)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To apply it to a concrete example, we use the previous \"levels\" and some made-up logit values (these logits values would be the neural network outputs).\n",
    "- Note that the rows represent the training examples, whereas the columns represent the logit value for each binary task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "logits = torch.tensor([[2.1, 1.8, -2.1, -1.8],\n",
    "                       [1.9, -1., -1.5, -1.3],\n",
    "                       [1.9, 1.8, 1.7, 1.6]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.6920)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_fn1(logits=logits, \n",
    "         levels=levels,\n",
    "         imp=importance_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, we found the loss function can be numerically more stable if we rewrite \n",
    "\n",
    "\n",
    "(1) \n",
    "\n",
    "```python\n",
    "torch.log(torch.sigmoid(logits))*levels\n",
    "```\n",
    "\n",
    "as \n",
    "\n",
    "```python \n",
    "F.logsigmoid(logits)*levels\n",
    "```\n",
    "\n",
    "and (2)\n",
    "\n",
    "```python \n",
    "torch.log(1 - torch.sigmoid(logits))*(1-levels)\n",
    "```\n",
    "\n",
    "as \n",
    "\n",
    "```python \n",
    "(F.logsigmoid(logits) - logits)*(1-levels)\n",
    "```\n",
    "\n",
    "Note that (2) if valid since\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "log\\bigg(\\frac{e^x}{1+e^x}\\bigg) - x &= log\\bigg(\\frac{e^x}{1+e^x}\\bigg) - log(e^x)\\\\\n",
    "&= log\\bigg(\\frac{1}{1+e^x}\\bigg)\\\\\n",
    "&= log\\bigg(1-\\frac{e^x}{1+e^x}\\bigg)\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Hence, in practice, we recommend using the following loss function (which produces the same results as the one outlined above):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_fn2(logits, levels, imp):\n",
    "    val = (-torch.sum((F.logsigmoid(logits)*levels\n",
    "                      + (F.logsigmoid(logits) - logits)*(1-levels))*imp,\n",
    "           dim=1))\n",
    "    return torch.mean(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.6920)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_fn2(logits=logits, \n",
    "         levels=levels,\n",
    "         imp=importance_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Modify the neural network architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The modification that has to be made to an existing deep neural network classifier is relatively simple as it only affects the last layer (i.e., output layer). In particular, the last fully connected layer has to be changed.\n",
    "\n",
    "In PyTorch, this means \n",
    "\n",
    "(1) changing the last fully connected layer\n",
    "\n",
    "```python\n",
    "...\n",
    "self.fc = nn.Linear(input_size, num_classes)\n",
    "```\n",
    "\n",
    "to\n",
    "\n",
    "```python\n",
    "...\n",
    "self.fc = nn.Linear(input_size, 1, bias=False)\n",
    "self.linear_1_bias = nn.Parameter(torch.zeros(self.num_classes-1).float())\n",
    "```\n",
    "\n",
    "(2) and changing the forward pass from\n",
    "\n",
    "```python\n",
    "...\n",
    "logits = self.fc(x)\n",
    "probas = F.softmax(logits, dim=1)\n",
    "return logits, probas\n",
    "```\n",
    "        \n",
    "to\n",
    "\n",
    "```python\n",
    "logits = self.fc(x) + self.linear_1_bias\n",
    "probas = torch.sigmoid(logits)\n",
    "return logits, probas\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test with pre-defined weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Below is an example that can be used to test the implementation given some pre-defined weights (these are usually randomly initialized in a real application) and layer inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logits:\n",
      "1st training example: tensor([-0.5699, -0.5699, -0.5699, -0.5699], grad_fn=<SelectBackward>)\n",
      "2nd training example: tensor([-0.5699, -0.5699, -0.5699, -0.5699], grad_fn=<SelectBackward>)\n",
      "\n",
      "\n",
      "Probas:\n",
      "1st training example: tensor([0.3613, 0.3613, 0.3613, 0.3613], grad_fn=<SelectBackward>)\n",
      "2nd training example: tensor([0.3613, 0.3613, 0.3613, 0.3613], grad_fn=<SelectBackward>)\n"
     ]
    }
   ],
   "source": [
    "num_classes = 5\n",
    "\n",
    "###################################################\n",
    "# initialize fully connected layer without bias\n",
    "fc = torch.nn.Linear(10, 1, bias=False)\n",
    "\n",
    "# overwrite weight for reproducibility\n",
    "fc.weight = torch.nn.Parameter(\n",
    "             torch.tensor([[ 0.3019, -0.1284, \n",
    "                            -0.2542, -0.0886,  \n",
    "                            0.1354,  0.1257, \n",
    "                            -0.2183, -0.2714,\n",
    "                            -0.1859,  0.0035]], \n",
    "                          requires_grad=True)\n",
    ")\n",
    "####################################################\n",
    "\n",
    "# Initialize bias\n",
    "linear_1_bias = torch.nn.Parameter(torch.zeros(num_classes-1).float())\n",
    "\n",
    "# Initialize 2 random training layer inputs\n",
    "example_inputs = torch.tensor([[0.2769, 0.5538, 0.8187, 0.5455, \n",
    "                                0.3085, 0.7225, 0.7978, 0.8664, 0.2754, 0.5541],\n",
    "                               [0.4267, 0.5409, 0.2538, 0.1381, \n",
    "                                0.3889, 0.7500, 0.0249, 0.1347, 0.0494, 0.0935]])\n",
    "\n",
    "# Compute outputs\n",
    "logits = fc(example_inputs) + linear_1_bias\n",
    "probas = torch.sigmoid(logits)\n",
    "\n",
    "print('Logits:')\n",
    "print('1st training example:', logits[0])\n",
    "print('2nd training example:', logits[0])\n",
    "\n",
    "print('\\n\\nProbas:')\n",
    "print('1st training example:', probas[0])\n",
    "print('2nd training example:', probas[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note that for each of the 2 training examples, the values are the same for all output units, because the network hasn't learned the bias parameter, yet. Assuming the learned bias parameter vector is [0, 1, 2, 3], you should see different outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logits:\n",
      "1st training example: tensor([-0.5699,  0.4301,  1.4301,  2.4301], grad_fn=<SelectBackward>)\n",
      "2nd training example: tensor([-0.5699,  0.4301,  1.4301,  2.4301], grad_fn=<SelectBackward>)\n",
      "\n",
      "\n",
      "Probas:\n",
      "1st training example: tensor([0.3613, 0.6059, 0.8069, 0.9191], grad_fn=<SelectBackward>)\n",
      "2nd training example: tensor([0.3613, 0.6059, 0.8069, 0.9191], grad_fn=<SelectBackward>)\n"
     ]
    }
   ],
   "source": [
    "linear_1_bias = torch.nn.Parameter(torch.tensor([0., 1., 2., 3.]))\n",
    "logits = fc(example_inputs) + linear_1_bias\n",
    "probas = torch.sigmoid(logits)\n",
    "\n",
    "print('Logits:')\n",
    "print('1st training example:', logits[0])\n",
    "print('2nd training example:', logits[0])\n",
    "\n",
    "print('\\n\\nProbas:')\n",
    "print('1st training example:', probas[0])\n",
    "print('2nd training example:', probas[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4) Evaluate the neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing performance metrics such as the mean absolute error, or simply obtaining class labels, requires a small modification. Whereas in cross entropy-based classifiers, we obtain the class labels via\n",
    "\n",
    "```python\n",
    "logits, probas = model(features)\n",
    "_, predicted_labels = torch.max(probas, 1)\n",
    "```\n",
    "\n",
    "we can change these lines to\n",
    "\n",
    "```python\n",
    "logits, probas = model(features)\n",
    "predict_levels = probas > 0.5\n",
    "predicted_labels = torch.sum(predict_levels, dim=1)\n",
    "```\n",
    "\n",
    "in CORAL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the `probas` from the example above, *4) Evaluate the neural network*, we can compute the predicted labels as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3, 4])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_levels = probas > 0.5\n",
    "predicted_labels = torch.sum(predict_levels, dim=1)\n",
    "predicted_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5) Example Run and Code Checks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Below is an example running the CORAL layer in a very simple convolutional neural network on MNIST.\n",
    "- Note that MNIST is not an ordinal classification task; however, we use it here for simplicity, since it's a dataset that is built into most deep learning frameworks already and serves as a good start for debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image batch dimensions: torch.Size([128, 1, 28, 28])\n",
      "Image label dimensions: torch.Size([128])\n"
     ]
    }
   ],
   "source": [
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Hyperparameters\n",
    "random_seed = 1\n",
    "learning_rate = 0.05\n",
    "num_epochs = 2\n",
    "batch_size = 128\n",
    "\n",
    "# Architecture\n",
    "NUM_CLASSES = 10\n",
    "\n",
    "# Other\n",
    "importance_weights = torch.ones(NUM_CLASSES-1, dtype=torch.float)\n",
    "\n",
    "\n",
    "##########################\n",
    "### MNIST DATASET\n",
    "##########################\n",
    "\n",
    "\n",
    "# Note transforms.ToTensor() scales input images\n",
    "# to 0-1 range\n",
    "train_dataset = datasets.MNIST(root='data', \n",
    "                               train=True, \n",
    "                               transform=transforms.ToTensor(),\n",
    "                               download=True)\n",
    "\n",
    "test_dataset = datasets.MNIST(root='data', \n",
    "                              train=False, \n",
    "                              transform=transforms.ToTensor())\n",
    "\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=batch_size, \n",
    "                          drop_last=True,\n",
    "                          shuffle=True)\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=batch_size, \n",
    "                         drop_last=True,\n",
    "                         shuffle=False)\n",
    "\n",
    "# Checking the dataset\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.shape)\n",
    "    print('Image label dimensions:', labels.shape)\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ConvNet(torch.nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes):\n",
    "        super(ConvNet, self).__init__()\n",
    "        \n",
    "        \n",
    "        self.conv_1 = torch.nn.Conv2d(in_channels=1,\n",
    "                                      out_channels=3,\n",
    "                                      kernel_size=(3, 3),\n",
    "                                      stride=(1, 1),\n",
    "                                      padding=1)\n",
    "\n",
    "        self.pool_1 = torch.nn.MaxPool2d(kernel_size=(2, 2),\n",
    "                                         stride=(2, 2),\n",
    "                                         padding=0)                                \n",
    "\n",
    "        self.conv_2 = torch.nn.Conv2d(in_channels=3,\n",
    "                                      out_channels=6,\n",
    "                                      kernel_size=(3, 3),\n",
    "                                      stride=(1, 1),\n",
    "                                      padding=1)           \n",
    "                  \n",
    "        self.pool_2 = torch.nn.MaxPool2d(kernel_size=(2, 2),\n",
    "                                         stride=(2, 2),\n",
    "                                         padding=0)\n",
    "\n",
    "        #######################\n",
    "        ##### CORAL LAYER #####\n",
    "        ###---------------------------------------------------------------------------###\n",
    "        self.fc = torch.nn.Linear(294, 1, bias=False)\n",
    "        self.linear_1_bias = torch.nn.Parameter(torch.zeros(num_classes-1).float())\n",
    "        ###---------------------------------------------------------------------------###\n",
    "\n",
    "\n",
    "        \n",
    "    def forward(self, x):\n",
    "        out = self.conv_1(x)\n",
    "        out = F.relu(out)\n",
    "        out = self.pool_1(out)\n",
    "\n",
    "        out = self.conv_2(out)\n",
    "        out = F.relu(out)\n",
    "        out = self.pool_2(out)\n",
    "        \n",
    "        # flatten\n",
    "        out = out.view(out.size(0), -1)\n",
    "        \n",
    "        #######################\n",
    "        ##### CORAL LAYER #####\n",
    "        ###--------------------------------------------------------------------###\n",
    "        logits = self.fc(out) + self.linear_1_bias\n",
    "        probas = torch.sigmoid(logits)\n",
    "        ###--------------------------------------------------------------------###\n",
    "        \n",
    "        return logits, probas\n",
    "\n",
    "    \n",
    "torch.manual_seed(random_seed)\n",
    "model = ConvNet(num_classes=NUM_CLASSES)\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/002 | Batch 000/468 | Cost: 6.2465\n",
      "Epoch: 001/002 | Batch 150/468 | Cost: 4.8635\n",
      "Epoch: 001/002 | Batch 300/468 | Cost: 4.6088\n",
      "Epoch: 001/002 | Batch 450/468 | Cost: 3.9067\n",
      "Epoch: 002/002 | Batch 000/468 | Cost: 4.1401\n",
      "Epoch: 002/002 | Batch 150/468 | Cost: 4.0048\n",
      "Epoch: 002/002 | Batch 300/468 | Cost: 3.5715\n",
      "Epoch: 002/002 | Batch 450/468 | Cost: 3.6428\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(num_epochs):\n",
    "    \n",
    "    model = model.train()\n",
    "    for batch_idx, (features, class_labels) in enumerate(train_loader):\n",
    "\n",
    "        \n",
    "        ###################################\n",
    "        ##### CORAL LABEL CONVERSION #####\n",
    "        ###------------------------------START-----------------------------------###\n",
    "        levels = []\n",
    "        for label in class_labels:\n",
    "            levels_from_label = label_to_levels(label, num_classes=NUM_CLASSES)\n",
    "            levels.append(levels_from_label)\n",
    "\n",
    "        levels = torch.stack(levels)\n",
    "        ###------------------------------END-------------------------------------###\n",
    "\n",
    "        ### FORWARD AND BACK PROP\n",
    "        logits, probas = model(features)\n",
    "        \n",
    "        \n",
    "        ###################################\n",
    "        ##### CORAL LOSS ##################\n",
    "        ###------------------------------START-----------------------------------###  \n",
    "        cost = loss_fn2(logits, levels, imp=importance_weights)\n",
    "        ###------------------------------END-------------------------------------###        \n",
    "        \n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        cost.backward()\n",
    "        \n",
    "        ### UPDATE MODEL PARAMETERS\n",
    "        optimizer.step()\n",
    "        \n",
    "        ### LOGGING\n",
    "        if not batch_idx % 150:\n",
    "            print ('Epoch: %03d/%03d | Batch %03d/%03d | Cost: %.4f' \n",
    "                   %(epoch+1, num_epochs, batch_idx, \n",
    "                     len(train_loader), cost))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sanity checks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Both the levels (converted labels) and logits (model outputs) should have the same dimension: `batch_size x num_classes-1`. In this case, 128 x 9 (since there are 10 classes in MNIST)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logits size: torch.Size([128, 9])\n",
      "Levels size: torch.Size([128, 9])\n"
     ]
    }
   ],
   "source": [
    "print(f\"Logits size: {logits.size()}\")\n",
    "print(f\"Levels size: {levels.size()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the weight parameters are shared among all output tasks, the number of parameters should be equal to the number of input parameters, in this case 294."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input, params: 294\n",
      "Outputs, params: 1\n"
     ]
    }
   ],
   "source": [
    "print(f\"Input, params: {model.fc.weight.size(1)}\")\n",
    "print(f\"Outputs, params: {model.fc.weight.size(0)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The bias should be equal to NUM_CLASSES-1, that is, 9 in this case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of bias units: torch.Size([9])\n"
     ]
    }
   ],
   "source": [
    "print(f\"Number of bias units: {model.linear_1_bias.size()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The learned bias values of the CORAL layer should now be in descending order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([ 0.8481,  0.8062,  0.7222,  0.4642, -0.0421, -0.4887, -0.7205, -0.8064,\n",
       "        -0.8451], requires_grad=True)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.linear_1_bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3yV9d3/8dcnO2FkM5MIBARZgkaUodW6cGKxVWi9q62tHY5ae1dte9th7U9re9fVYa3WWltFRa2zWrciQ4IgUyABkTAT9kgCCZ/fH7ngDjEBQk64cnLez8fjPHLONc55wwPOO9f6XubuiIhI7IoLO4CIiIRLRSAiEuNUBCIiMU5FICIS41QEIiIxLiHsAIcjJyfHe/XqFXYMEZGoMmvWrAp3z204PSqLoFevXhQXF4cdQ0QkqpjZisama9eQiEiMUxGIiMQ4FYGISIxTEYiIxDgVgYhIjFMRiIjEOBWBiEiMi8rrCA7XMx+WsX5bNfmZaeRnpZKfmUZGWiJmFnY0EZHQxFQRvDh3DW9+vH6/aR2TE8jLTCU/K22/gsjPqnuelhRTf0UiEoNi6lvur1ecwNaq3azcuJOVGysp27Sz7vmmSj6p2MGUpRVU7q7db53sDknkZaWR30hZ9MhIJSlBe9dEJLrFVBEAdE5JZFCPdAb1SP/MPHenYvuuuoLYVMnKjTuDsqhk3qotvDJ/LTV7/u+ObmbQrXMK+Zlp5NXfkghKo2vnFOLjtNtJRNq2mCuCAzEzcjslk9spmeEFmZ+ZX7vHWbu1KtiiqCuLso07WblpJ1NLNrBu2yrq3/kzMd7omVFXCnkNdztlppLVIUnHJ0QkdCqCZoiPq/ti75mRykl9sj8zv7qmllWbKvdtTazctJOyjZWs3LST+avWsGnn7v2WT0uK37erqV/XTozsk01Rr0wdlxCRI0rfOBGUnBBPn9yO9Mnt2Oj87dU1we6m/yuKvccq3l5czp/eLiUx3hien8nIwmxGFWYzrCCD5IT4I/wnEZFYYl5/X0aUKCoq8vY2DPWO6hqKV2xiamkF00o3MG/VFtwhJTGOE3plBcWQw+AenUmI1wFqEWk+M5vl7kWfma4iaJu27NzNjOUbmFq6gWmlG1i8bhsAnZITOLFPFif1qSuGAd06EacD0iJyCJoqAu0aaqPS0xI5a1A3zhrUDYDybdVMX1ZXDNOXbeD1RXXXQ2SmJTKyMJuRhTmMKsymT04HHYAWkWaJyBaBmY0F7gHigQfd/Y4G8+8CTgtepgFd3D0jmFcLzAvmferuFx7s82Jhi+BgVm+uZFppXTFMLa1gzZYqALp2TmZUYc6+Ywx5mWkhJxWRtqLVdg2ZWTywBDgTKANmAhPdfWETy18LDHf3rwevt7t740dXm6Ai2J+7s2LDzn2lMK10Axt27AKgICuNUYXZdVsNfbLp0jkl5LQiEpbW3DU0Aihx92XBB00CxgGNFgEwEfhZBD5XAmZGr5wO9MrpwJdPLMDdWbJu+75SeHneGibNXAlA3y4dGRVsLZzUJ5uMtKSQ04tI2CJRBD2BlfVelwEnNragmR0F9AberDc5xcyKgRrgDnf/VxPrXgVcBVBQUBCB2O2XmdG/Wyf6d+vE10b3pnaPs3D1VqaWVjC1dANPFZfx92krMIOB3TsHxZDDCb2z6Jisw0YisSYS/+sbOzLZ1P6mCcBkd68/oE+Bu682sz7Am2Y2z91LP/OG7g8AD0DdrqGWho4l8XHGkLx0huSl863PFbKrZg9zyzbv25X0yNQV/OW95cTHGcfmpTMqOPB83FGZpCTqGgaR9i4SRVAG5Nd7nQesbmLZCcDV9Se4++rg5zIzexsYDnymCCRykhLiKOqVRVGvLK47vR9Vu2uZFVzDMLV0A396p5Tfv1VCUkIcZxzThV9dNITMDtqFJNJeRaIIZgL9zKw3sIq6L/svN1zIzPoDmcC0etMygZ3uXm1mOcBo4M4IZJJmSEmMZ3TfHEb3zQFgW9Vuij/ZxLtLy/nn9E+ZWzaF+y87nsE9PztQn4hEvxZfouruNcA1wKvAIuBJd19gZreaWf1TQScCk3z/05SOAYrN7CPgLeqOETR1kFmOkE4piZw2oAs/u2AQT3zrJGpqnYv/NJVnZ5eFHU1EWoGuLJaDKt9WzdWPfcgHyzdyxahe/OS8Y0jUMBciUaep00f1v1kOKrdTMv/8xol8fXRv/jb1E77ylxms31YVdiwRiRAVgRySxPg4fnrBQO6+dBhzV23mgvum8OGnm8KOJSIRoCKQZrloeE+e/s4okhLiuPTP03hsxqdhRxKRFlIRSLMN6pHOC9eMYWRhDj9+dh43Pz2Xqgb3ehaR6KEikMOSkZbEw1ecwNWnFTJp5kou/fM0Vm+uDDuWiBwGFYEctvg444dnD+D+y46ntHwHF9w3hWmlG8KOJSLNpCKQFhs7uBv/uno06WmJXPbQDB58bxnReFqySKxSEUhE9O3SkeeuHs3pA7pw20uL+N6kOezcVRN2LBE5BCoCiZhOKYncf9nx/PDs/rwwdzXj/ziVFRt2hB1LRA5CRSARFRdnXH1aXx6+4gTWbKnigvum8Nbi9WHHEpEDUBFIqzi1fxdeuGYMPTJS+frfZnLfG0vZs0fHDUTaIhWBtJqC7DSe+e4oLjy2B//72hK+9Y9ZbKvaHXYsEWlARSCtKi0pgbsvHcYt5w/kzY/XM+4P71OyflvYsUSkHhWBtDoz48oxvfnHlSeyZeduxv3+fV6ZvybsWCISUBHIETOyMJsXrxtD366d+PY/PuTOVz6mVscNREKnIpAjqnt6Kk9+6yQmnJDPH98u5YqHP2DTjl1hxxKJaSoCOeKSE+K54+Kh3D5+CDOWbeSC309hweotYccSiVkqAgnNxBEF+90K81+zV4UdSSQmqQgkVMMLMnnh2jEMzcvg+ifm8IsXFrC7dk/YsURiiopAQlf/VpgPv/8JX3lwBuXbqsOOJRIzIlIEZjbWzBabWYmZ3dzI/CvMrNzM5gSPb9Sbd7mZLQ0el0cij0Sf/W6FWbaZ8+97T7fCFDlCWlwEZhYP/AE4BxgITDSzgY0s+oS7DwseDwbrZgE/A04ERgA/M7PMlmaS6FX/VpgT/jxdt8IUOQIisUUwAihx92XuvguYBIw7xHXPBl5z943uvgl4DRgbgUwSxfbeCvOkwmzdClPkCIhEEfQEVtZ7XRZMa+hiM5trZpPNLL+Z62JmV5lZsZkVl5eXRyC2tGWfuRXmA9N1K0yRVhKJIrBGpjW8XPQFoJe7DwVeBx5pxrp1E90fcPcidy/Kzc097LASPerfCrNk3TbdClOklUSiCMqA/Hqv84DV9Rdw9w3uvvc0kL8Axx/quiJjB3fjuWt0K0yR1hKJIpgJ9DOz3maWBEwAnq+/gJl1r/fyQmBR8PxV4CwzywwOEp8VTBPZT98unfa7FeaNk+eqDEQipMVF4O41wDXUfYEvAp509wVmdquZXRgsdp2ZLTCzj4DrgCuCdTcCv6SuTGYCtwbTRD5j760wv3NqIU/NKuO5Odp4FIkEi8bfqoqKiry4uDjsGBKS2j11Q1Ks2LCD12/4HNkdk8OOJBIVzGyWuxc1nK4riyXqxMcZd35xKNura/jliwvDjiMS9VQEEpWO7tqJ757al3/NWc1bi9eHHUckqqkIJGp997RC+nXpyE+emcf26pqw44hELRWBRK299zVYs7WK3766OOw4IlFLRSBR7fijMrl8ZC8emfYJs1ZokDqRw6EikKj3w7P70yM9lZuenkt1jcYkEmkuFYFEvQ7JCdz2hcGUrN/OH94qDTuOSNRREUi7cFr/Llw0rAd/eruExWu3hR1HJKqoCKTd+OkFg+iUkshNT8+ldk/0XSgpEhYVgbQbWR2S+NkFA5mzcjOPTP0k7DgiUUNFIO3Khcf24LT+ufzm1cWs3Lgz7DgiUUFFIO2KmXHbF4YQZ/DjZ+dphFKRQ6AikHanZ0YqN44dwHtLK3h29qqw44i0eSoCaZf+66SjOP6oTG59cSEV26sPvoJIDFMRSLsUF2f8+uIh7Kyu5RcvaIRSkQNREUi71bdLJ675fF9e+Gg1byxaF3YckTZLRSDt2rc/V0j/rp34n3/NZ1vV7rDjiLRJKgJp15IS4rjj4iGs3VrFna9ohFKRxqgIpN0bXpDJ10b15tHpK5j5iW6JLdKQikBiwn+ffTR5mXUjlFbt1gilIvVFpAjMbKyZLTazEjO7uZH5N5jZQjOba2ZvmNlR9ebVmtmc4PF8JPKINJSWlMD/+8IQlpXv4PdvloQdR6RNaXERmFk88AfgHGAgMNHMBjZYbDZQ5O5DgcnAnfXmVbr7sOBxYUvziDTllKNzGX9cT+5/p5RFa7aGHUekzYjEFsEIoMTdl7n7LmASMK7+Au7+lrvvHfhlOpAXgc8VabZbzhtIemoiN2uEUpF9IlEEPYGV9V6XBdOaciXw73qvU8ys2Mymm9lFTa1kZlcFyxWXl5e3LLHErMwOSfz8wkF8VLaFh99fHnYckTYhEkVgjUxr9FctM7sMKAJ+U29ygbsXAV8G7jazwsbWdfcH3L3I3Ytyc3Nbmlli2PlDu3P6gC789j+L+XSDRigViUQRlAH59V7nAasbLmRmZwA/AS50932Dv7j76uDnMuBtYHgEMok0qW6E0sEkxMVphFIRIlMEM4F+ZtbbzJKACcB+Z/+Y2XDgz9SVwPp60zPNLDl4ngOMBjQwjLS67ump3HTOAKaUVDB5VlnYcURC1eIicPca4BrgVWAR8KS7LzCzW81s71lAvwE6Ak81OE30GKDYzD4C3gLucHcVgRwRXxlRwAm9MrntpUWUb9MIpRK7LBo3i4uKiry4uDjsGNIOlKzfzrn3vMeZA7vyh68cF3YckVZlZrOCY7L70ZXFEtP6dunIdaf35aV5a/jPgrVhxxEJhYpAYt63PlfIgG6duOW5+WzVCKUSg1QEEvMS4+O484tDKd9WzR3//jjsOCJHnIpABBial8GVY3rz2IxPmb5sQ9hxRI4oFYFI4PtnHk1+Vio/emaeRiiVmKIiEAmkJSVw+xeGsrxiB/e+sTTsOCJHjIpApJ4x/XL40vF5/PndZSxYvSXsOCJHhIpApIGfnHcMmWlJ3PT0XGpq94QdR6TVqQhEGshIS+IXFw5i/qqtPDRFI5RK+6ciEGnEuUO6cebArvzutSV8UrEj7DgirUpFINIIM+OX4waTFK8RSqX9UxGINKFbego/OvcYppZu4MnilQdfQSRKqQhEDmDCCfmM6J3FbS8tYv3WqrDjiLQKFYHIAcTFGXeMH0J1zR5++tyCsOOItAoVgchB9MntyPVn9OOVBWt5Zf6asOOIRJyKQOQQfPPkPgzs3pmfPreALZUaoVTaFxWByCFIjI/j1xcPpWJ7Nbe/vCjsOCIRpSIQOURD8tL55sl9mDRzJVNLK8KOIxIxKgKRZrj+jKM5KjtNI5RKu6IiEGmG1KR4bh8/hBUbdnLX60vCjiMSEREpAjMba2aLzazEzG5uZH6ymT0RzJ9hZr3qzftRMH2xmZ0diTwirWlUYQ4TTsjnwfeWM3+VRiiV6NfiIjCzeOAPwDnAQGCimQ1ssNiVwCZ37wvcBfw6WHcgMAEYBIwF/hi8n0ib9qNzjiGrQxI3Tp7Lbo1QKlEuElsEI4ASd1/m7ruAScC4BsuMAx4Jnk8GTjczC6ZPcvdqd18OlATvJ9Kmpacl8stxg1i4ZisPvqcRSiW6RaIIegL1B2IpC6Y1uoy71wBbgOxDXBcAM7vKzIrNrLi8vDwCsUVaZuzg7owd1I27X1/Cco1QKlEsEkVgjUxrOFRjU8scyrp1E90fcPcidy/Kzc1tZkSR1nHruEEkJcRx89Nz2bNHI5RKdIpEEZQB+fVe5wGrm1rGzBKAdGDjIa4r0mZ16ZzCT849hhnLNzJppkYolegUiSKYCfQzs95mlkTdwd/nGyzzPHB58PyLwJteN8D788CE4Kyi3kA/4IMIZBI5Yi49IZ+RfbK5/eVFrN2iEUol+rS4CIJ9/tcArwKLgCfdfYGZ3WpmFwaLPQRkm1kJcANwc7DuAuBJYCHwCnC1u+sqHYkqZsbt44ewq3YPtzw3Xzexkahj0fiPtqioyIuLi8OOIbKfv7y7jF+9vIjbxw9h4oiCsOOIfIaZzXL3oobTdWWxSIRcOaY3Y/rm8IsXFrB03baw44gcMhWBSITExRm/u+RYOiQlcO3jszUWkUQNFYFIBHXpnML/XnIsH6/dxq9e0nDVEh1UBCIRdmr/Lnzz5N48On0Fr8xfG3YckYNSEYi0gh+ePYAhPdO56em5rNpcGXYckQNSEYi0gqSEOO6bOJya2j1cP2k2NRqYTtowFYFIK+mV04HbvjCYmZ9s4r43S8KOI9IkFYFIK/rC8DzGH9eT+95cyvRlG8KOI9IoFYFIK7t13GCOyu7A9ZPmsGnHrrDjiHyGikCklXVMTuDeCcPZsKOaH06eqyEopM1REYgcAUPy0rlp7ABeX7SOv09bEXYckf2oCESOkCvH9Oa0/rn86uVFLFy9New4IvuoCESOEDPjt186lozURK59/EN27qoJO5IIoCIQOaKyOyZz16XDWFaxg188vzDsOCKAikDkiBvdN4fvfK6QJ4pX8sJHuiGfhE9FIBKC7595NMMLMvjxM/NYuXFn2HEkxqkIREKQGB/HvROGg8G1j89mt4agkBCpCERCkp+Vxh3jhzJn5WZ+99qSsONIDFMRiITovKHdmTgin/vfKWXK0oqw40iMUhGIhOyn5w+iMLcj339yDhXbq8OOIzGoRUVgZllm9pqZLQ1+ZjayzDAzm2ZmC8xsrpldWm/e38xsuZnNCR7DWpJHJBqlJsXz+y8PZ0vlbn7w5Efs2aMhKOTIaukWwc3AG+7eD3gjeN3QTuCr7j4IGAvcbWYZ9eb/0N2HBY85LcwjEpUGdOvMLecdwztLyvnr+8vDjiMxpqVFMA54JHj+CHBRwwXcfYm7Lw2erwbWA7kt/FyRdueyk47i7EFd+fUrHzOvbEvYcSSGtLQIurr7GoDgZ5cDLWxmI4AkoLTe5F8Fu4zuMrPkA6x7lZkVm1lxeXl5C2OLtD1mxq8vHkpOx2SuffxDtldrCAo5Mg5aBGb2upnNb+QxrjkfZGbdgUeBr7n73pOmfwQMAE4AsoCbmlrf3R9w9yJ3L8rN1QaFtE8ZaUncM2E4n27cyS3/mh92HIkRCQdbwN3PaGqema0zs+7uvib4ol/fxHKdgZeA/3H36fXee03wtNrMHgb+u1npRdqhEb2zuO70ftz9+lLG9M3h4uPzwo4k7VxLdw09D1wePL8ceK7hAmaWBDwL/N3dn2owr3vw06g7vqBfgUSAaz/fjxG9s7jlufksK98edhxp51paBHcAZ5rZUuDM4DVmVmRmDwbLXAKcAlzRyGmi/zSzecA8IAe4rYV5RNqF+DjjngnDSEqI47pJs6muqQ07krRjFo23zSsqKvLi4uKwY4i0uv8sWMtVj87iyjG9ueX8gWHHkShnZrPcvajhdF1ZLNKGnTWoG18deRQPTVnOWx83eghOpMVUBCJt3I/PPYYB3Trxg6c+Yt3WqrDjSDukIhBp41IS64agqNxVy/efmEOthqCQCFMRiESBvl068fMLBzK1dAP3v1N68BVEmkFFIBIlLinK57yh3fnda0uYtWJT2HGkHVERiEQJM+P28UPonp7CdY/PZkvl7rAjSTuhIhCJIp1TErl34nDWbq3ix8/MIxpP/5a2R0UgEmWOK8jkB2cdzUvz1vDEzJVhx5F2QEUgEoW+fUohY/rm8PMXFrB03baw40iUUxGIRKG4OON3lxxLh6QErn18NlW7NQSFHD4VgUiU6tI5hd9eciwfr93Gr15aFHYciWIqApEodlr/LnxjTG8enb6CV+avDTuORCkVgUiUu3HsAIb0TOemp+eyenNl2HEkCqkIRKJcUkIc904cTk3tHq6fNIea2j0HX0mkHhWBSDvQO6cDv7xoMB98spH73iwJO45EGRWBSDsx/rg8xg/vyX1vLmX6sg1hx5EooiIQaUduvWgwBVlpXD9pDpt27Ao7jkQJFYFIO9IxOYH7Jh7Hhh3V3Pj0XA1BIYdERSDSzgzJS+emsQN4beE6Hp2+Iuw4EgVUBCLt0NdH9+bU/rnc9tIiFq7eGnYcaeNaVARmlmVmr5nZ0uBnZhPL1ZrZnODxfL3pvc1sRrD+E2aW1JI8IlInLs747ZeOJT01kWsf/5Cdu2rCjiRtWEu3CG4G3nD3fsAbwevGVLr7sOBxYb3pvwbuCtbfBFzZwjwiEsjpmMzdlw5jWcUOfvrcAvboFpfShJYWwTjgkeD5I8BFh7qimRnweWDy4awvIgc3um8O15zWl8mzyvjqXz9g3daqsCNJG9TSIujq7msAgp9dmlguxcyKzWy6me39ss8GNrv73m3WMqBnC/OISAM3nHk0t48fwqwVmzj77nc1JpF8RsLBFjCz14Fujcz6STM+p8DdV5tZH+BNM5sHNHYEq8ltVzO7CrgKoKCgoBkfLRLbzIyJIwoY0TuL6yfN4dv/mMXEEfnccv5A0pIO+hUgMeCgWwTufoa7D27k8Rywzsy6AwQ/1zfxHquDn8uAt4HhQAWQYWZ7/yXmAasPkOMBdy9y96Lc3Nxm/BFFBKAwtyNPf2cU3zm1kEkzV3L+vVOYW7Y57FjSBrR019DzwOXB88uB5xouYGaZZpYcPM8BRgMLve5Kl7eALx5ofRGJnKSEOG4aO4DHvnESlbtrGf/Hqfzx7RJqdSA5prW0CO4AzjSzpcCZwWvMrMjMHgyWOQYoNrOPqPviv8PdFwbzbgJuMLMS6o4ZPNTCPCJyCEYWZvPK907h7EHduPOVxXz5L9M1hHUMs2i8BL2oqMiLi4vDjiES9dydybPK+PnzC4iPM/7f+CGcP7RH2LGklZjZLHcvajhdVxaLxDAz40tF+bx03cn0ye3INY/N5gdPfsT2al2AFktUBCJCr5wOPPXtkVz3+b48O7uMc+95jw8/3RR2LDlCVAQiAkBifBw3nNWfJ741kto9zpfun8Y9ry/VHc9igIpARPZzQq8s/n39yVwwtDt3vb6ESx+YzsqNO8OOJa1IRSAin9E5JZG7Jwzn7kuHsWTtNs655z2enV0WdixpJSoCEWnSRcN78vL3TmZAt058/4mP+N6k2Wyp3B12LIkwFYGIHFB+VhqTrjqJG848mhfnruHce97jg+Ubw44lEaQiEJGDSoiP47rT+/HUt0cSH2dMeGAa//ufxezWgeR2QUUgIofsuIJMXv7eyYw/Lo/73izhi/dP45OKHWHHkhZSEYhIs3RMTuC3XzqW3395OMvLt3Puve/xZPFKonGUAqmjIhCRw3L+0B68cv0pDOmZzo2T53LNY7PZvHNX2LHkMKgIROSw9chI5bFvnsRNYwfw6oK1nHPPe0wtrQg7ljSTikBEWiQ+zvjOqYU8+93RpCbG85UHZ3DHvz9mV40OJEcLFYGIRMSQvHRevG4ME04o4P53Shn/p/cpLd8ediw5BCoCEYmYtKQEbh8/hD//1/Gs2lTJefe+x2MzPtWB5DZORSAiEXf2oG68cv0pnNArix8/O4+rHp3Fxh06kNxWqQhEpFV07ZzCI18bwf+cdwzvLC7n7Lvf5d0l5WHHkkaoCESk1cTFGd84uQ/PXj2K9NREvvrXD/jliwup2l0bdjSpR0UgIq1uUI90XrhmDF8deRQPTVnORX94nyXrtoUdSwIqAhE5IlKT4rl13GAeuryI8m3VXHDfFB54t5T1W6vCjhbzdPN6ETniyrdV88PJH/H24rpjBv26dGR03xxG983hxD5ZdE5JDDlh+9TUzetbVARmlgU8AfQCPgEucfdNDZY5Dbir3qQBwAR3/5eZ/Q34HLAlmHeFu8852OeqCESin7uzYPVWppRU8H5JBTM/2UjV7j3ExxlD89IZ0zeHUYU5HHdUBskJ8WHHbRdaqwjuBDa6+x1mdjOQ6e43HWD5LKAEyHP3nUERvOjuk5vzuSoCkfanuqaWD1ds5v2SCt4vreCjlZvZ45CSGMcJvbIYE2wxDOzembg4CztuVGqqCBJa+L7jgFOD548AbwNNFgHwReDf7q4boIrIfpIT4hlZmM3Iwmz+m/5srdrNjGUb64qhpILb//0xAJlpiYwszK7blVSYw1HZaZipGFqipVsEm909o97rTe6eeYDl3wR+5+4vBq//BowEqoE3gJvdvbqJda8CrgIoKCg4fsWKFYedW0Siz7qtVUEpbOD9kgrWBgeZe2ak1u1G6pvNqMIccjslh5y07TrsXUNm9jrQrZFZPwEeOdQiMLPuwFygh7vvrjdtLZAEPACUuvutB/vDaNeQSGxzd5ZV7Ni3tTCtdANbq2oAGNCtU3DgOZsRvbPpmNzSHR/tx2HvGnL3Mw7wpuvMrLu7rwm+1Ncf4K0uAZ7dWwLBe68Jnlab2cPAfx8sj4iImVGY25HC3I58dWQvavc481dtYUpJBVNLK3h0+goemrKchDhjWH7GvjOShuVnkJSgs+Ybaumuod8AG+odLM5y9xubWHY68CN3f6vetL0lYtSdWVTl7jcf7HO1RSAiB1K1u5ZZKzbVFUNJBXNXbcEd0pLiGdH7/w489+/aKaYOPLfWWUPZwJNAAfAp8CV332hmRcC33f0bwXK9gPeBfHffU2/9N4FcwIA5wToHHbdWRSAizbFl526mLduw74ykZeV191nO7pDEqL45jA4OPudnpYWctHW1ShGERUUgIi2xZkvlvoPO75dUsH5b3TkqBVlpjO6bw0l9sijISqNnRio5HZPbzVaDikBEpBHuTsn67bxfUsGUkg3MWLaBbdU1++YnxcfRPSOFHump9MxMpUdGKj0zUuiZkUaPjBR6ZKSSkhgdF7y11nUEIiJRzczo17UT/bp24orRvamp3UNJ+XZWbapk9eZKyjZXsnpzFas3VzJlaQXrtlXR8PfnnI5J9MhIbbIssjoktelrHVQEIiL1JMTHMaBbZwZ069zo/N21e1i7pYpVm+uKYvXmSlZtrmTV5ipKyrfzzpJyKhsMs52SGBeUQ92jR0bqfn1SIXYAAAUXSURBVK+7paeEejaTikBEpBkS4+PIz0pr8sCyu7N55+6gHD5bFh9/vJ7ybftfN2sGXTol7yuIvEbKonNqQqttVagIREQiyMzI7JBEZockBvdMb3SZqt21+7Yq9pbFqk2VrN5SycLVW3lt4Tp21ezZb52OyQn0yEjh/suOp09ux4hmVhGIiBxhKYnx9MrpQK+cDo3O37PH2bBj174tiX1bFJsqSU+N/BDdKgIRkTYmLs7I7ZRMbqdkjs3POPgKLf28Vv8EERFp01QEIiIxTkUgIhLjVAQiIjFORSAiEuNUBCIiMU5FICIS41QEIiIxLiqHoTazcuBw716fA1REME6kKFfzKFfzKFfztNdcR7l7bsOJUVkELWFmxY2Nxx025Woe5Woe5WqeWMulXUMiIjFORSAiEuNisQgeCDtAE5SreZSreZSreWIqV8wdIxARkf3F4haBiIjUoyIQEYlxMVUEZjbWzBabWYmZ3Rx2HgAz+6uZrTez+WFnqc/M8s3sLTNbZGYLzOx7YWcCMLMUM/vAzD4Kcv0i7Ez1mVm8mc02sxfDzrKXmX1iZvPMbI6ZFYedZy8zyzCzyWb2cfDvbGQbyNQ/+Hva+9hqZteHnQvAzL4f/Jufb2aPm1lKxN47Vo4RmFk8sAQ4EygDZgIT3X1hyLlOAbYDf3f3wWFmqc/MugPd3f1DM+sEzAIuagN/XwZ0cPftZpYITAG+5+7Tw8y1l5ndABQBnd39/LDzQF0RAEXu3qYukDKzR4D33P1BM0sC0tx9c9i59gq+M1YBJ7r74V7AGqksPan7tz7Q3SvN7EngZXf/WyTeP5a2CEYAJe6+zN13AZOAcSFnwt3fBTaGnaMhd1/j7h8Gz7cBi4Ce4aYCr7M9eJkYPNrEbzNmlgecBzwYdpa2zsw6A6cADwG4+662VAKB04HSsEugngQg1cwSgDRgdaTeOJaKoCewst7rMtrAF1s0MLNewHBgRrhJ6gS7X+YA64HX3L1N5ALuBm4E9oQdpAEH/mNms8zsqrDDBPoA5cDDwa60B82s8Tu5h2cC8HjYIQDcfRXwW+BTYA2wxd3/E6n3j6UisEamtYnfJNsyM+sIPA1c7+5bw84D4O617j4MyANGmFnou9TM7HxgvbvPCjtLI0a7+3HAOcDVwe7IsCUAxwF/cvfhwA6gTRy3Awh2VV0IPBV2FgAzy6RuD0ZvoAfQwcwui9T7x1IRlAH59V7nEcFNq/Yo2Af/NPBPd38m7DwNBbsS3gbGhhwFYDRwYbA/fhLweTP7R7iR6rj76uDneuBZ6naThq0MKKu3NTeZumJoK84BPnT3dWEHCZwBLHf3cnffDTwDjIrUm8dSEcwE+plZ76DtJwDPh5ypzQoOyj4ELHL334WdZy8zyzWzjOB5KnX/QT4ONxW4+4/cPc/de1H3b+tNd4/Yb2yHy8w6BAf7CXa9nAWEfoaau68FVppZ/2DS6UCoJyI0MJE2slso8ClwkpmlBf83T6fuuF1EJETqjdo6d68xs2uAV4F44K/uviDkWJjZ48CpQI6ZlQE/c/eHwk0F1P2G+1/AvGB/PMCP3f3lEDMBdAceCc7oiAOedPc2c6pmG9QVeLbuu4ME4DF3fyXcSPtcC/wz+MVsGfC1kPMAYGZp1J1d+K2ws+zl7jPMbDLwIVADzCaCw03EzOmjIiLSuFjaNSQiIo1QEYiIxDgVgYhIjFMRiIjEOBWBiEiMUxGIiMQ4FYGISIz7/x7i79+vQ7FvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.plot(model.linear_1_bias.detach().numpy())\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}