pytorch-lightning_ipynb/cnn/cnn-lenet5-mnist.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c60c3300",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch            : 1.10.1\n",
      "pytorch_lightning: 1.6.0.dev0\n",
      "torchmetrics     : 0.6.2\n",
      "matplotlib       : 3.3.4\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -p torch,pytorch_lightning,torchmetrics,matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0e5f9ee4",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext pycodestyle_magic\n",
    "%flake8_on --ignore W291,W293,E703"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b74fe944",
   "metadata": {},
   "source": [
    "<a href=\"https://pytorch.org\"><img src=\"https://raw.githubusercontent.com/pytorch/pytorch/master/docs/source/_static/img/pytorch-logo-dark.svg\" width=\"90\"/></a> &nbsp; &nbsp;&nbsp;&nbsp;<a href=\"https://www.pytorchlightning.ai\"><img src=\"https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/docs/source/_static/images/logo.svg\" width=\"150\"/></a>\n",
    "\n",
    "# Model Zoo -- LeNet-5 Trained on MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "864369a2",
   "metadata": {},
   "source": [
    "This notebook implements the classic LeNet-5 convolutional network [1] and applies it to MNIST digit classification. The basic architecture is shown in the figure below:\n",
    "\n",
    "![](../../pytorch_ipynb/images/lenet/lenet-5_1.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a919782c",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "LeNet-5 is commonly regarded as the pioneer of convolutional neural networks, consisting of a very simple architecture (by modern standards). In total, LeNet-5 consists of only 7 layers. 3 out of these 7 layers are convolutional layers (C1, C3, C5), which are connected by two average pooling layers (S2 & S4). The penultimate layer is a fully connexted layer (F6), which is followed by the final output layer. The additional details are summarized below:\n",
    "\n",
    "- All convolutional layers use 5x5 kernels with stride 1.\n",
    "- The two average pooling (subsampling) layers are 2x2 pixels wide with stride 1.\n",
    "- Throughrout the network, tanh sigmoid activation functions are used. (**In this notebook, we replace these with ReLU activations**)\n",
    "- The output layer uses 10 custom Euclidean Radial Basis Function neurons for the output layer. (**In this notebook, we replace these with softmax activations**)\n",
    "- The input size is 32x32; here, we rescale the MNIST images from 28x28 to 32x32 to match this input dimension. Alternatively, we would have to change the \n",
    "achieve error rate below 1% on the MNIST data set, which was very close to the state of the art at the time (produced by a boosted ensemble of three LeNet-4 networks).\n",
    "\n",
    "\n",
    "### References\n",
    "\n",
    "- [1] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. [Gradient-based learning applied to document recognition](https://ieeexplore.ieee.org/document/726791). Proceedings of the IEEE, november 1998."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f250573",
   "metadata": {},
   "source": [
    "## General settings and hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "542f0b67",
   "metadata": {},
   "source": [
    "- Here, we specify some general hyperparameter values and general settings\n",
    "- Note that for small datatsets, it is not necessary and better not to use multiple workers as it can sometimes cause issues with too many open files in PyTorch. So, if you have problems with the data loader later, try setting `NUM_WORKERS = 0` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c9c9f45f",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 256\n",
    "NUM_EPOCHS = 20\n",
    "LEARNING_RATE = 0.005\n",
    "NUM_WORKERS = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf4c7a28",
   "metadata": {},
   "source": [
    "## Implementing a Neural Network using PyTorch Lightning's `LightningModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "017ca5a5",
   "metadata": {},
   "source": [
    "- In this section, we set up the main model architecture using the `LightningModule` from PyTorch Lightning.\n",
    "- We start with defining our neural network  model in pure PyTorch, and then we use it in the `LightningModule` to get all the extra benefits that PyTorch Lightning provides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5bf908bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "class PyTorchLeNet5(torch.nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes, grayscale=False):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.grayscale = grayscale\n",
    "        self.num_classes = num_classes\n",
    "\n",
    "        if self.grayscale:\n",
    "            in_channels = 1\n",
    "        else:\n",
    "            in_channels = 3\n",
    "\n",
    "        self.features = torch.nn.Sequential(\n",
    "            torch.nn.Conv2d(in_channels, 6, kernel_size=5),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.MaxPool2d(kernel_size=2),\n",
    "            torch.nn.Conv2d(6, 16, kernel_size=5),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.MaxPool2d(kernel_size=2)\n",
    "        )\n",
    "\n",
    "        self.classifier = torch.nn.Sequential(\n",
    "            torch.nn.Linear(16*5*5, 120),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.Linear(120, 84),\n",
    "            torch.nn.Tanh(),\n",
    "            torch.nn.Linear(84, num_classes),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = torch.flatten(x, start_dim=1)\n",
    "        logits = self.classifier(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b5a989d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pytorch_lightning as pl\n",
    "import torchmetrics\n",
    "\n",
    "\n",
    "# LightningModule that receives a PyTorch model as input\n",
    "class LightningModel(pl.LightningModule):\n",
    "    def __init__(self, model, learning_rate):\n",
    "        super().__init__()\n",
    "\n",
    "        self.learning_rate = learning_rate\n",
    "        # The inherited PyTorch module\n",
    "        self.model = model\n",
    "\n",
    "        # Save settings and hyperparameters to the log directory\n",
    "        # but skip the model parameters\n",
    "        self.save_hyperparameters(ignore=['model'])\n",
    "\n",
    "        # Set up attributes for computing the accuracy\n",
    "        self.train_acc = torchmetrics.Accuracy()\n",
    "        self.valid_acc = torchmetrics.Accuracy()\n",
    "        self.test_acc = torchmetrics.Accuracy()\n",
    "        \n",
    "    # Defining the forward method is only necessary \n",
    "    # if you want to use a Trainer's .predict() method (optional)\n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "        \n",
    "    # A common forward step to compute the loss and labels\n",
    "    # this is used for training, validation, and testing below\n",
    "    def _shared_step(self, batch):\n",
    "        features, true_labels = batch\n",
    "        logits = self(features)\n",
    "        loss = torch.nn.functional.cross_entropy(logits, true_labels)\n",
    "        predicted_labels = torch.argmax(logits, dim=1)\n",
    "\n",
    "        return loss, true_labels, predicted_labels\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"train_loss\", loss)\n",
    "        \n",
    "        # To account for Dropout behavior during evaluation\n",
    "        self.model.eval()\n",
    "        with torch.no_grad():\n",
    "            _, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.train_acc.update(predicted_labels, true_labels)\n",
    "        self.log(\"train_acc\", self.train_acc, on_epoch=True, on_step=False)\n",
    "        self.model.train()\n",
    "        return loss  # this is passed to the optimzer for training\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"valid_loss\", loss)\n",
    "        self.valid_acc(predicted_labels, true_labels)\n",
    "        self.log(\"valid_acc\", self.valid_acc,\n",
    "                 on_epoch=True, on_step=False, prog_bar=True)\n",
    "\n",
    "    def test_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.test_acc(predicted_labels, true_labels)\n",
    "        self.log(\"test_acc\", self.test_acc, on_epoch=True, on_step=False)\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n",
    "        return optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a4f8ac0",
   "metadata": {},
   "source": [
    "## Setting up the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f99658ec",
   "metadata": {},
   "source": [
    "- In this section, we are going to set up our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0193d9e",
   "metadata": {},
   "source": [
    "### Inspecting the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a85aeca4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "train_dataset = datasets.MNIST(root='./data', \n",
    "                               train=True, \n",
    "                               transform=transforms.ToTensor(),\n",
    "                               download=True)\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE, \n",
    "                          num_workers=NUM_WORKERS,\n",
    "                          drop_last=True,\n",
    "                          shuffle=True)\n",
    "\n",
    "test_dataset = datasets.MNIST(root='./data', \n",
    "                              train=False,\n",
    "                              transform=transforms.ToTensor())\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         num_workers=NUM_WORKERS,\n",
    "                         drop_last=False,\n",
    "                         shuffle=False)\n",
    "\n",
    "# Checking the dataset\n",
    "all_train_labels = []\n",
    "all_test_labels = []\n",
    "\n",
    "for images, labels in train_loader:  \n",
    "    all_train_labels.append(labels)\n",
    "all_train_labels = torch.cat(all_train_labels)\n",
    "    \n",
    "for images, labels in test_loader:  \n",
    "    all_test_labels.append(labels)\n",
    "all_test_labels = torch.cat(all_test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7e7d853a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "Training label distribution: tensor([5915, 6732, 5945, 6121, 5832, 5412, 5909, 6256, 5839, 5943])\n",
      "\n",
      "Test labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "Test label distribution: tensor([ 980, 1135, 1032, 1010,  982,  892,  958, 1028,  974, 1009])\n"
     ]
    }
   ],
   "source": [
    "print('Training labels:', torch.unique(all_train_labels))\n",
    "print('Training label distribution:', torch.bincount(all_train_labels))\n",
    "\n",
    "print('\\nTest labels:', torch.unique(all_test_labels))\n",
    "print('Test label distribution:', torch.bincount(all_test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fd094dc",
   "metadata": {},
   "source": [
    "### Performance baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26126a41",
   "metadata": {},
   "source": [
    "- Especially for imbalanced datasets, it's quite useful to compute a performance baseline.\n",
    "- In classification contexts, a useful baseline is to compute the accuracy for a scenario where the model always predicts the majority class -- you want your model to be better than that!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8a23adbe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline ACC: 11.35%\n"
     ]
    }
   ],
   "source": [
    "majority_prediction = torch.argmax(torch.bincount(all_test_labels))\n",
    "baseline_acc = torch.mean((all_test_labels == majority_prediction).float())\n",
    "print(f'Baseline ACC: {baseline_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0db99229",
   "metadata": {},
   "source": [
    "### Setting up a `DataModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac1c0141",
   "metadata": {},
   "source": [
    "- There are three main ways we can prepare the dataset for Lightning. We can\n",
    "  1. make the dataset part of the model;\n",
    "  2. set up the data loaders as usual and feed them to the fit method of a Lightning Trainer -- the Trainer is introduced in the next subsection;\n",
    "  3. create a LightningDataModule.\n",
    "- Here, we are going to use approach 3, which is the most organized approach. The `LightningDataModule` consists of several self-explanatory methods as we can see below:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9bcf3be3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from torch.utils.data.dataset import random_split\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "class DataModule(pl.LightningDataModule):\n",
    "    def __init__(self, data_path='./'):\n",
    "        super().__init__()\n",
    "        self.data_path = data_path\n",
    "        \n",
    "    def prepare_data(self):\n",
    "        datasets.MNIST(root=self.data_path,\n",
    "                       download=True)\n",
    "        self.resize_transform = transforms.Compose(\n",
    "            [transforms.Resize((32, 32)),\n",
    "             transforms.ToTensor()])\n",
    "        \n",
    "        return\n",
    "\n",
    "    def setup(self, stage=None):\n",
    "        # Note transforms.ToTensor() scales input images\n",
    "        # to 0-1 range\n",
    "        train = datasets.MNIST(root=self.data_path, \n",
    "                               train=True, \n",
    "                               transform=self.resize_transform,\n",
    "                               download=False)\n",
    "\n",
    "        self.test = datasets.MNIST(root=self.data_path, \n",
    "                                   train=False, \n",
    "                                   transform=self.resize_transform,\n",
    "                                   download=False)\n",
    "\n",
    "        self.train, self.valid = random_split(train, lengths=[55000, 5000])\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        train_loader = DataLoader(dataset=self.train, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=True,\n",
    "                                  shuffle=True,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return train_loader\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        valid_loader = DataLoader(dataset=self.valid, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=False,\n",
    "                                  shuffle=False,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return valid_loader\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        test_loader = DataLoader(dataset=self.test, \n",
    "                                 batch_size=BATCH_SIZE, \n",
    "                                 drop_last=False,\n",
    "                                 shuffle=False,\n",
    "                                 num_workers=NUM_WORKERS)\n",
    "        return test_loader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d68d27e2",
   "metadata": {},
   "source": [
    "- Note that the `prepare_data` method is usually used for steps that only need to be executed once, for example, downloading the dataset; the `setup` method defines the the dataset loading -- if you run your code in a distributed setting, this will be called on each node / GPU. \n",
    "- Next, lets initialize the `DataModule`; we use a random seed for reproducibility (so that the data set is shuffled the same way when we re-execute this code):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "92f2d713",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(1) \n",
    "data_module = DataModule(data_path='./data')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e70c23c9",
   "metadata": {},
   "source": [
    "## Training the model using the PyTorch Lightning Trainer class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dec03e7",
   "metadata": {},
   "source": [
    "- Next, we initialize our model.\n",
    "- Also, we define a call back so that we can obtain the model with the best validation set performance after training.\n",
    "- PyTorch Lightning offers [many advanced logging services](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) like Weights & Biases. Here, we will keep things simple and use the `CSVLogger`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cf93512a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_lightning.callbacks import ModelCheckpoint\n",
    "from pytorch_lightning.loggers import CSVLogger\n",
    "\n",
    "\n",
    "pytorch_model = PyTorchLeNet5(\n",
    "    num_classes=10, grayscale=True)\n",
    "\n",
    "lightning_model = LightningModel(\n",
    "    model=pytorch_model, learning_rate=LEARNING_RATE)\n",
    "\n",
    "callbacks = [ModelCheckpoint(\n",
    "    save_top_k=1, mode='max', monitor=\"valid_acc\")]  # save top 1 model \n",
    "logger = CSVLogger(save_dir=\"logs/\", name=\"my-model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58fb64a9",
   "metadata": {},
   "source": [
    "- Now it's time to train our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0f5fef0d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name      | Type          | Params\n",
      "--------------------------------------------\n",
      "0 | model     | PyTorchLeNet5 | 61.7 K\n",
      "1 | train_acc | Accuracy      | 0     \n",
      "2 | valid_acc | Accuracy      | 0     \n",
      "3 | test_acc  | Accuracy      | 0     \n",
      "--------------------------------------------\n",
      "61.7 K    Trainable params\n",
      "0         Non-trainable params\n",
      "61.7 K    Total params\n",
      "0.247     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation sanity check: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dfb9f7e0835941c0a4d7db2203a79f0e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training took 1.67 min in total.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "trainer = pl.Trainer(\n",
    "    max_epochs=NUM_EPOCHS,\n",
    "    callbacks=callbacks,\n",
    "    progress_bar_refresh_rate=50,  # recommended for notebooks\n",
    "    accelerator=\"auto\",  # Uses GPUs or TPUs if available\n",
    "    devices=\"auto\",  # Uses all available GPUs/TPUs if applicable\n",
    "    logger=logger,\n",
    "    log_every_n_steps=100)\n",
    "\n",
    "start_time = time.time()\n",
    "trainer.fit(model=lightning_model, datamodule=data_module)\n",
    "\n",
    "runtime = (time.time() - start_time)/60\n",
    "print(f\"Training took {runtime:.2f} min in total.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fccc4e9",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "914f982f",
   "metadata": {},
   "source": [
    "- After training, let's plot our training ACC and validation ACC using pandas, which, in turn, uses matplotlib for plotting (you may want to consider a [more advanced logger](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) that does that for you):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0ed54af7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Epoch', ylabel='ACC'>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABXL0lEQVR4nO2dd3hb1dnAf8d7O/GM40yvxEmcvTcZEAJhj1BWWkYp0NLyfYzSBbRQ2n6lQFllBih7JQHCcnAWcfYisZPYcexsz8R763x/HCk4jmTLsq4k2+f3PHok3Xvuva+upPve804hpUSj0Wg0mtZ4uVsAjUaj0XgmWkFoNBqNxipaQWg0Go3GKlpBaDQajcYqWkFoNBqNxio+7hbAmURFRclBgwY5tG11dTXBwcHOFciJaPk6h5avc2j5Oocny7dt27YSKWW01ZVSym7zGDdunHSUjIwMh7d1BVq+zqHl6xxavs7hyfIBW6WNa6o2MWk0Go3GKlpBaDQajcYqWkFoNBqNxipaQWg0Go3GKlpBaDQajcYqWkFoNBqNxipaQWg0Go3GKlpBNNbChn8TfnqPuyXRaDQaj0IrCOEFmc8xKP99d0ui0Wg0HoVWED7+MOkOep/eDcd3ulsajUaj8Ri0ggAY/1OavANhwzPulkSj0Wg8Bq0gAALCOd73Ati7DE4VuFsajUaj8Qh6vIKobWjm2v9k8i4XghCw8Xl3i6TRaDQeQY9XEIF+3pRVN7C6NBzSroHtb0JNmbvF0mg0GrfT4xUEwJzUGA6cMlE1/hfQWANbXnW3SBqNRuN2tIIA5g6NpVnC2tPRkHw+bHpR5UdoNBpND0YrCGDsgF4E+8Kq7CKY+iuoKYFd77pbLI1Go3ErWkEAPt5epEV5s3p/EaYB06DvWNjwbzA1u1s0jUajcRtaQZgZFe1DaXUDu46Vw7RfQVke7PvC3WJpNBqN29AKwkxalDdeAr7bVwSpl0DvQfD90yClu0XTGE1BJpzY5W4pNBqPQysIMyF+gnEDeys/hJc3TLkbjm2Fw5nuFk1jJFLCx7fCsrvcLYlG43FoBdGCOUNjyTpRwYnyWhh9PQRFqlmEpvty+jBUHIXCH+BUvrul0Wg8Cq0gWjA3NQaAjH3F4BcEE2+HA19B0T43S6YxjJYzxOzP3SeHRuOBaAXRguSYEOJ7BfLdvkK1YMJt4BOoIpo03ZOCDRAQDjHDYZ9WEBpNS7SCaIEQgrmpMazPLaGusRmCI2HMDbD7fag44W7xNEZQsAEGTIFhl8DhjVBV5G6JNBqPQSuIVswZGkNdo4nMvFK1YMpdIJth0wvuFUzjfKqKoTRHKYjURYDUoc3uYP+XsPMdd0uhsYJWEK2YnBBJoK8332Wb7yQjBsOwS2Hr61BX4V7hNM7F4n8YOBVihkHvwZD9mXtl6mlICV89CF8+AM2N7pZG0wqtIFoR4OvNtKQovttXhLTkQEz9FdRXwPY33CucxrkczlQ+prjRqtR76sVwaC3Ulbtbsp7DSXP0WH0FHNnsbmk0rdAKwgpzU2M4drqWA4VVakH8WBg0AzKfh6YG9wqncR4F30O/8eDjp96nXgKmRjjwjXvl6klkf6b6wnv5QO637pbGMdIfhlcv6JZJtVpBWOG8ISrcdZUlmglg2j1QeRz2fOwmqTROpa5C3b0OnPrjsvjxENIHsle4T66eRvYKGDgN+k+G3HR3S9NxTM2w/S04shGO73C3NE5HKwgr9AkPYHjfsB/9EABJ85SdesMz3fJOocdxdDNIk3JQW/DygqEL1YVKl3s3nuIDULxPBQgkz1MKu/Kku6XqGIczVfVngF3vuVcWA9AKwgZzh8aw/fApTlWbTUpCKF9EUVbXvNPRnE1BpjJr9J949vLURapp1MHv3CNXT8IyU0tdpG7AoOv9t7JWgE+A6iOz56Nu52jXCsIGc1JjMUlYc6D4x4UjroTQvrr8RnegYAPEjQK/4LOXD5qhEud0VrXxZH8G/SZAWF+IHQGhcZDThfwQJpP6DIlzYfwtUFPa9RRcO2gFYYOR8eFEhfixal8LM5OPH0y5E/LXwbFt7hNO0zma6tX319K8ZMHbF1IuhANfdru7QY/iVAGc2GnOP0HN0JPmQl4GNDe5VTS7Ob5d+SWHXaJkD4rqdo3GtIKwgZeXYPaQGNbsL6Kp2fTjirE3g38YfP+M+4TTdI5j26G5/mwHdUtSL4baUyrKSWMMlrImFgUBkDRfhRgf3eIemTpK1nJlpkxZoG4s0q5SSX+1p9wtmdPQCqIN5g6NoaKuiW0FLb7wgDAY/zNlPy3Lc59wGsexXPitzSBAmQx8ArWZyUiyVkBsGkQk/LgsYTYI765hppFSmZcGz4LAXmrZqMXQ3AB7l7lTMqeiFUQbTE+OwtdbqCZCLZl0h/ohZz7nHsE0neNwJkSnQlCE9fV+QcpksO9zZWfWOJfKk3BkkzLNtCSwF/Sf1DXyIQr3wKlDZ3+GuNEQPbRbRTNpBdEGoQG+TBwccbYfAiAsDkZdCzvehuoS9wincQxTs8rYHWhj9mAhdRFUnlB2Zo1z2fc5IM82L1lImqu6+1UWnrvOk7Ak+A256MdlQsDIa1VORDexLmgF0Q5zhsaSW1TF4dKas1dM/RU01cLml90jmMYxCveosg4DbPgfLKRcoOzLujaT88laAZHJ6m67Ncnz1fPBVa6VqaNkrVC/oZDos5ePvAYQsPsDt4jlbAxVEEKIBUKI/UKIXCHEg1bWCyHEM+b1u4UQY1us+40QYq8QYo8Q4l0hRICRstpi7lCVVf3dvlZ3NNFDVLTL5pegocbKlhqPpGCDem5vBhHYW4W8Zn+mEyOdSU0Z5K9Xphkhzl3fZySExHp2uGtJDhRnn2siAwjvB4NnKDNTN/jdGKYghBDewHPAhcAw4DohxLBWwy4Eks2P24EXzNvGA78CxkspRwDewGKjZG2LQVHBJEQFn2tmApj2K6gtg51vu14wjWMUbIBeA9QfuT1SL4aygyrb15MpPageXYH9K1X5fGvmJTCHu85TiYqeGu5qSfAberH19aOuU/6JblB80MgZxEQgV0qZJ6VsAN4DLm015lLgTanYCPQSQsSZ1/kAgUIIHyAIOG6grG0yZ2gMm/LKqK5v9YMdMEUl+mQ+q2zbGs9GSuWgbs+8ZGHoxYDw/GimD5fA0ou7RnmQ7M8gfIBy6NoiaR7UnfbcXKOsFapuV3i89fWpi8A3qFvkRPgYuO944EiL90eBSXaMiZdSbhVC/B9wGKgFvpFSWi2xKYS4HTX7IDY2ltWrVzskbFVVlc1tI+ubaWg28eKy1YyLPfuURYXPZcTRJ9j70eMUx8xw6Nidlc8T6ArybfryHSZVF7O/LpITdso6JiwFry3vsE1OMFw+R86fX30pU0/uBuDguw9wZMAVTpZM4Yzv17uphmk56RyLX8jBNWtsjvNp9GMaXhSseoX8wfYpPVf9/vzriphyYicHE27mSBvHS+09gYhdH5AZtBCTt5/H/z9sIqU05AFcDbzS4v2NwL9bjfkCmN7i/SpgHNAb+A6IBnyBZcAN7R1z3Lhx0lEyMjJsrmtoapYj/viVvP/DXeeubG6S8tlJUj45Qsr6KoeP3xn5PIEuId/WpVL+KUzK4gP2b7j+abVNWb5hsknZifO3/b9Kvn9PkPKv/aWsLnWqXBac8v3u/lDJWpDZ/thXzpfyP7Ps3rXLfn8bnlWfoSS37XE56Wrc3mVSSs/+fwBbpY1rqpEmpqNA/xbv+3GumcjWmHnAISllsZSyEfgEsNMu4Hx8vb2YmRJNxv4iTKZWjicvb7j4X1B+GDIed4+AGvs4nAnB0RCZZP82qWY78z4PNTPlpiun7lWvqRLm6590t0S2yV6hZO03sf2xSfNU+eyq4vbHupKsFapuVGRi2+MSZqvS8V08J8JIBbEFSBZCDBZC+KGczK0L7a8AbjJHM00GyqWUJ1CmpclCiCAhhADmAtkGytouc4bGUFRZz97jVtqODpwC434KG5+H4ztdLpvGTgo2wIDJ1qNnbBGRADHDPdMPYWpWztykedBnhHKObnoJTh9pf1tX01CjIpOGXqzKqrdHsrm6qyeFu1oS/FKtRC+1xssbRl4NOd906VwpwxSElLIJuBv4GnVx/0BKuVcIcYcQ4g7zsJVAHpALvAzcad52E/ARsB34wSznS0bJag+zh0QjRKsmQi2Z97C6O/3sV54bfdGD8a8rgdMF9juoW5K6SM0+qqxEsrmTY9uVMzdprnp/3kPq2RNnsge/U2XUrYWGWqPPKPV/8qRwV0uCn72fYdR1YGqCPZ8YKpaRGJoHIaVcKaVMkVImSikfMy97UUr5ovm1lFLeZV6fJqXc2mLbP0kph0opR0gpb5RS1hspa3tEhvgzun8vMqyFu4IqE3Dh31UW6KYXXSqbpn3Cy7PUC1sF+toi9WJAqhBNTyI3XWXzJpyn3vfqD5NuV9EzhXvdK1trsleo3JKB0+wb7+VlDndd5TkRgtmfKfOktQQ/a8QOhz5pXTqaSWdSd4C5Q2PYdbScoso66wOGXaqS5zIeU+WMNR5DeHkW+IWqP2xHiR0BvQZ6npkpNx3ix51dU2r6varacPoj7pOrNU0NsP8rVZbC29f+7ZLmqcqontDKs6YMDq1T5qWOmChHLobj2wmqPmqcbAaiFUQHmDM0FoDV+2w4zoSAi/5P3dV9cW+3yKTsLoSXZ6nucV7eHd9YCGVmylutylF7AjVlKk/A0onNQlAEzPgN5HytMpY9gUNrob7cdnKcLRLnqP+SJ5iZ2kvws0Xa1SC8iC3MMEYug9EKogOkxoUSFx5g2w8BKkN3zh/U3d2ej10nnMY2NWWEVBe0X16jLVIXganRMy5WYG6JKs9VEKCqDYf2hW//6Bk3Kdkr1OwtYXbHtguKUDMkT6jumv0ZhPeHvmM6tl1oLCTOJbZwTZesDKwVRAcQQnDe0BjW55RQ39SGXXTibeqH/dWD6k5P414Ob1TP9tq/rdFvogrR9JTifbmrlE3f2gXLN1A5rI9t+7EshLswNcO+LyDlfPB1oJxa0nzljHdnJFBdhVLIqYs6Zl6yMGoxAfXFXbIBlVYQHWTu0BiqG5rZfKiNC7+XNyx6RtlPv/2D64TTWOfwBkzCB/qObX+sLby8YMhCNYNwd0kLk0nNUBPn2DaZjbpOOVPTH3Fv69TDmVBTYl9oqDWS5wHSPGNyEznfqEZAjn6GIQtp8g7skjkRWkF0kKmJUfj7eLEqu52Qxz4jYOovYcd/lQ1W4z4KMqkIS3HsDrYlqRdDY7XyRbiTwj1QXWTdvGTB20eFXpcdhO1vuky0c8haAT4BP5bx7ihxY1SvZ3ea9iwJfv1bVwqyE78giqOnQdayLlf5WSuIDhLo583UxEi+21dkKQ9im1kPQO/B8NmvodFG5JPGWBqq4cROysNbFxJ2gEEzwT/c/WYmS0vOxDltj0tZoApKrn4C6quMl6s1JpM6V0nzwC/YsX14eak8j4Or3GPDP5Pgd5F9CX42KIydDQ1Vnhcq3Q5aQTjAnNRYDpfVcLC4uu2BvoGqDEfZQVj3f64RTnM2R7eAqYny8OGd35ePn2oktP9L9yZD5q5S4bqhfdoeJwTMe0TNNjY+7xrZWnJ8O1Qed9w0YyFpPtSUwgk3hLtaEvw6+RlO9xqunNxdLCdCKwgHmGOriZA1Es9T9uD1/4LCLIMl05xDQSYIL8rD7Uxuao/URaoHyOENztlfR6mrUC0t2zIvtWTAJFXe4vunXe/ozVoOXr5KqXaGxDmAgJx0p4jVIbJXQEAvGDS9c/sRXqrb3MHvVMmOLoJWEA4Q3yuQoX1C+c5WVnVrzn9MJS99dk+XDHXr0hzeALEjaPYJcs7+kuYqm7q7zEyH1qryDfYqCIC5f1KO9bX/ME6u1kipzlHCLFVloDMER0L8WNeHu1oS/IZ2MMHPFiMXgzTBDx91fl8uQisIB5kzNIYt+acor7UjQiQ4Ehb8FY5uhm2vGS+cRtHUAEe2OFZewxZ+wZA4V2VVu0PZ56arnAJ7KqJaiE6BsTfClleh7JBxsrWkcI/qqtbRxDJbJM2Ho1tdGzZ+JsGvkyYyC9EpKvy9C0UzaQXhIHNTY2g2SdYesLMc8chrVaJQ+iNQ4bbmeD2LE7ugqda5CgLURa/yuOtLQEip/A8Js5Q/pCPMehC8fOC7vxgjW2uyP1NmFVttOTtK8nxcHu6avQL8Qjqe4NcWIxdD4Q9wco/z9mkgWkE4yOj+vekd5Gu7eF9rhFAO6+YG+PJ+Y4XTKCx+ggGdyKC2RsoFILxhn4vNTCU5qu+IpXprRwiLgyl3wp6PXFOSPmuFSkwMjnLO/vqOgcAI14W7nknwu6Dz4dEtGXGlUtS7u8YsQisIB/H2EsweEkPG/iKaWzcRskVEAsx+UN1deVrht+5IQaaqvhkS49z9BkUop2X2Z64tZWEJb+2I/6El0+5RF9n0PzlPJmuU5EBxtvPMS6ASAl0Z7lqwwZzg58TPAMrcnHwB7P7Qc6rUtoFWEJ1gztAYTtU0svPIKfs3mnK3qg668j4VkaIxBpNJZfE6e/ZgIXURlOZC8X5j9m+N3HSIGgK9Bji2fUA4zLxPJfoZaaqxlPdwlnnJQtJ8qC6Gk7ucu19rZH+mghGSHEzwa4tR10LVSfcnXNqBVhCdYGZKNN5eov2s6pZ4+6oyHJUnYNWjxgnX0ynOVs10OlN/qS2GXqSeXWVmaqxVtXwcnT1YmHCLUjDf/sm4O/GsFRA/HsLjnbtfS2Kg0eGulgS/xLngH+L8/acsUMq6CzirtYLoBOGBvowf2Nv+cFcL/cbBpJ/DllfgyGZjhOvpFJj9D52p4NoWYX2h3wTXmQrzv4emOsf8Dy3x8VfVhk/uNqba8OnDcGKn/V3XOkJItPJFtAp3feLLfTy304mVCiwJfkZ8BlDfwYgrlRKqrzTmGE5CK4hOMjc1hn0nKzl2uoMF3Ob8Xl1kPrtHhWN2BClV0tORzeouZPUTuhd2awo2qJLXvQYad4yhF6uL4enDxh3DQm46+AQ6Z0Y04iqITYPvHoUmJzdqtOSHONt2byFpvsqON4e7SilZtuMYu4ub2y99Yy9Zy5UjOWWBc/ZnjZGLVYSdu8u2tINWEJ3E0kSow7MI/1C46J9QlAUbnjl3vZRQWagcrTveZnDef+HDJfCfmfDEAPhHIrw6Hz79Oaz+K3xym+6FbUFK5X8YOMWx8sz2YrkI7vvCuGNYyE1XjnFnRNR4ecH8h5Vi2/p65/fXkuzPlPKJSHDufi0kz1fJZnmqAU9BaQ0nK+qob4aTFU6YRUipfCiDnZDg1xb9J6o6bR5eesPH3QJ0dRKjg4kN82dHwSlunNzBu9UhF8Kwy2DN31W8deUJKMtTyUxleapyqJkBeEHEIPXH6z9JPVseJ3fDRz+D3e/DmOud+vm6JKfy1bl0dv5DayITIWaYuihO/oVxxzmVD6U5MOFW5+0zca66CK79O4z+CQSEdX6flYWq98Z5D3V+X7aIH6f6YOSkw4grycwrPbPqYFE1ceGBndt/4R51vqf/pnP7aQ8hVAme1X+F8qOq0ZgHohVEJxFCkBIbyoEiB22JF/5N3Q199QB4+0HvQeqiP3iGWQEMhogE1u7MY9YcGw7KyCTY8G/1Y0u7Stk4ezKHM9XzAIMVBCgz07r/UyY/Z8X8tyZ3lXrurIO6JUKocuAvn6dmsHN+3/l97vsckMaZl0CFuybOUTMqk4nMg6UE+3lT3dDMweIqpid38jvIWqES/IZc5Bx522LkNbD6cdj9Acy41/jjOYA2MTmB5JhQcouqMNmbD9GS0D5w12b49Q/wu5Nw9xb4yfuqNMfE29RFISIB6dWGLhcC5v4Ryo/AtqUOf45uQ8H3qsBatJMK9LVF6iJl8jCyjHPuKuVLiUx07n7jx8LwKyDzOecUkMteAZHJxp/3pPlQXYQ8uZvMvFLmpMYS6AMHi51Q0jz7M3VjERLd+X21R8RgFYa96z3PaA1rBa0gnEBKbAh1jSaOnnKw01hoHxV6aKs7mD0knAeDZqiCbO6o/e8ohXth2V2w8UXn7bPAnP/Qifr9dtMnTV28d7xtzJ+8qQEOrVE3Ckb4U+b+QWX3f3wr7F3meG5OTRkcWud4W86OYI7kKtu1kuLKeqYkRBIX7NV5BWFJ8DMqeskaoxZDyX7Xl22xE60gnEByrIqVznHUzOQMhFBVO6uLYZMTL7ZGcWQLvHsdvDBVOeq+ekCVRO8slYWq/4bR/gcLQqgM5SMbjen/fGSTajTjTPNSSyISYP6flR/rw5vh7wnwxiVqVlF60P797P8SZLNrLq4hMRA3iuYD3wAwJVEpiLz2+rO0R9Zy9ezsBL+2GHYZePurkPfTRzoe0Wgw2gfhBJJiQgE4UFjF3NRY9wnSf4Lqm/z9MyohKrC3+2SxhpTqbnjdP1WlzMDeMPshJeuXD0D6w6rG0bRfOX4Mi//BVQoCYNwS2PoafP17SD5fNYpyFrnpqqfC4BnO22drptwJE29XyijnazjwNXz9kHpEJkHyBfSq7QNNU20XCcxeAeEDIG60cXK2JGk+keueJDG0iUGRQcQFC74/XkdVfRMh/g5e1rI/MybBry0Ce6mky51vqwdAUCSExqk2p6F91COkz4+vQ/uodS7wNWoF4QTCA32JDfN37wzCwnm/gxenqwYx8x52tzQKkwkOfKkUw7Ft6sd+/mPqwmrJVL38P8qW/+0flKltyl2OHatgA/gGQdwop4nfLl7eymf0xiLY8CzMus95+85NhwGTVVi0kXj7wKBp6jH/URXJc+AbOPAVbHmZ0c0NsO8fqgFWygLlB7DY6esrVemOCbcZb14yI5Pm4b3u/7ghOg8hBHEhyhhyqLiatH7hHd/hqQKV0zLvEecKag+LnlKmpsqT6lF18sfXRdlQVahmZ60JjPhRYfQaqPbjZLSCcBIpsaHkFHqA7b/PCBXJtPFFmPQLCHXjjKa5CfZ+AuueVLbd3oPg4qdUWGXrux9vH7jiZfVH+PohFUniSOjo4Q0qw9kZDV46wuCZqm/A+ifV53PGXWjFCRV26Y6LVu9BMOl29aiv4ofPniPN75iaXWQtB4QKOU1ZoJRCc4NLbfe5fkOJkUHM8lJ1meKClYI4WFzlmILYZ86INzICyxYB4W133TM1q5ar1hRIVaEK6S4yplulVhBOIikmhPc2H8Fkknh5ueYuyiazfwt7P1UO64vc0Au7sQ52vQPrn4LTBRCdCle8AsMvV4rAFt4+cOWraibx1YPK3DTpdvuPW1eu6uzPfrDTH8Ehzv+zuoCmPwxXvtz5/R00ILzVEfxDKI2aBLNnKzPhiV2QY55dZDwGSGXy6EgTo06SmV9OhGkkC05lgpTEBAm8vYTjjuqsFaqIprMjxZyBl7fyu4TEQNxIlx5aKwgnkRIbSm1jM8dO19I/wkntLR0lMhHG3KhCXqfere4GXUF9FWx7XZlZqk6qO8wFT6i7THsjirx94crXVNb4l/ep7exNEDu8CZDGVXBtj96DYOovVV7EhFtVP+jOkJuuzHGxw50inlMQAvqOVo9Z90NVkZKz1wDXRI2Z2ZhXSrz/OC6u3giFe/DxEgyICHJMQVSeVP6X2b91vqBdHB3F5CSSYzwgkqkls+5Xdx6rnzD+WDVlDMx/D54aAd/8HqKHwE0r4NZVMHRhxy8cPn5w9VJIuRC++B/7y0Ec3qBq6PSb0OGP4DRm3KtqQH31QOeqpTY3wcEM48JbnUVIjDKpDZruskOaTJKNeWU0DDYXLjQ3EUqMDuZgUQcjmRrrzLlD0rXhrV0ErSCcRHKLSCaPIKyvSrTb9Z5ydBlF+VF4cTqD899Vd+63pMPNK1RbzM5c2Hz84Jo3VHOVz38N295of5uCTFXt08+NMzi/YJj/iIpr70ydnePbVbnyzlZv7YYcKKqkrLqBEUNSVB6KuZFSYnQIh0qr227g1Vin8jUy/gqvX6Tqmq3+q/rduCKxsouhFYSTCA/yJSbU3zMc1Ram36tqPGU8Zsz+a0/Df6+C+kq2jf07XPeuCrV1Fj7+cM2b6i76s3tgx39tj22sVRdVd5mXWpJ2tbLHpz/seOJZbrpy1DuzH3I3IfOgqr80OSFSRVMd3oh3UzWJ0SE0NJk41jJhtbEO8termfTSi5VCeONiWPM3aKhUN1HXvQc3f+7ZMzU3oX0QTiQlNtRzTEygWmNO/aWq93Jsm/IJOIumenjvetVV7YaPqTxsUKkA3wC49m147zpYfre6aI7+ybnjjm1TkTSuzH+whRBw4RPw8hzlj5jvQGOo3HQVkx8U4Xz5ujgb80rp1ztQ+fqS58P6J+l9ahdByVPwp4HSvasYYNqrFMORzdBcDwjl4J14mzKHDZhibLXWboJWEE4kKSaED7Z6SCSThSl3wub/wKo/w03LnLNPkwk+vQMK1qvopIRZcHi1c/ZtDd8AWPwOvLsYlt2poptGXXv2mAJzglz/TjqGnUX8OBh9PWQ+D2Nv7lh0THUpHNuunaZWMJkkmw6VMd+SkNpvIviHM7DgIwIq1rPbfzP+3zWiFYJz0CYmJ5ISG0pNQ3PHmwcZiX8ozPgfVTH20Frn7PPbP6j8hnmPwMirnbPP9vANhMXvqoziZXeopu8tObwBYoZ71h333D8qM9k3HayUmpcBSPeHt3og2ScrOF3TqMxLoEKjhy4kpCoP36YqPvRawBuDnoAH8uHna+GCx1RZfa0cHEIrCCdiqcmUW+RBfgiA8bdAWLzqgd3ZgnIbX4DMZ1Vphmn3OEc+e/ELguveV13VPr0dfvhILW9uUqYEo9qLOkpoH5h5n6r0ainZbQ+56SpLtu9ow0TrqmzMU53kpiRG/rjwkn+zfvo78PO1LI+9ky/qRmuF4CQMVRBCiAVCiP1CiFwhxDnZS0LxjHn9biHE2BbregkhPhJC7BNCZAshPOzffy6WUNcDhR7khwBlopn1gGrVuP9Lx/ezdxl89VtVzGzBE+5x6vkFqXLoA6bAJ7erhMCTu1VBO09wULdm8i9U57CvH4LmxvbHm0xKmSTO6Vx1325K5sFSBkYG0bdXi3pX3r40+6jItcToEOeU/dYABioIIYQ38BxwITAMuE4IMazVsAuBZPPjduCFFuueBr6SUg4FRgEGxmo6h15BfkSH+pPjaTMIUPbwiET47s+OxecXbFAX5P4T4cpX3Hvx8guGn3ygZPnoFsh4XC33BAd1a3z84YLHoXifKujXHoU/QHWRNi9Zodkk2XSolMmDI22OSYwOobS6gdM1nlUVtati5AxiIpArpcyTUjYA7wGXthpzKfCmVGwEegkh4oQQYcBM4FUAKWWDlPK0gbI6jZTYEHI8bQYBylY753eqZsuejzu2bfF+VZq71wAVEujMaqWO4h8C138I/cZD7rcqizmsr7ulss6QC1W/jozHlAO6Lcwx/STOMV6uLkb2iQoq65rONi+1IjEmGICDnS39rQGMjWKKB460eH8UaB1iYm1MPNAEFAOvCyFGAduAe6SU53zrQojbUbMPYmNjWb16tUPCVlVVObxtSwIb6tl6oomMjAyEE00wTpFP9mZcyGB8Vv6ezSW92+5SZ8avvpSx2x/AywTbk+6jbvNu4+RzAO+Bvya15kkqwoZwuI3ju0s+C0GRVzAhbw3H376LnJQ7zllvkW/0jo/xDklg27ZsPGnS7O7zB/DlIWWik4X7Wb0656x1FvlKatTseOW6rVQecnHBxjbwhPPnEFJKQx7A1cArLd7fCPy71ZgvgOkt3q8CxgHjUUpiknn508Cf2zvmuHHjpKNkZGQ4vG1L/rsxXw584HN5pKzaKfuz4Cz55P6vpfxTmJRbXm1/bG25lM9Pk/KxvlIe29HmUKfJZxAeId8X90n5cC8pT/xwzqqMjAwpa09L+UiElN8+7HrZ2sETzt9PX98sz/uHdTks8jU1m2TyQyvl4yuzXCeYHXjC+bMFsFXauKYaaWI6CvRv8b4fcNzOMUeBo1LKTeblHwFj6QJYSm54pB8CVGJR/8mw5u8q+9gWTQ3wwY2qTPc1b+iIGmcw+0HVK/urB61Hkx1aC6Ym7X+wQlOzic2HypiUYNu8BODtJRgc5UBNJo1VjFQQW4BkIcRgIYQfsBho3ZNxBXCTOZppMlAupTwhpTwJHBFCDDGPmwsYU/DcyZwp2ueJfghQkUfz/qRqyG+2UZJaSljxS8hbDYue0RcsZxEUofxA+etU97LW5KaDX6hyvmvOYu/xCqrq2/Y/WEiIDiZPRzI5BcMUhJSyCbgb+BplTP1ASrlXCHGHEMJihF0J5AG5wMvAnS128UvgbSHEbmA08LhRsjqT3sF+RIV4WE2m1gycqi76659UPRRas+pR2P0enPd7GHO96+XrzoxdohL6vvmdqhNkQUoV3powy/XNjroAmXmW+kvtJ0ImRodwuKyGxuZOVNPVAAbnQUgpV0opU6SUiVLKx8zLXpRSvmh+LaWUd5nXp0kpt7bYdqeUcryUcqSU8jIp5SkjZXUmKbEhHPBUE5OFOX+A2lOqOX1LtryiFMfYm2Hm/7pHtu6Mt49qT3r6sEo4NBNUcxTKj+jZmg0yD5aSGB1MTGhAu2MTY4JpMkkKSmtcIFn3RmdSG0ByTAi5hZUWx7tn0nc0DLtMKYjqErVs3xew8j7V4OeiJ3V1S6NImKVaW657EiqUWy6ibLtap8t7n0Njs4mt+WV2mZdAzSAAnTDnBLSCMIDk2FCqG5o5Xl7X/mB3ct7voLFGXaiObFFJZ33HwFWvtd0aVNN55v9ZOaTTHwbMCiJqiMo10ZzFD8fKqW5oZkpClF3jE7SCcBpaQRiAxzuqLUSnqNLZW16Bd65RtYOue19lKmuMJWKwage7+33IW0Ov03u1eckGlv4Pk+zwPwCE+PvQJyxARzI5Aa0gDCAl1hzq6smOaguzHgSk6rNww8cQEu1uiXoO0++F0Dj44Ca8ZKM2L9lgY14pKbEhRIX4271NYkywnkE4Aa0gDEBFMvl5VvMgW/TqDzcth1u+6VjPAk3n8Q9RJdPrTtPs5aeq1GrOoqHJxNb8U0xpJ/+hNQlRqmifR/sBuwDa0GwQyTGhntOfuj08schdTyHtatjxFqXVkhjf9iN0ehq7j56mtrHZbge1hcToYCrrmiipaiA61P6Zh+Zs7JpBCCGChRBe5tcpQohLhBA6WLsNkmNDyC3SdzCadvDygpuWkzVMhxRbw+J/mNhGBVdrJMZoR7UzsNfEtBYIEELEo+ol/RRYapRQ3YHk2FCq6ps44emRTBr34+WtQ4ptsPFQKUP7hBIR7Neh7XSoq3OwV0EIKWUNcAWq4N7lqB4PGhuciWTy9IQ5jcZDqW9qVv6HDpqXAPqEBRDk560jmTqJ3QrC3NHtelQFVtD+izb5MZKpCziqNRoPZOfh09Q3mTrsoAbw8hIkROtIps5ir4L4NfBb4FNzPaUEIMMwqboBEcF+RAb7dY1QV43GA9mYV4YQMKmD/gcLlkgmjePYNQuQUq4B1gCYndUlUspfGSlYdyA5NoQDXSHUVaPxQDLzShgWF0Z4kGPxMInRIXy2+zh1jc0E+Or+3o5gbxTTO0KIMCFEMKrs9n4hxH3Gitb1SY4JJbdQRzJpNB2lrrGZ7YdPO2RespAYE4yUcKhE+yEcxV4T0zApZQVwGapE9wBUhzhNG6TEhlBZ38TJCh3JpNF0hB2HT9PQZGJyZxSEjmTqNPYqCF9z3sNlwHIpZSOgb4vbISmmC5Xc0Gg8iMy8UrwETLSz/pI1BkcFIwQ6kqkT2Ksg/gPkA8HAWiHEQKDCKKG6Cymx6g7mgI5k0mg6xMaDpYyIDycswPF83ABfb/r1DtQziE5gl4KQUj4jpYyXUi40N/kpAM4zWLYuT2SIPxHBfuTqXAiNxm5qG5rZcaTj9ZeskRitI5k6g71O6nAhxJNCiK3mxz9RswlNOyTHhOgZhEbTAbYfPkVjs+yU/8FCQlQIecXVmEzaIu4I9pqYXgMqgWvMjwrgdaOE6k4kx4aQo2syaTR2k3mwFG8vwYTBjvsfLCTGBFPb2MwJHSjiEPZmQydKKa9s8f4RIcROA+TpdqTEhlJZ10RhRT19wnW1To2mPTLzSkmLDyfEv/PFGiyRTHnFVcT3Cuz0/noa9s4gaoUQ0y1vhBDTgFpjROpeJJ2pyaTNTBpNe9Q0NLHryGmnmJegRair9gM6hL0q+g7gTSFEuPn9KeBmY0TqXlhqMh0orGJGsu7WptG0xdb8UzSZpEMF+qwRFeJHWIAPB4t1qKsj2FtqYxcwSggRZn5fIYT4NbDbQNm6BZHBfvQO8iVXzyA0mnbJzCvFx0swfmBvp+xPCEFijI5kcpQOtRyVUlaYM6oB7jVAnm6HEILk2C7UXU6jcSMb80oZ1b8XwU7wP1jQoa6O05me1LrDiZ0kx4SQU1ipI5k0mjaoqm9i99FyJncie9oaCdHBFFbUU1nX6NT99gQ6oyD01c5OUmJDqahroqiy3t2iaDQey5b8MppNkikJUU7d74+RTNoP0VHaVBBCiEohRIWVRyXQ10UydnnOdJfTZiarPPtdDst3HnO3GBo3s/FgKb7egnFO8j9YOKMgSvT/r6O0qSCklKFSyjArj1Appe4oZyfJZyKZtKO6NRV1jTyVnsNzGbnuFkXjZjbmlTK6fy8C/Zzbu2FgZBA+XkIX7XOAzpiYNHYSFeJHryBf3Z/aCmv2F9NkkhworOJEuU6t6alU1DXyw7Fyp9Rfao2vtxcDIoO0o9oBtIJwAUIIUmJCdX9qK6RnF+Lno36G6w6UuFkajbvYcqgMk4TJTsp/aI2OZHIMrSBcRJKuyXQOjc0mMvYVccmovkSH+rMmp9jdImncxMa8Uvy8vRg7wLn+BwuJ0SHkl9TQ1GwyZP/dFa0gXERKTAjltY0U60imM2zJL6Oiron5w2KZkRzF97klNOuqmz2SzLxSxgzoZVjv6IToYBqaTRw9pc2YHUErCBdhcVRrP8SPpGcV4efjxYzkKGalRHO6RtmhNT2L8ppG9h6vcFp5DWvo9qOOoRWEi0jW3eXOQkrJt9knmZ4URZCfD9OTohAC1h7QZqaexub8MqTEEAe1hcRo1b5G50J0DK0gXER0iD/hgZ4ZyfRcRi4Z+4pcesycoiqOlNUyLzUWUN33RvQN1wqiB5J5sBR/Hy9GD+hl2DF6BfkRFeKnZxAdRCsIFyGEICU2xOMimVZlF/KPr/fz1y+zXepA/zarEIC5qTFnls1IjmLHkdNU6JIIPYa6xmZW7DrG9KQo/H2M8T9YSNCRTB3GUAUhhFgghNgvhMgVQjxoZb0QQjxjXr9bCDG21XpvIcQOIcTnRsrpKpJiVNE+T4lkqmlo4o/L9+Ln7cWBwiqyTlS0v5GTSM8uZFS/cGLDfmyiNDMlmmaTZENuqcvk0LiXZTuOUVLVwC3TBxt+LBXqqk1MHcEwBSGE8AaeAy4EhgHXCSGGtRp2IZBsftwOvNBq/T1AtlEyupqUWHMkU5VnRDI9nZ7DsdO1PHf9WHy9Bct2uKbcRVFlHTuPnGb+sNizlo8d0JtgP2/W6nDXHoHJJHl5XR7D+4YZ6qC2kBgdTFl1A2XVDYYfq7tg5AxiIpArpcyTUjYA7wGXthpzKfCmVGwEegkh4gCEEP2Ai4BXDJTRpSTHqEimXA+oyZR1vIJX1h/iuon9mT8sltlDYli+87hLwky/yy5CSpjXSkH4+XgxJTGKtQeKPWaWpTGOjP1FHCyu5vaZCQhhfHHolu1HNfZhZD2leOBIi/dHgUl2jIkHTgBPAfcDoW0dRAhxO2r2QWxsLKtXr3ZI2KqqKoe3tZfTdSpJ54vvd9Bw1LdD2zpTPpOU/GVjHUE+kmkhpaxevZoUvya+raznxU++Y3hUx23BHZHvvW11RAUKTmRv4+S+sy8McaKR9FMNvL8ygz7Bzrt/ccX32xl6onz/2FxLRIAguOwAq1fndGpf9shXXKP+fyvXb6Mqv2P/v87i6d+vTaSUhjyAq4FXWry/Efh3qzFfANNbvF8FjAMuBp43L5sNfG7PMceNGycdJSMjw+Ft7cVkMsm0P30lH/pkd4e3daZ8b2bmy4EPfC4/3X70zLLahiY54o9fyd+8v8OhfdorX019k0z53Ur5p+V7rK4/VFwlBz7wuVz6/SGH5LCFK77fztDT5Nt15JQc+MDn8qU1B52yP3vka2o2yeTfrZSPf5HllGN2BE/+foGt0sY11UgT01Ggf4v3/YDjdo6ZBlwihMhHmabmCCH+a5yorkFFMoW6tex3UUUdf/9yH9OTorh09I8V2wN8vVmYFsfXe05S09Bk2PHX55ZQ32Q6x/9gYVBUMAMignS4azfn5XWHCPX3YfHE/u0PdhLeXoKEqGAdydQBjFQQW4BkIcRgIYQfsBhY0WrMCuAmczTTZKBcSnlCSvlbKWU/KeUg83bfSSlvMFBWl5EcG8KBIvd1l3v08yzqm038+bIR59h9Lx8bT3VD85kQVCNIzyokNMCHiYNtdw2bmRJFZl4pDU26bk535EhZDSt/OMF1kwYQGuBaU4+OZOoYhikIKWUTcDfwNSoS6QMp5V4hxB1CiDvMw1YCeUAu8DJwp1HyeArJMaGcrmmkpMr1kRSr9xfx+e4T3H1eEoOjgs9ZP3FQBH3DAwyLZjKZJKv2FTJ7SAy+3rZ/ejOTo6lpaGZbwSlD5NC4l9e/z0cAS6YOcvmxE6ODOVxWQ31Ts8uP3RUxtOmPlHIlSgm0XPZii9cSuKudfawGVhsgnluwlNzIKaokOtTfZcetbWjmD8v3kBgdzM9nJVgd4+UluHRMPC+tzaOkqp6oEOfKt/PoaUqqGpjXIjnOGlMSI/HxEqzNKXZJ+KPGdZTXNPLelsMsGtWXvr0CXX78xJgQmk2Sw6U1Z+qjaWyjM6ldTIqlaJ+L/RDPfJfDkbJaHrs8rc2M1cvHxNNskny2q7W7qPOkZxXi4yWYndK2gggN8GXsgN7aD9ENeWfzYWoamrl1hvGJcdZIiNJF+zqCVhAuJibUn9AAH3KKXFdyY//JSl5em8fV4/oxuZ2CaCmxoQzvG2aImSk9u5CJgyMID2rf7jwzJYq9xyt0efRuREOTiaUbDjE9KYrhfcPdIkOCuWif9kPYh1YQLsYSyXTARTMIk0ny0Kc/EBrgw28Xptq1zeVj4tl1tNypd1kFpdUcKKw6U5yvPWamRAPwfa7uMtdd+GzXcQor6t02ewAI9vchLjxAzyDsRCsIN5AcE0Kui6q6vr/1CNsKTvG7i4YREexn1zaLRvXFS8ByJ84i0rNVtVh7FcTwvuH0DvLVZqZugpSqrMaQ2FBmmZW/u9CRTPajFYQbSI4Npay6gRKDazIVV9bz15XZTE6I4Mqx8XZvFxsWwLSkKD7decxp4bjpWYUMiQ1lQGSQXeO9vQTTk6NZm1OCSXeZ6/Ksyylh38lKbp0x2CVlNdoiMTqYPN3+1y60gnADyTHmSCaDzUyPfZFFXaOJv1yW1uE/5WWj4zlSVuuUUNPymkY255cxb1jbzunWzEyOoqSqnuyTrqsyqzGGl9flERPqzyUtkjPdRWJMCJX1Tdq/ZQdaQbiBM5FMBjqq1+UUs2znce6YnUiSWSF1hAUj+hDo682nTjAzZewvotkk7TYvWbD4IdblaD9EVybreAXrckpYMm2Q4T0f7MFStC9X+yHaRSsINxAb5k+ov49hM4i6xmZ+v2wPg6OCuXN2okP7CPb34fzhsXy++0SnM5q/zS4kKsSfUf16dWi72LAAhsSGaj9EF+eVdXkE+Xlz/cSB7hYF0JFMHUErCDcghFAlNwzqLvdcRi4FpTU8dtkIAnwdv2O7bEw85bWNrN7veDvShiYTa/YXMy81Bi+vjtueZ6ZEsTX/lKH1oTTGcaK8lhW7jnPthP52hTe7gj5hAQT5eeuy33agFYSbSI4JNSSSKbeokhfXHOSKMfFMTYrq1L5mJEURFeLHsp2Om5k2HSqlqr6pw+YlCzNTomloNrExT3eZ64os3ZCPSUp+Ns19oa2tEULoSCY70QrCTSTHhlBa3UCpEyOZTCbJQ5/sIcjPh4cusi/noS18vL1YNKov6dlFlNc61ic6PauQAF8vpic7pqwmDIrA38eLtQe0H6KrUVnXyDsbD3NhWhz9I+yLXnMVidHBHHRRqHlXRisIN5F8xlHtvB/pR9uOsjm/jIcWDnVaHaXLx8TT0GTiyx9OdHhbKSXp2UXMSI522NQV4OvNpIRI3Ya0C/L+liNU1jdx+wzrtb/cSWJ0CMdO11LboIv2tYVWEG4ixVK0z0l+iNKqeh7/MpsJg3pz9Tjn1dhPiw8nITrYoWim7BOVHDtdy3wHzUsWZiZHkVdczdFTNZ3aj8Z1NDabeP37fCYOjmBU/17uFuccEs2RfXklehbRFlpBuIk+YQEqkslJM4jHVmZTXd/E45enOeQMtoUQgstHx7PpUBnHTtd2aNv07EKEgPOGdiz/oTWWzFttZuo6rPzhBMdO13rk7AF0JJO9aAXhJoQQJDkpkmnDwRI+2X6M22cmGFLC+LIxKgu7owX80rMLGdO/V6fLmifFhBAXHsA6bWbqEljKaiREBzOnkzcHRjEoMhgh0H6IdtAKwo04oybTquxCfvP+TgZEBPHLOclOkuxs+kcEMWFQbz7dYX/pjZPldew+Ws48G61FO4IQghnJUazPLaGpWXeZ83Qy80rZc6yC22YkOHU260wCfL3p3zuIvBI9g2gLrSDcSEpsKCVVDZRVd7y73PHTtfz8ra3c8sZWQgN8ef76sZ3KeWiPy8bEk1tUxd7j9pW9WLVPtS3trP/BwsyUaCrrmth19LRT9qcxjpfX5hEV4sflY+yv/+UOdCRT+2gF4UaSYjruqG5sNvHS2oPMe3INaw4Uc/+CIaz81QxGxBtbX/+itDh8vYXdZqb0rEIGRgY5VObDGtOTovASsEb7ITyanMJKMvYXc+PkQYbesDiDxOgQ8kqqdDHINtAKwo1YajIdsPMuZltBGYv+vZ7HV+5jSkIk3/5mFnfOTsLPx/ivsVeQH+cNiWH5ruM0t/OHqq5v4vuDpcxLjXVa5c5eQX6M7NdL+yE8nFfWHcLfx4sbp3hGWY22SIwJoa7RxPHyjgVf9CS0gnAjceEBhPj7kNvODOJUdQOv7annyhcyKa9t5D83juOVm8e7PPnoirHxFFfWt9vEZ11OCQ1NJoezp20xMzmKXUdOU17jWNKexliKKuv4dMcxrh7fz+7eI+7EUrRPRzLZRisINyKEICkmxGZ3OSklH2w9wtwn17D+WBO3z0wg/d5ZXDC8j1tq6s8eEkNYgE+7Zqb07ELCA30ZP6i3U48/MyUak4T1usucR/LmhgIaTSZume6Zoa2tORPqqv0QNtEKws0kx4RYzYU4UFjJtf/ZyP0f7WZwVDCPTA3koYWpBPv7uEFKRYCvNxeNjOOrvSdtFs9rNkm+21fEeUOi8fV27s9rdP9ehAb46OquHkhNQxNvbSzg/GGxDI4Kdrc4dhEZ7Ed4oK9uP9oGWkG4GRXJVM8pcyRTTUMTT3y5j4VPr+NAUSV/uzKND38+hf6hnvFVXTY6npqGZr7NKrS6fvvhU5RVNzglvLU1Pt5eTEuMYl1OcbfqBtYdPsuHW49SXtvI7TO7xuwBLEX7gsnTJiabuO92VANAkqXkRlEV5bWNPLxiL8dO13L1uH78dmGqx9lyJwyKIL5XIJ/uOMalo88NY0zPKsTXW5xp9uNsZqRE8dXekxwsriIpxvlJge7gp0u38ENBDXf5HGLxxP4E+XWtv2WzSfLK+jzGDujFuIER7hanQyRGh7BGz0ht4hm3pT0YSyTTvR/s5LY3txLs780HP5/CP64e5XHKAcDLS3DZmL6syymx2rLx2+xCJidEEhZgTO3/mclK8XSXcNct+WWs3q8uUI9+nsXUJ77jyW8POLXKr9F8vfckR8pquc1Dy2q0RWJMCEWV9VTU6cAHa2gF4Wb6hgfQK8iXkqp6HlgwlM9/OYOJgz37Luyy0fE0mySf7Tp+1vKDxVXkFVc7PXqpJf0jgkiICu42fojnMnKJDPbj8RmBfHTHFMYPjOCZVTlM+9t3/HH5Hg6XenaBwsyDpfxh2R4GRQZx/vA+7hanw1gimbSZyTpday7bDRFC8MHPpxAa4ENceKC7xbGL5NhQRsSHsWznMX42/cdGMKuylV9ibqqx9XdmpkTz3pbD1DU2e3wyVlvsOVbO6v3F3HfBEPzFUcYPiuCVQRHkFlXynzV5vLv5MP/dWMBFI/vy85kJhidDdgQpJa+uP8Rfv9zHwMggXrpxHN4eWlajLRJbRDKN9sCqs+5GzyA8gJTY0C6jHCxcPqYfu4+Wn1VLKj2riNS4MPr1NjY/Y0ZyFHWNJrbmnzL0OEbz/OpcQgN8zkkqS4oJ5R9Xj2Ld/XO4dUYCGfuKuPjf67nx1U18n1vidqd2dX0Tv3x3B3/5Ipt5qTEsv2tal/UH9Y8IwsdL6EgmG2gFoXGIRaPi8BI/VnitbJBsLShjvgHRS62ZnBCJr7fo0k2Ecosq+XLPSW6eMsimv6ZPeAAPLUzl+wfncP+CIWSfqOT6VzZxybPf8/nu424pXHiopJornt/Ayh9OcP+CIbx4wzhCDfI3uQJfby8GRgZpBWEDrSA0DhETGsD05GiW7TyGySTZXdyESTqvOF9bBPv7MH5gRJf2Qzy/+iABPt5nmehsER7oy52zk1j/wHn89Yo0quqbuPudHcz55xre2lhAXaNruqKtyi7kkmfXU1hZxxs/m8ids5PckrDpbJJjQtmaf4oTuuTGOWgFoXGYy8f05eipWrYdPsWOomZiw/wZER/mkmPPTIlm38lKiirqXHI8Z3KkrIblO49z3cQBHYpUC/D15rqJA0i/dxYv3jCOiGA//rBsD9Oe+I6n03MoMSjyyWSSfJrTwC1vbGVgZBCf3T2dGcnGhDG7g1/OTaK+ycTNr23mdE3HKyt3Z7SC0DjM+cP6EOjrzXubj/BDSbNTi/O1x4zkKADW5nS9cNf/rD2Il8DhpDJvL8GCEX349M6pvH/7ZEb2C+df6QeY+sR33P/RLvadtK8kuz2U1zRyyxtbWH6wkSvH9uOjO6a6vAaY0QzvG85LN40jv6SGW97YqvtUt0ArCI3DBPv7sGBEHz7efpT6ZgzJnrbFsLgwokL8upyZqaiijg+2HuWqcf3oEx7QqX0JIZiUEMnrP51I+r2zuGZ8P1bsOs6Cp9Zx/SsbWZVd2KlS1lnHK1j07HrW55Zw0zA//u/qkV06aqwtpiZG8fTi0Ww/fIq73tlOo25MBWgFoekklnak/t4wJSHSZcf18hLMSI5mfW5Jl6rn/8r6QzQ1m7hjVqJT95sUE8JfLktj42/n8sCCoeQVV3PLG1uZ++Qa3tiQT3W99dpZtli+8xhXvPA9dY3NvHf7ZOYM8O0W/oa2uDAtjr9cNoLv9hXx4Mc/uD1azBPQCkLTKaYlRtInLIBR0d4uv7ucmRJFWXWD3V3u3M3pmgb+u7GAS0b1ZWCkMQXtegX58YvZiay9/zz+fd0YwgN9+dOKvUz+6yoeX5nN0VNtJ941Npt45LO93PPeTkbG9+LzX03vcuUzOsP1kwZy7/wUPt5+lCe+3OducdyOoQpCCLFACLFfCJErhHjQynohhHjGvH63EGKseXl/IUSGECJbCLFXCHGPkXJqHMfH24tP75rKzcP9XX7s6UnKUdpVwl1f/z6fmoZmfjE7yfBj+Xp7sWhUX5bdNY1P7pzKrJRoXl1/iJl/z+DOt7exraDsnDvkoso6rn95E69/n89Ppw3i7dsmERPaOTNYV+SXc5K4acpA/rM2j5fWHnS3OG7FsExqIYQ38BwwHzgKbBFCrJBSZrUYdiGQbH5MAl4wPzcB/yOl3C6ECAW2CSG+bbWtxkOICw8k2Nf15ofoUH+GxYWx5kAxd51n/EW3M1TVN7F0Qz7nD4tlSB/XJpWNHdCbsT/pzfHTtbyRmc+7mw6z8oeTjOoXzs+mD2ZhWhy7j5Zz59vbKK9t5OnFo60WYuwpCCH406LhlFY38PjKfUQG+3PluH7uFsstGFlqYyKQK6XMAxBCvAdcCrS8yF8KvCnVrcxGIUQvIUSclPIEcAJASlkphMgG4lttq9EwMyWaV9blUVnX6NEJW29vLKC8ttGtiqxvr0B+e2Eq98xN5uPtx3h9/SHueW8nj32RzamaBuLCA/n0zomkxrkmVNmT8fYSPHnNKMprGrn/4930DvZlzlDXBWF4CkYqiHjgSIv3R1Gzg/bGxGNWDgBCiEHAGGCTI0I0NjZy9OhR6urajpcPDw8nOzvbkUO4BE+QLyAggH79+uHr6zkX4pkpUby45iAb81yTxe0IdY3NvLzuEDOSoxjlAfV+gvx8uHHyQK6fOIA1B4p5MzOf8EBfHrlkBOFBnvPduht/H29evHEcP3l5I3e+vZ23b53Uo/wxYKyCsGZzaB0W0OYYIUQI8DHwaymlVU+kEOJ24HaA2NhYVq9efdb6kJAQYmNjiY+PbzMKo7m5GW9vzw3hc7d8UkrKy8vZtWsXVVXnliWoqqo659y7gkaTxN8b/vvdTnyLbPtB3CUfwKrDjZRUNTCtt7Apg7vkE8DN5mTuHZu/tznOnefPHoyU79YUyWNlkhtfzuShSYH0c6B5l6efP5tIKQ15AFOAr1u8/y3w21Zj/gNc1+L9fiDO/NoX+Bq4195jjhs3TrYmKytLmkymc5a3pqKiot0x7sQT5DOZTDIrK8vquoyMDNcK04L7P9wlkx9aKfNLqmyOcZd8DU3NcupfV8krnv++zd+hO8+fPfR0+Q6XVssJf/lWTnzsW3mkrLrD23vy+QO2ShvXVCOjmLYAyUKIwUIIP2AxsKLVmBXATeZopslAuZTyhFC3+q8C2VLKJzsrSHeP33YVnnoe/+f8FHy8BY994XkmwuU7j3PsdC13n9c96hb1VPpHBPHmLROpbWjmptc2U1bdM0pyGKYgpJRNwN2oWUA28IGUcq8Q4g4hxB3mYSuBPCAXeBm407x8GnAjMEcIsdP8WGiUrJquTUxYAHedl8Q3WYVsyPWc0hvNJsnzq3MZFhfG7CHdp3ZRT2VonzBeXTKBY6dq+enrmzucfNgVMTQPQkq5UkqZIqVMlFI+Zl72opTyRfNrKaW8y7w+TUq51bx8vZRSSClHSilHmx8rjZRV07W5Zfpg+vUO5NHPs2j2kMzqr/acJK+4mrv07KHbMGFQBM/9ZCx7jldwx3+30dDUvUty6Exqgzl9+jTPP/98h7dbuHAhp0+f7vB2S5Ys4aOPPurwdl2dAF9vHlqYyr6Tlby/5Uj7GxiMlJLnMnJJiA5mwYiu14pTY5t5w2L56xVprMsp4X8/3NWlSr10lB7VcvSRz/aSZaMsg6NRQsP6hvGnRcNtrrcoiDvvvPOs5e0db+VKPWHqKBeO6MPEQRH885v9XDwqzmYjHlewen8xWScq+MdVI7tkK05N21wzvj+lVQ387at9RAT78adFw7rlLFHPIAzmwQcf5ODBg4wePZoJEyZw3nnn8ZOf/IS0tDQALrvsMsaNG8fw4cN56aWXzmw3aNAgSkpKyM/PJzU1lV/+8pcMHz6c888/n9pa+xqbrFq1ijFjxpCWlsbPfvYz6uvrz8g0bNgwRo4cyf/+7/8C8OGHHzJixAhGjRrFzJkznXwWXIMQgj8uGkZZTQP/XpXjNjmklDybkUt8r8AzxQw13Y87ZiVw6/TBLN2Qz9Orcrplcb8eNYNo606/srKS0FDnl0B44okn2LNnDzt37mT16tVcdNFF7Nmzh8GDVfD5a6+9RkREBLW1tUyYMIErr7ySyMizq6Lm5OTwyiuvsHTpUq655ho+/vhjbrjhhjaPW1dXx5IlS1i1ahUpKSncdNNNvPDCC9x00018+umn7Nu3DyHEGTPWo48+ytdff018fLxDpi1PYUR8OFeP68fSDfn8ZNJABkcZUxSvLTYdKmNbwSkevXQ4vt76Hqy7IoTgoYWpnKpp5Kn0HAor6rvdd959PkkXYeLEiWeUA8AzzzzDqFGjmDx5MkeOHCEn59w738GDBzNy5EgAxo0bR35+frvH2b9/P4MHDyYlJQWAm2++mbVr1xIWFkZAQAC33norn3zyCUFBqvnLtGnTWLJkCS+//DLNzV27Ycr/XjAEfx9vt4W9PpeRS1SIP9eM7++W42tch5eX4B9XjeTO2Ym8u/kwS17fTHlNo7vFchpaQbiY4OAf72hXr15Neno6mZmZ7Nq1izFjxlgtCeLv/2OGsLe3N01N7YfX2Zru+vj4sHnzZq688kqWLVvGggULAHjxxRf5y1/+wpEjRxg9ejSlpaUd/WgeQ0yoCntNzy5kvYs7zu06cpp1OSXcOmNwt22uozkbLy/B/QuG8o+rRrL5UBlXvPA9BaXVLpWhyaAGR1pBGExoaCiVlZVW15WXl9O7d2+CgoLYt28fGzdudNpxhw4dSn5+Prm5uQC89dZbzJo1i6qqKsrLy1m4cCFPPfUUO3fuBODgwYNMmjSJRx99lKioKI4ccX8kUGf46bRB9I8I5M+fZxn257HGcxm5hAf6csPkgS47psYzuHp8f966ZRKl1Q1c9tz3bD5UZvgxK+saeeSzvVz38kZDoqm0gjCYyMhIpk2bxogRI7jvvvvOWrdgwQKampoYOXIkf/jDH5g8ebLTjhsQEMDrr7/O1VdfTVpaGl5eXtxxxx1UVlZy8cUXM3LkSGbNmsW//vUvAO677z7S0tIYMWIEM2fOZNSoUU6TxR0E+Hrzu4Wp7C+s5F0Xhb0eKKzkm6xClkwdRIh/j3LvacxMTojk0zun0TvIj+tf2cjH244achwpJSt/OMG8J9ewdEM+Q/qE0mDAjZD+FbuAd955x+pyf39/vvzyS6vrLH6GqKgo9uzZc2YWYok6ssXSpUvPvJ47dy47duw4a31cXBybN28+Z7tPPvmkzf12RS4Y3odJgyN48pv9/GWK8SGvz2fkEuTnzZKpgww/lsZzGRwVzKd3TuMXb2/jfz7cxaGSasb6Oe/uvqC0mj8u38uaA8UM7xvGf24cz2iDqgTrGYSm22IJez1d28iKXGNr5xwurWHFruPcMHkgvYP9DD2WxvMJD/LljZ9NZPGE/jybkcvzO+upbehc8Ed9UzP/XpXD+f9ay7aCU/zx4mEsv2uaYcoBtILostx1112MHj36rMfrr7/ubrE8juF9w1k8oT/ph5vIKz63TLmzeGHNQXy8vbh1+uD2B2t6BL7eXvz1ijR+tzCVbYXNLH4pk6LKtvvS2GLDwRIufHod//z2APNSY0m/dxY/mz4YH4NDarWJqYvy3HPPuVuELsO984fw6bYjPPZFNq8umeD0/Z8sr+PjbUe5ZkI/YsJ6Xg9njW2EENw2M4HKE3m8vKeKy579nldunsCwvvZ17SuurOfxldl8uuMYAyKCWPrTCcweEmOw1D+iZxCabk90qD+XJPmyal8Raw8UO3XfzSbJv749QLOU/HxmolP3rek+jI314cM7pmCScPWLG1iVXdjmeJNJ8t+NBcz952o+332cX85J4pvfzHSpcgCtIDQ9hPkDfRkYGeTUsNf8kmoWv5TJ+1uPcNOUgfSPCHLKfjXdkxHx4Sy/exoJ0SHc9uZWXl1/yGq+0p5j5Vz+wgZ+v2wPw/uG8+U9M/mf84e4Ja9GKwhNj8DXS5VFyCmq4p3Nhzu1L5NJsvT7Qyx4ei37Tlbyz6tH8ceLhzlJUk13JjYsgPd/Ppn5w2L58+dZ/H7ZHhrNNyxV9U08+lkWlzy7nmOnanjq2tG8c9skkmJC3Cav9kFoegznD4tlamIkT357gEtHxRMe1PHQ18OlNdz30S42HSpj9pBonrhiJH3Ctd9BYz9Bfj68cP04/v71fl5cc5DDZTVcMTaeJ77cR1FlPT+ZOID7Lxjq0O/T2egZhIcREqLuFo4fP85VV11ldczs2bPZunWrzX1YKsFqzkYIwR8uHkZFbSNPrTrQoW1NJslbGwtY8PRaso5X8PcrR/L6kglaOWgcwstL8OCFQ/n7VSPJPFjKb97fRWSwP5/8YiqPXZ7mEcoBetoM4ssH4eQPVlcFNjeBtwOno08aXPhEJwU7l759+/bIxj9GkxoXxuKJA3grs4DrJw20a/p+9FQND3y8m+9zS5mRHMUTV44kvlegC6TVdHeuGd+fxOgQDhZVccXYeMPDVjuKZ0nTDXnggQfO6ij38MMP88gjjzB37lzGjh1LWloay5cvP2e7/Px8RowYAUBtbS1Llixh5MiRXHvttXb3gwB48sknGTFiBCNGjOCpp54CoLq6mosuuohRo0YxYsQI3n//fcB6n4juyL3zUwj09eaxL7LaHCel5N3Nh7ngX2vZefg0j1+exps/m6iVg8apjBvYm2sm9Pc45QA9bQbRxp1+rUH9IBYvXsyvf/3rMx3lPvjgA7766it+85vfEBYWRklJCZMnT+aSSy6x2ZHqhRdeICgoiN27d7N7927Gjh1r17G3bdvG66+/zqZNm5BSMmnSJGbNmkVeXh59+/bliy++AFTRwLKyMqt9IrojUSH+/GpuMo+tzGb1/iKroYPHT9fywMe7WZdTwtTESP525UgdpaTpcXieyupmjBkzhqKiIo4fP86uXbvo3bs3cXFxPPTQQ4wcOZJ58+Zx7NgxCgttx0WvXbuWa6+9FoCRI0ee6Q3RHuvXr+fyyy8nODiYkJAQrrjiCtatW0daWhrp6ek88MADrFu3jvDwcJt9IrorN08dxKDIIP7yRfaZKBJQs4YPth7hgn+tZWv+Kf586XD+e8skrRw0PRKtIFzAVVddxUcffcT777/P4sWLefvttykuLmbbtm3s3LmT2NhYq30gWuJIv1tbPSFSUlLYtm0baWlp/Pa3v+XRRx+12Seiu+Ln48XvLhpGblEVb28sAFRG9M+WbuH+j3aT2jeMr389kxunDMJL95TW9FB6lonJTSxevJjbbruNkpIS1qxZwwcffEBMTAy+vr5kZGRQUFDQ5vYzZ87kgw8+ONOudPfu3XYdd+bMmSxZsoQHH3wQKSWffvopb731FsePHyciIoIbbriBkJAQli5dSlVVFTU1NSxcuJDJkyeTlJTkjI/u0cxLjWF6UhT/Ss/B29uLf3y1j4ZmE39aNIybtWLQaLSCcAXDhw+nsrKS+Ph44uLiuP7661m0aBHjx49n9OjRDB06tM3tf/GLX3DDDTcwcuRIRo8ezcSJE+067tixY1myZMmZ8bfeeitjxozh66+/5r777sPLywtfX19eeOEFKisrufTSS6mrq0NKeaZPRHfGEvZ64dNr+cOyPYwf2Jt/XD3KLX2sNRpPRCsIF/HDDz+G10ZFRZGZmWl1XFWVqjg6aNAg9uzZA0BgYCBLly6124nesmf1vffey7333nvW+gsuuIALLrjgnO2s9Yno7gzpE8rjl6fR0Gzi+kkD8dazBo3mDFpBaHo8iycOcLcIGo1HohVEF2bSpEnU19efteytt94iLS3NTRJpNJruRI9QEFJKh6KAPJ1Nmza59Hi2oqI0Gk33pNuHuQYEBFBaWqovbp1ESklpaSkBAbr2kEbTU+j2M4h+/fpx9OhRiovbbhRTV1fn0Rc/T5AvICCAfv36uVUGjUbjOrq9gvD19WXw4Pb7BK9evZoxY8a4QCLH8HT5NBpN96Pbm5g0Go1G4xhaQWg0Go3GKlpBaDQajcYqojtF9wghioG2CxvZJgrw5DZsWr7OoeXrHFq+zuHJ8g2UUkZbW9GtFERnEEJslVKOd7ccttDydQ4tX+fQ8nUOT5fPFtrEpNFoNBqraAWh0Wg0GqtoBfEjL7lbgHbQ8nUOLV/n0PJ1Dk+XzyraB6HRaDQaq+gZhEaj0WisohWERqPRaKzSoxSEEGKBEGK/ECJXCPGglfVCCPGMef1uIcRYF8vXXwiRIYTIFkLsFULcY2XMbCFEuRBip/nxRxfLmC+E+MF87K1W1rvtHAohhrQ4LzuFEBVCiF+3GuPS8yeEeE0IUSSE2NNiWYQQ4lshRI75ubeNbdv8vRoo3z+EEPvM39+nQoheNrZt87dgoHwPCyGOtfgOF9rY1l3n7/0WsuULIXba2Nbw89dppJQ94gF4AweBBMAP2AUMazVmIfAlIIDJwCYXyxgHjDW/DgUOWJFxNvC5G89jPhDVxnq3nsNW3/dJVBKQ284fMBMYC+xpsezvwIPm1w8Cf7Mhf5u/VwPlOx/wMb/+mzX57PktGCjfw8D/2vH9u+X8tVr/T+CP7jp/nX30pBnERCBXSpknpWwA3gMubTXmUuBNqdgI9BJCxLlKQCnlCSnldvPrSiAbiHfV8Z2EW89hC+YCB6WUjmbWOwUp5VqgrNXiS4E3zK/fAC6zsqk9v1dD5JNSfiOlbDK/3Qi4rca7jfNnD247fxaE6lJ2DfCus4/rKnqSgogHjrR4f5RzL772jHEJQohBwBjAWtu4KUKIXUKIL4UQw10rGRL4RgixTQhxu5X1nnIOF2P7j+nO8wcQK6U8AeqmAIixMsZTzuPPUDNCa7T3WzCSu80msNdsmOg84fzNAAqllDk21rvz/NlFT1IQ1nqOto7xtWeM4QghQoCPgV9LKStard6OMpuMAv4NLHOxeNOklGOBC4G7hBAzW613+zkUQvgBlwAfWlnt7vNnL55wHn8HNAFv2xjS3m/BKF4AEoHRwAmUGac1bj9/wHW0PXtw1/mzm56kII4C/Vu87wccd2CMoQghfFHK4W0p5Set10spK6SUVebXKwFfIUSUq+STUh43PxcBn6Km8i1x+zlE/eG2SykLW69w9/kzU2gxu5mfi6yMcet5FELcDFwMXC/NBvPW2PFbMAQpZaGUsllKaQJetnFcd58/H+AK4H1bY9x1/jpCT1IQW4BkIcRg8x3mYmBFqzErgJvMkTiTgXKLKcAVmG2WrwLZUsonbYzpYx6HEGIi6jssdZF8wUKIUMtrlDNzT6thbj2HZmzeubnz/LVgBXCz+fXNwHIrY+z5vRqCEGIB8ABwiZSyxsYYe34LRsnX0qd1uY3juu38mZkH7JNSHrW20p3nr0O420vuygcqwuYAKrrhd+ZldwB3mF8L4Dnz+h+A8S6WbzpqGrwb2Gl+LGwl493AXlRUxkZgqgvlSzAfd5dZBk88h0GoC354i2VuO38oRXUCaETd1d4CRAKrgBzzc4R5bF9gZVu/VxfJl4uy31t+gy+2ls/Wb8FF8r1l/m3tRl304zzp/JmXL7X85lqMdfn56+xDl9rQaDQajVV6kolJo9FoNB1AKwiNRqPRWEUrCI1Go9FYRSsIjUaj0VhFKwiNRqPRWEUrCI2mAwghmsXZFWOdViVUCDGoZVVQjcbd+LhbAI2mi1ErpRztbiE0GlegZxAajRMw1/b/mxBis/mRZF4+UAixylxYbpUQYoB5eay518Iu82OqeVfeQoiXheoH8o0QItBtH0rT49EKQqPpGIGtTEzXtlhXIaWcCDwLPGVe9iyq/PlIVNG7Z8zLnwHWSFU0cCwqmxYgGXhOSjkcOA1caein0WjaQGdSazQdQAhRJaUMsbI8H5gjpcwzF1w8KaWMFEKUoEpBNJqXn5BSRgkhioF+Usr6FvsYBHwrpUw2v38A8JVS/sUFH02jOQc9g9BonIe08drWGGvUt3jdjPYTatyIVhAajfO4tsVzpvn1BlQlUYDrgfXm16uAXwAIIbyFEGGuElKjsRd9d6LRdIzAVk3ov5JSWkJd/YUQm1A3XteZl/0KeE0IcR9QDPzUvPwe4CUhxC2omcIvUFVBNRqPQfsgNBonYPZBjJdSlrhbFo3GWWgTk0aj0WisomcQGo1Go7GKnkFoNBqNxipaQWg0Go3GKlpBaDQajcYqWkFoNBqNxipaQWg0Go3GKv8PlF/fIs1nsmcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA9NUlEQVR4nO3dd3hUdfb48fdJhxR6QpWiKD1IKJaVIhZcXXvBviq6+rWuqz/d3l3X3XXXjnVdXBVdFRcV2yrYQAUE6R2UiCS0MGmTen5/fCZxCJOQSeZOMuG8nmeezNw2J5PJPfdTr6gqxhhjTF1xLR2AMcaY1skShDHGmJAsQRhjjAnJEoQxxpiQLEEYY4wJKaGlA4ikrl27ar9+/Zq0b3FxMampqZENKAIsrvBYXOGxuMLTFuNavHjxTlXtFnKlqraZR05OjjbV3Llzm7yvlyyu8Fhc4bG4wtMW4wIWaT3nVKtiMsYYE5IlCGOMMSFZgjDGGBOSJQhjjDEhWYIwxhgTkiUIY4wxIVmCMMYYE5IlCGOMiUGqyte7SvjPoq28sanck/doUyOpjTEm2qqqlS9zC/h00y4yUhIZ1D2dw7unk5GSGNH3qa5W1uUXsnDzbj7bvJuFW3aT5ysDoFOyUFWtxMdJRN/TEoQxJqKqqpVv95aSnpJIh3aRPUm2Ftv3+vlw3Q4+WLeDjzfsZG9pxX7b9OrYjiO6p3NE93SXNLLSObRbGkkJjau4qaiqZvk3e1kYSAYLt+ypfZ/uGSmM69+FMf07M65/Z3JXLYp4cgBLEMaYJtpdXM7mnUVs2lHMpp3FbN5RzKadRWzZVUJ5ZTUAh3Ruz9CeGQzr1YEhPTMY1rMD3dKTWzjy8Pkrqli4ZTcz15TxpyUfsjavEIDM9GROHJLFhMO7cexhXSkpr2Tt9kLWbC9kbeDx4bodVFa7O3cmxAkDuqVyRPcMBnVP54gsl0B6d2qHv6KaJV/vqS0dLPm6gNKKKgAGdE1lytDujO3fmbH9O9O7UztEvksI21ZHPjmAJQhjTAP8FVVs2VVz8i8OJIMiNu8spqDku6vmhDjhkC7tGdA1jYlHZNKvSyp7SspZtc3Him17eXPF9tptM9OTa5PG0J4ZDO3ZYb8TXktTVTbtLK4tJXy6aRf+imoSBMYdmsTZowYx/vBuDOqevk/cnVOT6N2pPZMHZ9UuK6+sZvPOYtZs99UmjS++2sNrX26r3SY1KZ6yymoqq5U4gcE9MrhgTB/G9e/M6H6dWyypWoIwxgBQWl7Fym17Wbq1gCVbC1iWW0DunlKCb1uflZHMgK5pfH94DwZ0TWVAt1T6d02jT6d2JMTXX3Xi81ewapuPldt8rPxmLyu3+fhw/U6qAlfWHdolMqRHBsN6uYRRsLeKfJ+fLmnJnlSdBKuuVnYVl5Pn8/P17hI+3rCTD9bu4JuCUsBdvU8dcwjjD+9KRe4qTj7hqLCOn5QQV1vVFKzQX8G6vCLWbi9kXV4h7ZPiGdO/Mzl9O0W8/aKpLEEYcxCqrlY27Szio9wK3p21nKVbC1izvbD2hN2rYztG9unIuaP60L9bKgO6ptK/ayqpyU07ZWSkJHLUgC4cNaBL7TJ/RRVrtheyctteVnzjY9W2vfxrwVe11VO/WfAecQJd0pLplpZMZkYymenJZKan0C098Dzju9cpifH7/Y67S9yJP99XRn6hnzxfmXtdWEa+z73eWVRWWwUEkJacwDGHduG6iYcy4fBu9OncvnbdvO2rm/T7h5KekkhO307k9O0UsWNGmiUI06ZVVyvLvtnLvLX5LP5qD6lJCbUnmm7BJ5uMZLqkNu1qVVXZW1pBXp2T0I7C705Gu/eUMn3dAlKTEmifnEBqUjztkxJITXY/2yfF0z4pntTkhH1/JiXQLimelIR4khPjSIqPI64JMe4sKmPp1wUs3eoeX+YWUOivBCA9eRsj+nTg2gkDGNmnE9l9OpCZnhL2e4QrJTGekX06MrJPx9plFVXVbNxRxBsffE7mIYcFTuRl7Chyn+2qbT52FpURdD6vlZ6SQGZ6MqnJCewoLGNH4b4n/hqd2ieSleH+7gOz0slMTyYrI4WsjGS6d2jH0J4ZJDZQGjqYWIIwbc7u4nI+Wr+DeWtd/fHu4nJEYFD3DL6t8jN/4058gZNjsJqr1cz0OleqGcl0TUumpLyK/EJ3NVpz4q/5WXPVGyw9JYGsjBQy05NJTxKqFfIK/ZTsrKK4vJKSMvcz1MmuIUkJcSQnxJGcEO9+JrrnKYl1l8dTGegJk7vHVZfExwlHZKXzg+yejOzTkYrt67nw+5OalHS8kBgfx6DuGWzPSmDi0f1CblNVrewuLie/0CXh/EAyyPf52VFURnFZFYcHnfhdScMlgG7pySQnxIc8rtmfJQgT84JLCbMXlrL57XdRdQ2GEw7vxsQjunHcwG50Tk2q3cdfURV0cvnuRFNTFbGjqIyV9VytBp/4x/TrvM8JKDP9u5/tkr47Ec2bN4+JE4/eL3ZVpayymuKySkrKXcIoLquiJOhnSXkVZZXVlFVWUVZRXfvcXxFYVlkdWO7WF5SUB7ZxSWtE7w5cdnRfRvbpxLBeGbRPSgiKa2OrSQ6NFR8ndAuUAI23LEGYmLSnuJwPA6WED9ftYFeglNA/I46bJw9k4hGZjOjVod6TX0piPH06t9+nfjmUqmplV3EZOwvLSU2O3+/E31wiQkpiPCmJ8XQ58ObGRJUlCNOqqKq7ki6rpLCskiJ/JUVlgYe/kq17Svhg3Q6Wbi2oLSWMH9iViUdkMv7wbixbOJ+JEw+PWDzxcUJmekpU6uSNaW0sQZioKC2v4svcAhZ/tYctO4spLq+k0P/dib8mIRSXNVwnLwIjenesLSUM79XB826QxhysPE0QIjIFuA+IB55Q1bvrrO8EPAUcCviBK1V1RWDdzcDVgACPq+o/vIzVRFaez8+iLXtY/NUeFn+1m5XbfLU9SrIykklPSSQtOcHV56enkJaSQFpy4BF4np6SQGrSvq87pSa1mj7ixrR1niUIEYkHHgJOBHKBhSIyW1VXBW32M2Cpqp4lIoMC208WkWG45DAWKAfeEpE3VHW9V/GapquqVtZs9/HFV3tY9NUeFm3ZUzvIKDkhjuw+Hblm/ABy+nZi1CGd6BTUWGyMab28LEGMBTao6iYAEZkJnAEEJ4ghwJ8AVHWNiPQTkSxgMPCpqpYE9v0AOAu4x8N4TSP5/BWs2FnJknfXsfirPSz5eg/F5W7OmMz0ZEb368SV3+tPTt9ODOmR0ejJyYwxrYuohtkJu7EHFjkXmKKq0wKvLwXGqeoNQdvcBaSo6q0iMhaYD4wDSoD/AkcDpcB7wCJVvTHE+1wDXAOQlZWVM3PmzCbFW1RURFpaWpP29VJLxlWtyo4S5evCarYGPXaWuu+MAH3S4zisUxwDO8ZzWMc4uraTFp1Tx/6O4bG4wtMW45o0adJiVR0dap2XJYhQZ4m62ehu4D4RWQosB5YAlaq6WkT+DLwLFAFfAvuPbAJU9THgMYDRo0frxIkTmxSs66fetH29FK24Cv0VrN1eyOpvfawO/Fy7vZCSQMkgTmBAtzSOGpjO4B4Z6K6vuPy08aS3svaAg/3vGC6LKzwHW1xeJohcoE/Q697AtuANVNUHXAEg7rJzc+CBqj4JPBlYd1fgeCYCVJUP1u1gydcFgYTgY+vu0tr1GSkJDO6Rwfmj+zCkRwaDeri57IPnupk3L7fVJQdjTGR5mSAWAgNFpD/wDTAVuCh4AxHpCJSoajkwDfgwkDQQkUxVzReRQ4CzcdVNppm27i7hZ7OW89H6nW5gWddURvTqyAWj+zC4RwaDe2TQo0NKq5p62RjTMjxLEKpaKSI3AG/jurk+paorReTawPrpuMboGSJShWu8viroEC+LSBegArheVfd4FevBoKpa+df8Lfzl7bXECfz29KGcP7pPREcFG2PaFk/HQajqHGBOnWXTg54vAAbWs+9xXsZ2MFmfV8j/e3kZS74uYOIR3fjjWcPp1bFdS4dljGnlbCR1G1ZeWc0j8zby4Nz1pCUn8PcLsjlzZC+rPjLGNIoliDZq6dYC7nhpGWvzCvlBdk9+/YMhdE2z2S+NMY1nCaKNKS2v4m/vrOWpTzaTmZ7CE5eN5oQhWQfe0Rhj6rAE0YbM37CTO19Zzte7S7ho3CHcecogm7fIGNNkliDagL2lFfxpzmpmLtxKvy7tmXnNUfvc+9e0sF0bSfbvaOkojAmbJYgY9/bK7fzy1RXsKi7nRxMG8OMTDt/v5u2mhVRXwfz74f0/MiohHY47HlItcZvYYQkiRhX6K7jz5eW8sfxbBvfI4MnLxzC8d4eWDsvUKNgKs66Frz6GgSeRuOF9+O/1cOHz7qYWxsQAm2YzBlVXK7fMXMrbK7dz+8lHMPuGYy05tCbL/gOPHAvfLoUzH4GLXmTjoT+EdW/C54+1dHTGNJqVIGLQfe+t5701+fzujKFcdnS/lg7H1CgtgDm3wfL/QJ9xcNaj0Lk/AN/0Oo2BshXe+QUccjT0GNGysYZSXQ2VfqgohYoS0GroeIiVeKKheBf4C6DLoS0dyT4sQcSYd1flcd976zk3pzeXHtW3pcMxNbZ87KqUfNtg0s/he7dCfNC/lwic+bArWbx0JfzoA0hK9TYm37fw6cNQuidw0g+c+IN+HlW0Bz6rdssqS/c/xoipcMaDEG+94SJuz1ew5g33+Ho+DDwZLmra7Qq8YgkihmzIL+LHLyxlRO8O/OHMYTYiujWoLId5d8HH/3Clhavehd45obdN7QpnPwYzzoA3/x+c8ZB3cRXvhBmnw+7NkNoNEttBYvvAz3bQrhMktmPPrr306DOgzvrAz51rYf4DULwDzp8Bya3vPggxRRXyVsKa191j+3K3PHMIHPcTGPyDlo0vBEsQMaLQX8GPnllEckIc0y/JsZ5KrcGOdfDKNPj2Sxh1OZx814FPogMmwHG3wkd/gwGTYPi5kY+rtACeOcs1lF8+G/oeU++ma+fNo0dD9xHoeji8djP86zS46D+Q1i3i4bZp1VXw9aeBksLrUPAVIK4K8sTfw6BTW121UjBLEDGgulr5yYtfsmVXCf++ahw9mzvRXlkRLHgI+h4N/cdHJsiDiSosfALe+aW70r7gWRh8WuP3n/hT2PwRvP5j6JVT204REeXF8Nz5kL8aLpzZYHJolFGXuRLIf34IT50El7wS2XjboopS2DTPJYS1b0LJLohPchcEx/0EjjgF0jJbOspGsQQRAx6au4F3VuXxq9OGcPShzexHv3M9zLzYVR8A9DvO1Zn3jYHbbeze5KpMDpvccjEU5bvuquvfgcNOcNVE6d3DO0Z8IpzzBEw/Dl6eBle+FZk6/go/zLwIchfCeU/DwBOaf0xwJ7TLZrvE8+RJcMlL0CM7MsduK1Rh9WxY/hJseA8qiiG5Axx+kislHHYCJKe3dJRhswTRyi3Nr+S+Jes468heXHFsv+YdbPVrMOs6SEhy1QW7N8JH98I/p7irm0k/hz5jIhJ3xFT43ZXYF/+CzR+6ZVOfh0Hfj34sa9+E/94A5UVwyl9g7NVN7+HTqS+cfp+7Mn//D3Dib5sXW1WFa/zeNM91rR1yRvOOV9ch4+Cqd+CZs+Gfp8LUZ111mRf8PlKLNsOuPt+1mSS2d1fhrbHdrbwYXrsFlr8I6T1g5IUuKfT9nvtfi2GWIFqxzTuLeXRZGUN6ZHDXWcOb3ihdVQlz/wAf/91VaZw/Azr0dutGXe6qSz75Bzx5Agw8yVWB9BoVsd+jSfJWwRczYNlM1wun4yEw6Rew8hV44yfQ73uQkhGdWFThzTvg80chazic8zhkDm7+cYeeBRvnus9+wAQ49PimHae6Gl79P1j7hktcIy868D5N0e0IlyT+fY57nP0oDDsncsf374VPH4EFDzGmzAeL6qyXuH0b2us2qtc879AHjr0JUqIwNmjXRnjhUshfBcf/Ar73E4hrO8PLLEG0UsVllfzomUXEC0y/JKfpd34r3umuLDd/ADlXwCl/hoSgab+T2rt/ptFXukFc8++HxyfBEafCxDuj21+/rMglgC9muGqSuERXtz/qcug/wf3jHToJnjgB3vsdnPrX6MS1+GmXHMb+CE76/b6fX3NNuds1Yr7yI7hufviNwKow5yfu6nXyr2DcNZGLLZQOveDKN+H5C+Glq6BoBxx1bfOOWVYInz3qekz5C2DQaayMG8zQQYfv1y03VFddKkrdRYRvm1u2/D/w5fPwg/sjV80Wytq34JVr3PfykpdbturTI5YgWiFV5faXvmRDfhG3jU6hT+f2TTvQN4vhhctcN8UzHoIjL6l/2+Q017tmzDT4bDrMfxAePQ4Gnw6TfhaZK+ZQVOGbL+CLp2HFK676pusRrkfQiKn7z13UezSM+5E7oQw/z1V9eGn3Jnj75zBgojuZR/rqMKk9nPdPeGwSvHqtq/pr7Huowru/gkVPuXEXx/0ksrHVp10nuHSWaz956w4o2g6Tfx1+9U95MXz+OHxyH5TuhsNPcRclPUeyY948GDGxafF9s9iVqJ49x33nT/ojtOvYtGOFUl0F8+6GD+9xbTHnP+OqDNsgSxAtad3b7kp5yBnQfXjt4ukfbGLO8u38/PuDGVj9ddOOvfhpmHM7pHWHq96Gnkc2br+UDJjw/1z9+oKHXZF/9Wsw7GyYcCd0O7xp8dRVshuWvehKC/krXdXA0LNdr5k+Yxs+2Rz/C1j9Orx2E/zow8he0QerrnKD3+IT4IyHvas6yBoKJ//RjcL+9CE45sbG7ffhX12Jb8zVrvQQTYntXFXlGz9xVZeFeXD6/Y1rbK8odUnt47+7i5fDToCJP6t//Ei4euXANR/AB3921Xcb3nexDTyx2YdOqCh0jfUb/gcjL3Gl2MS2e/teSxAtwbfNDZRa/Zp7/eFfoOcoGHUZH7ebwD1vr+EH2T2Zdlx/PvggzARR4XcnmiXPuDrtc56E9p3Dj7FdJzj+53DUde4k9NmjsHIWDD/fJZC6qiqDivzFDVQHlLgqlVWzoarMJa7T/g7Dzm18m0JyOpx2r/tH/fgfMPGO8H+/xvjkPtj6GZz9hKta8dKYaa6B+X+/hb7HHrgN6NNHXLtS9oVwyj0t03gbF+/+dundYd6foGSn6z1V3wjxCr/rbPDRva7U0X+C6xjhRSkwMQVO+LWronz1/+DZc90J/eRmlCa2LSVn8a1QUQA/uM9VfbbGRvMIsgQRTdVVsPBJV39eXeGK5SMvdifeL/4Fr99CDslMTx/PhHE/IeyvXsHXrsHs26Uw/nbX2BzXzAF17TvDCb+Bo653V2MLn4Dl/2FccjfXiFhz0q+uaPwxkzu4ksKoy5rexnH4ya6B9KO/wtAzXQNqJH27DObeBUPO9GYwW10icPoDMP178PJVrmRUX7fIL56Bt+50I29Pf7BlG0VFXLVQWqYrTfzrdLjoxX2rBivL3QXLR38D3zcuAZ77pOto4LVeOe6znHe3+/5ubGJpYsmz8MatSHwqXPFW5Eo7rZynCUJEpgD3AfHAE6p6d531nYCngEMBP3Clqq4IrPsxMA1QYDlwhar6vYzXU98ug9dvcfWjhx4Pp/4NOg9w6466lpIjr+KnDzzN+KI3OUs/IW7Gu9BtEL0zjoHi4Qe+j8DG912jYXWlN91A07q5q69jboRPH8a3fintevervydJQ71M2nWKTPe/KXe7Puezb4Ir3ozcibKyDGb9yCXH0/4evavE9p3d+IinT3Un27NDzPy64hVXtXboZFc6jG8l13ijr4TUTNch4qmTXaNtRk/XWPzBX2Dv19B7rJuPqv+E6F55JyQHlSauD680UVnmerAt/if0H8/iHtM49iBJDuBhghCReOAh4EQgF1goIrNVdVXQZj8DlqrqWSIyKLD9ZBHpBdwEDFHVUhF5EZgKPO1VvJ4pL3bF7wUPuxPA2U+4K9KgfxBV5Y5XVvDGrp6cc8WjxPVNcSeCL2Zw2Man4N5/w6DT3BV3TW+eGtXV8PG9ri995mC44N/eDt1P7w4n/o7VifPIamiKhmhIy3T/5P+93v0Dj7kqMsd9/w+u2+LFLzWteq45+h4DE+5w35kBk1yf+hrr3oFXroY+R7m/s1dtL001+DS47L/w/AVuQF1iO9iz2VWfnvZ318unJatkeuW4SRI/+LOrmtz4vqsqOvyk0NvvzYUXL3MXdcfeAsf/koqPPo5mxC3Oy7LpWGCDqm5S1XJgJlB39M4Q4D0AVV0D9BORrMC6BKCdiCQA7YFtHsbqjXXvwENHue57R14C138OI87b75/kiY8289qX27j95EGMP7ybq1rIuRyufo+Fo+9zV2cb34dnzoT7R7o2C98212/8hUvg/d+76pZp/2vV87p4YuTFbrqQ//3GfSbNteUT9/fKuSIijZpNMv52Vw3zxk9g5wa3bPNH8OKlkDXMzfiZ1MSebV7rezRc+bZrrE5Od9N9XP2+627aGurrE5Jdg/60/7lxEs+d59ooSgv23W7TB/DoBDff1vnPuIGMraW0FkWiqt4cWORcYIqqTgu8vhQYp6o3BG1zF5CiqreKyFhgfmCbxSJyM/BHoBR4R1Uvrud9rgGuAcjKysqZObNp0+UWFRWRlhaZ2SqTynZz2IYnyNzxCcXt+7Du8P9jb8chIbddubOKvy7yk5MVz/Ujk/cbDFcTV1xVOV13fkqPb9+lU8EylDgqEtNJqCxi46FX8E2v06L6DxjJz6u52pV8y+hFN7G785F81u+mJscVX1nC6EU3A3EsGv0PqhIi1zsl3M8r2b+D0YtuwZ+SxYbDpjF8+W8pS+7G0pF3UZEUuQGCnv0dtdoNbGuiaHy/pLqCfltmcsjXr1Ce1JG1R1zP7s459Nk6iwGbnqGkfU9WDv0pJam9oxpXUzQnrkmTJi1W1dEhV6qqJw/gPFy7Q83rS4EH6myTAfwTWAo8AywEsoFOwPtANyAReBW45EDvmZOTo001d+7cJu9bq6pK9fPHVe/qrfq7bqof3KNaUVbv5lt3F+vI376tJ947T4v8FY2Pa+cG1Xd/o/qv01W3fNL8uJsgIp9XJH30d9VfZ+jyF+9q+jFevV71Nx1Vv/o0YmHVaNLntfp11V9nqP66g+o/Rqju3RbpsFrf3zEgqnHlLlZ9cJz7rGt+vnCZqt/XsnGFoTlxAYu0nnOql2WmXKBP0Ove1KkmUlUfcAWAuEvnzYHHycBmVd0RWPcKcAzwbw/jbZ68lW5a5NyFrp3gtL8fsLrn2c++pqiskln/dyypyWH8Kboc6hrdzHeOvgFWvMTA9Y9C6XXhd2VcM8f1tPnerd4PvmusQafCMTe57tCXvQoZPVo6orap16hA28Q9bpDoSX9w36fWUCXWwrxsg1gIDBSR/iKShGtknh28gYh0DKwD12Ppw0DS+Bo4SkTaBxLHZGC1h7E2XXmJq/9+dLwbdXvWo66hrhFtAd8WlNK9Qwr9unp8Z7GDQXwCnP4ASeV74X9hJs/ina5nUNZw1zW4NTnp93DTEujUr6UjadsSkmHyL+Gnua6nniUHwMNeTKpaKSI3AG/jurk+paorReTawPrpwGBghohUAauAqwLrPhORl4AvgEpgCdD67vZestvdtWv7ctcIfeLvw+r1kucrIys9xcMADzI9jyS39+n0Wfy0G9DX79gD76PqSn7+vS6xt8bZN+1kFT32We/D02Z5VZ0DzKmzbHrQ8wXAwHr2/TXQeutRSve4XkU71rmBQYefHPYh8gr9DO4epRlJDxKb+19In6IvXIng2k/ciNqGfPm8m078xN+5KS+MMbXazry00eTf6+bFz1/t+qM3ITkA5PvKyMxoZX3ZY1x1fAqc9g/YtcGNsm5IwdduENQhx7g6Z2PMPixBhKusEP59rqtWOn9G/YNsDqC4rJKiskqyMqyKKeIOm+xmgv34767zQCg191DQajjrkeZPSWJMG2QJIhxlRfDseW5k5Xn/dLdibKL8wjIAsqwE4Y2T73IDoWbf5ObAquuzR2DLRzDlT9YAbEw9LEE0VnkxPHcBbP3cTTQ2+AfNOlyez00rZY3UHkntAif/Cb5Z5CYYDJa/xs2aevgpcOSlLROfMTHAEkRjVJTC81Ph6/luArWhZzX7kDUJwtogPDTifDep3Xu/g4KtblllOcy6xt0g6fT7rdeKMQ2wBHEgFX6YeZGbC+fM6RGb+jnf56qYMq0NwjsibsCiVrt5jVTdXcC+/dJN0paW2dIRGtOqWYJoSGWZmwxv4/twxoOQfUHEDp3n89MuMZ70cEZQm/B16uvuQLf+bXj3l+6eBNkXNbuK0JiDgSWI+lSWw4uXw4Z33dVmQ/dzboK8wjKyMvafnM94YNy17s518x+AjF5wyt0H3scYYwkipKoKeOkKWPemu7FPzg8j/hZ5Pr9VL0VLXLy781rmUDcVSkqHlo7ImJhg9Rt1VVXCy9Pc6NpT7nH3CvbAjsIyhvWyE1XUdB8G/ze/paMwJqZYCSJYdZW71eSqV+GkP8K4H3nyNqpKns9PVrr1YDLGtF6WIGpolRtZu+IlOOG3cIx3Uy8UlVVSUl5lXVyNMa2aVTEBVFdzxNoHYfv7rsfL927x9O3yfDWjqK0NwhjTelkJoroaXr+FHtvfhwl3uvsBeyy/ZpCcjaI2xrRiliDK9sLXC/jqkPNg4p1Recu8wsA0G1bFZIxpxSxBtOsE095jc/+LozbtQp6NojbGxABLEAApGVGdkyfP5yctOYE0G0VtjGnFLEG0gPxCu1GQMab1swTRAvJ9fpvm2xjT6lmCaAF5dqtRY0wMsAQRZbWjqK2B2hjTynmaIERkioisFZENIrJfH1IR6SQis0RkmYh8LiLDAsuPEJGlQQ+fiNziZazR4iutpKyymkybZsMY08p51o1GROKBh4ATgVxgoYjMVtVVQZv9DFiqqmeJyKDA9pNVdS0wMug43wCzvIo1mr4bA2ElCGNM6+ZlCWIssEFVN6lqOTATOKPONkOA9wBUdQ3QT0Sy6mwzGdioql95GGvU1N6L2hKEMaaVE1X15sAi5wJTVHVa4PWlwDhVvSFom7uAFFW9VUTGAvMD2ywO2uYp4AtVfbCe97kGuAYgKysrZ+bMmU2Kt6ioiLS0tCbtG46Pv6ngieXl3DO+HZntD5yfoxVXuCyu8Fhc4bG4wtOcuCZNmrRYVUeHXKmqnjyA84Angl5fCjxQZ5sM4J/AUuAZYCGQHbQ+CdgJZDXmPXNycrSp5s6d2+R9w/HQ3PXa947XtaSsslHbRyuucFlc4bG4wmNxhac5cQGLtJ5zqpdDeXOBPkGvewPbgjdQVR9wBYC4e29uDjxqnIIrPeR5GGdU5fvKyEhJoF1SfEuHYowxDfKyDWIhMFBE+otIEjAVmB28gYh0DKwDmAZ8GEgaNS4EnvcwxqizW40aY2KFZyUIVa0UkRuAt4F44ClVXSki1wbWTwcGAzNEpApYBVxVs7+ItMf1gPLmtm4txI2BsC6uxpjWz9PZ4lR1DjCnzrLpQc8XAAPr2bcE6OJlfC0hz1fGuP6dWzoMY4w5IBtJHUWqSn6hVTEZY2KDJYgo2lNSQUWVWhWTMSYmWIKIIhskZ4yJJZYgoii/0N1JzkoQxphYYAkiimpKEJl2LwhjTAywBBFF+YEE0c1mcjXGxABLEFGU5yujY/tEUhJtFLUxpvWzBBFFeXarUWNMDLEEEUV5hXarUWNM7LAEEUX5dqtRY0wMsQQRJdXVyo7CMuviaoyJGfUmCBG5p2ZivTrLfywif/Y2rLZnd0k5ldVqJQhjTMxoqARxGvBYiOX3Aad6E07bZWMgjDGxpqEEoapaHWJhNSDehdQ25fvcKGprpDbGxIqGEkSJiOw3FXdgWal3IbVNNg+TMSbWNHQ/iF8Bb4rIH4DFgWWjgZ8Ct3gcV5uTFyhBdEuzEoQxJjbUmyBU9U0RORO4HbgxsHgFcI6qLo9CbG1KXqGfLqlJJCVYxzFjTGyoN0GISAqQp6qX11meKSIpqur3PLo2JN/uRW2MiTENXc7eDxwXYvmJwN+9CaftyvPZGAhjTGxpKEF8T1VfqbtQVZ8FxnsXUtuUX2jzMBljYktDCaKhrqyNqkgXkSkislZENojInSHWdxKRWSKyTEQ+F5FhQes6ishLIrJGRFaLyNGNec/WqMpGURtjYlBDJ/p8ERlbd2Fg2Y4DHVhE4oGHgFOAIcCFIjKkzmY/A5aq6gjgMtwgvBr3AW+p6iAgG1h9oPdsrXYVlVGt0M3aIIwxMaShbq63Ay+KyNPs2831MmBqI449FtigqpsARGQmcAawKmibIcCfAFR1jYj0E5Es3DiL8cAPA+vKgfLG/UqtT00X1yy7UZAxJobUW4JQ1c+Bcbiqph8CNb2ZLscliQPpBWwNep0bWBbsS+BsqC2Z9AV6AwNwpZR/isgSEXlCRFIb8Z6tkg2SM8bEIlHVA28kciRwIXA+sBl4WVUfPMA+5wEnq+q0wOtLgbGqemPQNhm4qqQjgeXAIGAakAh8Chyrqp+JyH2AT1V/GeJ9rgGuAcjKysqZOXPmAX+fUIqKikhLS2vSvgcy9+sK/rWqnL9PbEenlPDGQXgZV3NYXOGxuMJjcYWnOXFNmjRpsaqODrlSVUM+gMNxo6lXAx/jBst9Vd/2IfY/Gng76PVPgZ82sL0AW4AMoDuwJWjdccAbB3rPnJwcbaq5c+c2ed8D+ds7a7Xfna9rRWVV2Pt6GVdzWFzhsbjCY3GFpzlxAYu0nnNqQ5eza4DJwA9U9Xuq+gBQFUZiWggMFJH+IpKEa7eYHbxBoKdSUuDlNOBDVfWp6nZgq4gcEVg3mX3bLmJKvs9P17RkEuJtFLUxJnY01Eh9Du6kPldE3gJmEsYsrqpaKSI3AG8D8cBTqrqy5h4TqjodGAzMEJEqXAK4KugQNwLPBhLIJuCKxv9arUu+dXE1xsSghuZimgXMCjQOnwn8GMgSkUeAWar6zoEOrqpzgDl1lk0Per4A2G/G2MC6pbheUzEvz+enuzVQG2NizAHrPFS1WFWfVdXTcD2MlgL7DXoz9cvzldl9IIwxMSesSnFV3a2qj6rq8V4F1NZUVFWzq7jM7iRnjIk51mrqsZ1FZajaGAhjTOyxBOGx2lHUVsVkjIkxliA8ZqOojTGxyhKEx/IDCcIaqY0xscYShMfyC8uIjxO6pFqCMMbEFksQHsvz+emWlkx8XKPHGBpjTKtgCcJjNgbCGBOrLEF4LM/ntzEQxpiYZAnCYzYPkzEmVlmC8FBZZRW7i8uti6sxJiZZgvDQjkIbJGeMiV2WIDxUM4o600oQxpgYZAnCQzsKA6OorZHaGBODLEF4yOZhMsbEMksQHsrz+UmIEzq1TzrwxsYY08pYgvBQnq+MzPRk4mwUtTEmBlmC8FB+od8aqI0xMcsShIfyfH5rfzDGxCxLEB7K85XZIDljTMzyNEGIyBQRWSsiG0TkzhDrO4nILBFZJiKfi8iwoHVbRGS5iCwVkUVexukFf0UVe0srLEEYY2JWglcHFpF44CHgRCAXWCgis1V1VdBmPwOWqupZIjIosP3koPWTVHWnVzF6Kb9mkFy6VTEZY2KTlyWIscAGVd2kquXATOCMOtsMAd4DUNU1QD8RyfIwpqjJL7RbjRpjYpuoqjcHFjkXmKKq0wKvLwXGqeoNQdvcBaSo6q0iMhaYH9hmsYhsBvYACjyqqo/V8z7XANcAZGVl5cycObNJ8RYVFZGWltakfUP5fHslDy8t4/fHtqNPetPzcKTjihSLKzwWV3gsrvA0J65JkyYtVtXRIVeqqicP4DzgiaDXlwIP1NkmA/gnsBR4BlgIZAfW9Qz8zAS+BMYf6D1zcnK0qebOndvkfUN58qNN2veO13V3UVmzjhPpuCLF4gqPxRUeiys8zYkLWKT1nFM9a4PAtTv0CXrdG9gWvIGq+oArAEREgM2BB6q6LfAzX0Rm4aqsPvQw3ojKK/STFB9Hx/aJLR2KMcY0iZdtEAuBgSLSX0SSgKnA7OANRKRjYB3ANOBDVfWJSKqIpAe2SQVOAlZ4GGvE5QduNerynjHGxB7PShCqWikiNwBvA/HAU6q6UkSuDayfDgwGZohIFbAKuCqwexYwK3ByTQCeU9W3vIrVC26QnDVQG2Nil5dVTKjqHGBOnWXTg54vAAaG2G8TkO1lbF7L8/k5ont6S4dhjDFNZiOpPZLvKyPT7gNhjIlhliA8UFJeSWFZJZk2D5MxJoZZgvBAzShqu5OcMSaWWYLwQJ7PRlEbY2KfJQgP5BXarUaNMbHPEoQH8gMlCLtZkDEmllmC8ECez09KYhwZKZ72IjbGGE9ZgvBAzY2CbBS1MSaWWYLwQJ7Pbz2YjDExzxKEB3YUltHNGqiNMTHOEoQHrARhjGkLLEFEWFFZJcXlVdbF1RgT8yxBRJgNkjPGtBWWICIsr3YMhJUgjDGxzRJEhNXOw2QlCGNMjLMEEWFWxWSMaSssQURYnq+M1KR40pJtFLUxJrZZgoiw/EK/zcFkjGkTLEFEmLuTnDVQG2NinyWICMsr9Fv7gzGmTbAEEUGq6kZRWxdXY0wb4GmCEJEpIrJWRDaIyJ0h1ncSkVkiskxEPheRYXXWx4vIEhF53cs4I8Xnr8RfUW0lCGNMm+BZghCReOAh4BRgCHChiAyps9nPgKWqOgK4DLivzvqbgdVexRhpdqMgY0xb4mUJYiywQVU3qWo5MBM4o842Q4D3AFR1DdBPRLIARKQ3cCrwhIcxRlRezSA5a6Q2xrQBoqreHFjkXGCKqk4LvL4UGKeqNwRtcxeQoqq3ishYYH5gm8Ui8hLwJyAduE1VT6vnfa4BrgHIysrKmTlzZpPiLSoqIi0trUn71vjkmwoeX17On49rR1ZqZHJvJOLygsUVHosrPBZXeJoT16RJkxar6uiQK1XVkwdwHvBE0OtLgQfqbJMB/BNYCjwDLASygdOAhwPbTAReb8x75uTkaFPNnTu3yfvWeHjuBu17x+taXFbR7GPViERcXrC4wmNxhcfiCk9z4gIWaT3nVC+H++YCfYJe9wa2BW+gqj7gCgBx9+fcHHhMBU4Xke8DKUCGiPxbVS/xMN5my/P5SU9OoH2SjaI2xsQ+L9sgFgIDRaS/iCThTvqzgzcQkY6BdQDTgA9V1aeqP1XV3qraL7Df+609OUDNKGprfzDGtA2eXeqqaqWI3AC8DcQDT6nqShG5NrB+OjAYmCEiVcAq4Cqv4omGPF+ZdXE1xgMVFRXk5ubi9/tbNI4OHTqwenXr61jZmLhSUlLo3bs3iYmJjT6up3UhqjoHmFNn2fSg5wuAgQc4xjxgngfhRVyez8+Yfp1bOgxj2pzc3FzS09Pp168frja6ZRQWFpKent5i71+fA8WlquzatYvc3Fz69+/f6OPaSOoIUVU3D5NVMRkTcX6/ny5durRocohlIkKXLl3CLoFZgoiQgpIKyquqyUq3KiZjvGDJoXma8vlZgoiQvEK7UZAxpm2xBBEhNbcatSomY0xbYQkiQmpvNWpVTMa0OQUFBTz88MNh7/f973+fgoKCyAcUJTaiK0LyC60EYUw0/Pa1laza5ovoMYf0zODXPxha7/qaBHHppZfus7yqqor4+Ph695szZ06962KBlSAiJM/np0O7RFIS6/+yGGNi05133snGjRs59thjGTNmDJMmTeKiiy5i+PDhAJx55pnk5OQwdOhQHnvssdr9+vXrx86dO9myZQuDBw/m6quvZujQoZx00kmUlpbW+36PP/44Y8aMITs7m3POOYeSkhIA8vLyOOuss8jOziY7O5v58+cD8NxzzzFixAiys7P3S2LNYSWICLEbBRkTHQ1d6Xvl7rvvZsWKFXzyyScsXryYU089lRUrVtSOKXjqqafo3LkzpaWljBkzhnPOOYcuXbrsc4z169fz/PPP8/jjj3P++efz8ssvc8kloSeIOPvss7n66qsB+MUvfsGTTz7JjTfeyE033cSECROYNWsWVVVVFBUVsXLlSv7617+yYMECunbtyu7duyP2e1uCiBAbRW3MwWPs2LH7DDi7//77mTVrFgBbt25l/fr1+yWI/v37M3LkSABycnLYsmVLvcdfsWIFv/jFLygoKKCoqIiTTz4ZgPfff58ZM2YAEB8fT4cOHZgxYwZnnnkmXbt2BaBz58gN1rUEESH5Pj+Hduva0mEYY6IgNTW19vm8efP43//+x4IFC2jfvj0TJ04MOSAtOfm7Gob4+PgGq5h++MMf8uqrr5Kdnc3TTz/NvHnz6t1WPbplA1gbRERUVyv5hWVWxWRMG5Wenk5hYWHIdXv37qVTp060b9+eNWvW8Omnnzb7/QoLC+nRowcVFRU8++yztcsnT57MI488ArgGcp/Px+TJk5k1axa7du0CiGgVkyWICNhTUk5ltZJpd5Izpk3q0qULxx57LOPGjeP222/fZ92UKVOorKxkxIgR/PKXv+Soo45q9vv9/ve/Z9y4cZx44okMGjSodvl9993H3LlzGT58ODk5OaxcuZKhQ4dy2223MWHCBLKzs7n11lub/f41rIopAmpvNWptEMa0Wc8991zISfGSk5N58803Q+5T087QtWtXVqxYUbv8tttua/C9rrvuOq677rr9lmdlZfHf//53v+UXX3wx11577YF+hbBZCSICaqbZyLQEYYxpQ6wEEQH5NaOorQ3CGBOG66+/nk8++WSfZTfffDNXXHFFC0W0L0sQEVBTxdTN2iCMMWF46KGHWjqEBlkVUwTk+fx0Tk0iOcFGURtj2g5LEBGQ5yuzHkzGmDbHEkQE5Bf6rQeTMabNsQQRAflWgjDGtEGWIJqpqlrZUWTzMBljvpOWlgbAtm3bOPfcc0NuM3HiRBYtWhTNsMLmaS8mEZkC3AfEA0+o6t111ncCngIOBfzAlaq6QkRSgA+B5ECML6nqr72Mtal2FZdRVa3WxdWYaHnzTti+PLLH7D4cTrn7wNuFqWfPnrz00ksRP260eFaCEJF44CHgFGAIcKGIDKmz2c+Apao6ArgMl0wAyoDjVTUbGAlMEZHmj1/3wHe3GrUShDFt1R133LHPHeV+85vf8Nvf/pbJkyczatQohg8fHnKE85YtWxg2bBgApaWlTJ06lREjRnDBBRc0OFkfuNHUo0ePZujQofz6199dHy9cuJBjjjmG7Oxsxo4dS2FhIVVVVdx2220MHz6cESNG8MADD0Tk9/ayBDEW2KCqmwBEZCZwBrAqaJshwJ8AVHWNiPQTkSxVzQOKAtskBh7eTVnYDLW3GrUEYUx0eHClfyBTp07llltuqb0Zz4svvshbb73Fj3/8YzIyMti5cydHHXUUp59+OiIS8hiPPPII7du3Z9myZSxbtoxRo0Y1+J5//OMf6dy5M1VVVUyePJlly5YxaNAgLrjgAl544QXGjBmDz+ejXbt2PPDAA2zevJklS5aQkJAQsQn7vEwQvYCtQa9zgXF1tvkSOBv4WETGAn2B3kBeoASyGDgMeEhVPwv1JiJyDXANuHlKGpoWtyFFRUVN2vejrRUAbFr5BQUbI18ga2pcXrO4wmNxhaduXB06dKh3NtVoOOyww9i+fTu5ubns2bOHjIwM0tLSuO2225g/fz5xcXF88803bNy4kaysLMDNyFpUVER1dTWFhYW8//77XHvttRQWFtK/f3+GDRtGcXFxvb/XjBkzePrpp6msrGT79u0sXryYkpISMjMzGTRoEIWFhYgIpaWlzJ07l6uuuqq2VJKYmBjyuH6/P6y/t5cJIlQarVsKuBu4T0SWAsuBJUAlgKpWASNFpCMwS0SGqeqKOvujqo8BjwGMHj1aJ06c2KRg582bR1P2XfLuOmTVen5w4kQS4yOfIJoal9csrvBYXOGpG9fq1av3myQv2s4//3xee+01CgoKuPjii5k9ezZ79+5lyZIlJCYm0q9fPxISEmrjTE9PJy0tjbi4ONLT00lISCA1NbV2fVxc3D6vg23evJkHH3yQhQsX0qlTJ374wx8iIrRv336f9whW37GCpaSkcOSRRzb6d/ayF1Mu0CfodW9gW/AGqupT1StUdSSuDaIbsLnONgXAPGCKh7E2WX6hny6pyZ4kB2NM6zF16lRefvllXnrpJc4991z27t1LZmYmiYmJzJ07l6+++qrB/cePH197b4cVK1awbNmyerf1+XykpqbSoUMH8vLyameLHTRoENu2bWPhwoWAK6VUVlZy/PHHM336dCorK4HI3RPCy7PaQmCgiPQXkSRgKjA7eAMR6RhYBzAN+FBVfSLSLVByQETaAScAazyMtclsDIQxB4ehQ4dSVFREr1696NGjBxdffDGLFi1i9OjRPPvss/vctyGU6667jqKiIkaMGME999zD2LFj6902OzubI488kqFDh3LllVdy7LHHApCUlMQLL7zAjTfeSHZ2NieeeCJ+v5/LL7+cQw45hBEjRpCdnc1zzz0Xkd/ZsyomVa0UkRuAt3HdXJ9S1ZUicm1g/XRgMDBDRKpwjddXBXbvAfwr0A4RB7yoqq97FWtzjOjdkSE9q1s6DGNMFHz66ae11Thdu3ZlwYIFIbcrKnJ9bPr161d7H4h27doxc+bMRr/X008/HXL5mDFj9rtrXWFhIffeey/33ntvo4/fGJ6Og1DVOcCcOsumBz1fAAwMsd8yoPEVZS3o5hP2C98YY9oEm+7bGGNa0Lhx4ygrK9tn2TPPPMPw4cNbKKLvWIIwxsQEVa13jEEs++yzkD34I041/KFk1vXGGNPqpaSksGvXriad5IxLDrt27SIlJbwBvVaCMMa0er179yY3N5cdO3a0aBx+vz/sk2w0NCaulJQUevfuHdZxLUEYY1q9xMRE+vfv39JhMG/evLAGmkWLV3FZFZMxxpiQLEEYY4wJyRKEMcaYkKQt9QoQkR1AwxOi1K8rsDOC4USKxRUeiys8Fld42mJcfVW1W6gVbSpBNIeILFLV0S0dR10WV3gsrvBYXOE52OKyKiZjjDEhWYIwxhgTkiWI7zzW0gHUw+IKj8UVHosrPAdVXNYGYYwxJiQrQRhjjAnJEoQxxpiQDqoEISJTRGStiGwQkTtDrBcRuT+wfpmIjIpSXH1EZK6IrBaRlSJyc4htJorIXhFZGnj8KkqxbRGR5YH3XBRifdQ/MxE5IuhzWCoiPhG5pc42Ufm8ROQpEckXkRVByzqLyLsisj7ws1M9+zb4ffQgrr+IyJrA32lWzW19Q+zb4N/cg7h+IyLfBP2tvl/PvtH+vF4IimmLiCytZ18vP6+Q54aofcdU9aB44G57uhEYACQBXwJD6mzzfeBNQICjgM+iFFsPYFTgeTqwLkRsE4HXW+Bz2wJ0bWB9i3xmdf6u23GDfaL+eQHjgVHAiqBl9wB3Bp7fCfy5Kd9HD+I6CUgIPP9zqLga8zf3IK7fALc14u8c1c+rzvq/Ab9qgc8r5LkhWt+xg6kEMRbYoKqbVLUcmAmcUWebM4AZ6nwKdBSRHl4HpqrfquoXgeeFwGqgl9fvGyEt8pkFmQxsVNWmjqBvFlX9ENhdZ/EZwL8Cz/8FnBli18Z8HyMal6q+o6qVgZefAuHN/exRXI0U9c+rhri7FJ0PPB+p92usBs4NUfmOHUwJohewNeh1LvufhBuzjadEpB/uftyhbjN1tIh8KSJvisjQKIWkwDsislhErgmxvqU/s6nU/4/bEp8XQJaqfgvuHxzIDLFNS39uV+JKfqEc6G/uhRsCVV9P1VNd0pKf13FAnqqur2d9VD6vOueGqHzHDqYEEepehXX7+DZmG8+ISBrwMnCLqvrqrP4CV42SDTwAvBqlsI5V1VHAKcD1IjK+zvoW+8xEJAk4HfhPiNUt9Xk1Vkt+bj8HKoFn69nkQH/zSHsEOBQYCXyLq86pqyX/Ny+k4dKD55/XAc4N9e4WYllYn9nBlCBygT5Br3sD25qwjSdEJBH3BXhWVV+pu15VfapaFHg+B0gUka5ex6Wq2wI/84FZuGJrsBb7zHD/kF+oal7dFS31eQXk1VSzBX7mh9imRT43EbkcOA24WAMV1XU14m8eUaqap6pVqloNPF7P+7XU55UAnA28UN82Xn9e9ZwbovIdO5gSxEJgoIj0D1x5TgVm19lmNnBZoGfOUcDemmKclwJ1nE8Cq1X13nq26R7YDhEZi/vb7fI4rlQRSa95jmvkXFFnsxb5zALqvbJric8ryGzg8sDzy4H/htimMd/HiBKRKcAdwOmqWlLPNo35m0c6ruA2q7Pqeb+of14BJwBrVDU31EqvP68Gzg3R+Y550fLeWh+4HjfrcC37Pw8suxa4NvBcgIcC65cDo6MU1/dwRb9lwNLA4/t1YrsBWInrifApcEwU4hoQeL8vA+/dmj6z9rgTfoegZVH/vHAJ6lugAnfFdhXQBXgPWB/42TmwbU9gTkPfR4/j2oCrk675jk2vG1d9f3OP43om8N1ZhjuB9WgNn1dg+dM136mgbaP5edV3bojKd8ym2jDGGBPSwVTFZIwxJgyWIIwxxoRkCcIYY0xIliCMMcaEZAnCGGNMSJYgjAmDiFTJvjPJRmxWURHpFzybqDEtLaGlAzAmxpSq6siWDsKYaLAShDERELgnwJ9F5PPA47DA8r4i8l5gIrr3ROSQwPIscfdk+DLwOCZwqHgReTww9/87ItKuxX4pc9CzBGFMeNrVqWK6IGidT1XHAg8C/wgsexA3HfoI3OR49weW3w98oG4ywVG4UbgAA4GHVHUoUACc4+lvY0wDbCS1MWEQkSJVTQuxfAtwvKpuCkyutl1Vu4jITtzUERWB5d+qalcR2QH0VtWyoGP0A95V1YGB13cAiar6hyj8asbsx0oQxkSO1vO8vm1CKQt6XoW1E5oWZAnCmMi5IOjngsDz+bhZNAEuBj4OPH8PuA5AROJFJCNaQRrTWHZ1Ykx42sm+N69/S1Vruromi8hnuAuvCwPLbgKeEpHbgR3AFYHlNwOPichVuJLCdbjZRI1pNawNwpgICLRBjFbVnS0dizGRYlVMxhhjQrIShDHGmJCsBGGMMSYkSxDGGGNCsgRhjDEmJEsQxhhjQrIEYYwxJqT/DxWfGlfA9aCMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "metrics = pd.read_csv(f\"{trainer.logger.log_dir}/metrics.csv\")\n",
    "\n",
    "aggreg_metrics = []\n",
    "agg_col = \"epoch\"\n",
    "for i, dfg in metrics.groupby(agg_col):\n",
    "    agg = dict(dfg.mean())\n",
    "    agg[agg_col] = i\n",
    "    aggreg_metrics.append(agg)\n",
    "\n",
    "df_metrics = pd.DataFrame(aggreg_metrics)\n",
    "df_metrics[[\"train_loss\", \"valid_loss\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='Loss')\n",
    "df_metrics[[\"train_acc\", \"valid_acc\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='ACC')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cbe0151",
   "metadata": {},
   "source": [
    "- The `trainer` automatically saves the model with the best validation accuracy automatically for us, we which we can load from the checkpoint via the `ckpt_path='best'` argument; below we use the `trainer` instance to evaluate the best model on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b726c1a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at logs/my-model/version_22/checkpoints/epoch=10-step=2353.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from checkpoint at logs/my-model/version_22/checkpoints/epoch=10-step=2353.ckpt\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "02006ec6d039406f8c71dcda15169e82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">        Test metric        </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9873999953269958     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m       Test metric       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9873999953269958    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'test_acc': 0.9873999953269958}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.test(model=lightning_model, datamodule=data_module, ckpt_path='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "461b364a",
   "metadata": {},
   "source": [
    "## Predicting labels of new data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f32de2de",
   "metadata": {},
   "source": [
    "- You can use the `trainer.predict` method on a new `DataLoader` or `DataModule` to apply the model to new data.\n",
    "- Alternatively, you can also manually load the best model from a checkpoint as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ba4570eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logs/my-model/version_22/checkpoints/epoch=10-step=2353.ckpt\n"
     ]
    }
   ],
   "source": [
    "path = trainer.checkpoint_callback.best_model_path\n",
    "print(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8c1f7796",
   "metadata": {},
   "outputs": [],
   "source": [
    "lightning_model = LightningModel.load_from_checkpoint(\n",
    "    path, model=pytorch_model)\n",
    "lightning_model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4389e871",
   "metadata": {},
   "source": [
    "- Note that our PyTorch model, which is passed to the Lightning model requires input arguments. However, this is automatically being taken care of since we used `self.save_hyperparameters()` in our PyTorch model's `__init__` method.\n",
    "- Now, below is an example applying the model manually. Here, pretend that the `test_dataloader` is a new data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3ff7cd66",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([7, 2, 1, 0, 4])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataloader = data_module.test_dataloader()\n",
    "\n",
    "all_true_labels = []\n",
    "all_predicted_labels = []\n",
    "for batch in test_dataloader:\n",
    "    features, labels = batch\n",
    "    \n",
    "    with torch.no_grad():  # since we don't need to backprop\n",
    "        logits = lightning_model(features)\n",
    "    predicted_labels = torch.argmax(logits, dim=1)\n",
    "    all_predicted_labels.append(predicted_labels)\n",
    "    all_true_labels.append(labels)\n",
    "    \n",
    "all_predicted_labels = torch.cat(all_predicted_labels)\n",
    "all_true_labels = torch.cat(all_true_labels)\n",
    "all_predicted_labels[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff0a65db",
   "metadata": {},
   "source": [
    "Just as an internal check, if the model was loaded correctly, the test accuracy below should be identical to the test accuracy we saw earlier in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6c06ef45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9874 (98.74%)\n"
     ]
    }
   ],
   "source": [
    "test_acc = torch.mean((all_predicted_labels == all_true_labels).float())\n",
    "print(f'Test accuracy: {test_acc:.4f} ({test_acc*100:.2f}%)')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e80fe10",
   "metadata": {},
   "source": [
    "## Single-image usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "335380c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "45998e5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab0bdec5",
   "metadata": {},
   "source": [
    "- Assume we have a single image as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "9fd24838",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAN8UlEQVR4nO3dUYxUdZbH8d+RHaKREUFa7Tgg7GiMZBNhrOAmbibquBPxQSRmzGAygunImEAcDA9L2BjQJ7JhZpiHzSSM4MBmFjIRjDzg7BiCUTQBCwMtLrqiYQdGAk0woSdRseHsQ182DXb9q6l7b93qPt9P0qmqe+rW/1jy61tV/9v1N3cXgLHvqqobANAehB0IgrADQRB2IAjCDgTxd+0cbMqUKT59+vR2DgmEcvToUZ0+fdqGq+UKu5k9JOk3ksZJesnd16TuP336dNXr9TxDAkio1WoNay2/jDezcZL+XdJcSTMlLTCzma0+HoBy5XnPPkfSEXf/zN3PSdoqaV4xbQEoWp6w3yLp2JDbx7NtlzCzxWZWN7N6X19fjuEA5JEn7MN9CPCtc2/dfb2719y91tXVlWM4AHnkCftxSVOH3P6epM/ztQOgLHnC/p6k281shpmNl/RTSTuKaQtA0VqeenP3ATNbKum/NDj1ttHdPyysMwCFyjXP7u47Je0sqBcAJeJ0WSAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQuZZsNrOjkvolnZc04O61IpoCULxcYc/c7+6nC3gcACXiZTwQRN6wu6Q/m9l+M1s83B3MbLGZ1c2s3tfXl3M4AK3KG/Z73f0HkuZKWmJmP7z8Du6+3t1r7l7r6urKORyAVuUKu7t/nl2ekvSqpDlFNAWgeC2H3cyuNbPvXrwu6ceSDhXVGIBi5fk0/iZJr5rZxcf5T3f/UyFdAShcy2F3988k3VVgLwBKxNQbEARhB4Ig7EAQhB0IgrADQRTxhzDI6d13303We3t7k/Wenp6Gtf7+/uS+b775ZrJ+6FB1p07cf//9yfqdd97Z8mPfcMMNyXo2pTymcGQHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSCYZ2+DtWvXJuurVq1K1r/66qtkffv27Q1rb7/9dnLfc+fOJetVeuGFF0p77McffzxZbzbP/uKLLybrt9122xX3VDaO7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQhLl72war1Wper9fbNl6RUs/Tnj17kvvOnTs3Wf/yyy9b6qkI48aNS9bvvvvuZP2jjz5K1pudI5AyMDCQrF+4cKHlx85r4sSJyfqZM2fa1MmlarWa6vX6sCcJcGQHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSD4e/bM119/nayvW7euYW3lypUFd3Opa665JllfsGBBw9r8+fOT+95zzz3JerPvVy/TkSNHkvV58+Yl683OAchj/PjxpT12WZoe2c1so5mdMrNDQ7ZNNrM3zOyT7HJSuW0CyGskL+N/L+mhy7atkLTL3W+XtCu7DaCDNQ27u78l6fJz/+ZJ2pRd3yTp0WLbAlC0Vj+gu8ndT0hSdnljozua2WIzq5tZva+vr8XhAORV+qfx7r7e3WvuXuvq6ip7OAANtBr2k2bWLUnZ5aniWgJQhlbDvkPSwuz6QkmvFdMOgLI0nWc3sy2S7pM0xcyOS1olaY2kP5pZj6S/SPpJmU0W4eOPP07WFy1alKzv27evwG4utXTp0mR9+fLlyfq0adOKbKdjHDx4MFkvcx69u7s7WX/nnXdKG7ssTcPu7o3O2PhRwb0AKBGnywJBEHYgCMIOBEHYgSAIOxBEmD9xfe6555L1PFNrzZb3ffbZZ5P1Zsv/Tpgw4Yp7Gg2OHTuWrD/zzDOljd3s/9m2bduS9VtvvbXIdtqCIzsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBBFmnv31119P1q+6Kv177+abb25YW716dXLfp59+Olkfy86dO9ew9sgjjyT3LXPZ47Vr1ybrzb5iezTiyA4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQYSZZ3/ppZeS9bNnzybrTz31VMPa9ddf30pLY8LAwECynvqa7N7e3qLbucTMmTMb1pp9x8BYxJEdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4IIM8/e09NTdQuj0vnz55P1/fv3J+sbNmwosp1L3HXXXcn67t27G9aafX/BWNT0v9jMNprZKTM7NGTbajP7q5kdyH4eLrdNAHmN5Nfb7yU9NMz2X7v7rOxnZ7FtASha07C7+1uSyvt+IABtkeeNy1Iz681e5k9qdCczW2xmdTOr9/X15RgOQB6thv23kr4vaZakE5J+2eiO7r7e3WvuXuvq6mpxOAB5tRR2dz/p7ufd/YKk30maU2xbAIrWUtjNrHvIzfmSDjW6L4DO0HSe3cy2SLpP0hQzOy5plaT7zGyWJJd0VNLPy2sRVdq6dWuy/uSTT5Y29owZM5L1LVu2JOsTJ04ssp1Rr2nY3X3BMJvLO1MCQCninUYEBEXYgSAIOxAEYQeCIOxAEGH+xBXD27t3b7K+ZMmSNnXybc2m/e644442dTI2cGQHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSCYZx/jvvnmm2R92bJlyXp/f3+B3Vzq+eefT9Znz55d2tgRcWQHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSCYZx/jXnnllWR93759pY7/2GOPNaytWLEiue+4ceOKbic0juxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EATz7GPc8uXLS338KVOmJOubN29uWLv66quLbgcJTY/sZjbVzHab2WEz+9DMfpFtn2xmb5jZJ9nlpPLbBdCqkbyMH5C03N3vlPSPkpaY2UxJKyTtcvfbJe3KbgPoUE3D7u4n3P397Hq/pMOSbpE0T9Km7G6bJD1aUo8ACnBFH9CZ2XRJsyXtlXSTu5+QBn8hSLqxwT6LzaxuZvW+vr6c7QJo1YjDbmYTJG2TtMzdz450P3df7+41d691dXW10iOAAowo7Gb2HQ0G/Q/uvj3bfNLMurN6t6RT5bQIoAhNp97MzCRtkHTY3X81pLRD0kJJa7LL10rpEE2tWbOmYe3kyZOljv3EE08k60yvdY6RzLPfK+lnkj4wswPZtpUaDPkfzaxH0l8k/aSUDgEUomnY3X2PJGtQ/lGx7QAoC6fLAkEQdiAIwg4EQdiBIAg7EAR/4joKfPHFF8n6unXrSht7/vz5yfratWtLGxvF4sgOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0Ewzz4KHDx4MFkv8+u+pk2blqyzrPLowZEdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Jgnn0U2LlzZ2VjP/jgg5WNjWJxZAeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIEayPvtUSZsl3SzpgqT17v4bM1st6WlJF/+YeqW7VzchPIYtWrQoWX/55Zcb1s6ePZvcd+nSpcn6Aw88kKxj9BjJSTUDkpa7+/tm9l1J+83sjaz2a3dnlQBgFBjJ+uwnJJ3Irveb2WFJt5TdGIBiXdF7djObLmm2pL3ZpqVm1mtmG81sUoN9FptZ3czqZX59EoC0EYfdzCZI2iZpmbuflfRbSd+XNEuDR/5fDrefu69395q717q6uvJ3DKAlIwq7mX1Hg0H/g7tvlyR3P+nu5939gqTfSZpTXpsA8moadjMzSRskHXb3Xw3Z3j3kbvMlHSq+PQBFGcmn8fdK+pmkD8zsQLZtpaQFZjZLkks6KunnJfQHSTNnzkzWP/3004a1gYGB5L6TJ09uqSeMPiP5NH6PJBumxJw6MIpwBh0QBGEHgiDsQBCEHQiCsANBEHYgCL5Kegy47rrrqm4BowBHdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0Iwty9fYOZ9Un63yGbpkg63bYGrkyn9tapfUn01qoie7vV3Yf9/re2hv1bg5vV3b1WWQMJndpbp/Yl0Vur2tUbL+OBIAg7EETVYV9f8fgpndpbp/Yl0Vur2tJbpe/ZAbRP1Ud2AG1C2IEgKgm7mT1kZh+b2REzW1FFD42Y2VEz+8DMDphZveJeNprZKTM7NGTbZDN7w8w+yS6HXWOvot5Wm9lfs+fugJk9XFFvU81st5kdNrMPzewX2fZKn7tEX2153tr+nt3Mxkn6H0n/LOm4pPckLXD3/25rIw2Y2VFJNXev/AQMM/uhpL9J2uzu/5Bt+zdJZ9x9TfaLcpK7/0uH9LZa0t+qXsY7W62oe+gy45IelbRIFT53ib4eVxuetyqO7HMkHXH3z9z9nKStkuZV0EfHc/e3JJ25bPM8SZuy65s0+I+l7Rr01hHc/YS7v59d75d0cZnxSp+7RF9tUUXYb5F0bMjt4+qs9d5d0p/NbL+ZLa66mWHc5O4npMF/PJJurLifyzVdxrudLltmvGOeu1aWP8+rirAPt5RUJ83/3evuP5A0V9KS7OUqRmZEy3i3yzDLjHeEVpc/z6uKsB+XNHXI7e9J+ryCPobl7p9nl6ckvarOW4r65MUVdLPLUxX38/86aRnv4ZYZVwc8d1Uuf15F2N+TdLuZzTCz8ZJ+KmlHBX18i5ldm31wIjO7VtKP1XlLUe+QtDC7vlDSaxX2colOWca70TLjqvi5q3z5c3dv+4+khzX4ifynkv61ih4a9PX3kg5mPx9W3ZukLRp8WfeNBl8R9Ui6QdIuSZ9kl5M7qLf/kPSBpF4NBqu7ot7+SYNvDXslHch+Hq76uUv01ZbnjdNlgSA4gw4IgrADQRB2IAjCDgRB2IEgCDsQBGEHgvg/av0i7OGhSbEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from PIL import Image\n",
    "\n",
    "\n",
    "image = Image.open('data/mnist_pngs/613.png')\n",
    "plt.imshow(image, cmap='Greys')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "039753a3",
   "metadata": {},
   "source": [
    "- Note that we used a resize-transformation in the `DataModule` that rescaled the 28x28 MNIST images to size 32x32. We also have to apply the same transformation to any new image that we feed to the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a8944f63",
   "metadata": {},
   "outputs": [],
   "source": [
    "resize_transform = transforms.Compose(\n",
    "            [transforms.Resize((32, 32)),\n",
    "             transforms.ToTensor()])\n",
    "\n",
    "image_chw = resize_transform(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0937a26c",
   "metadata": {},
   "source": [
    "- Note that `ToTensor` returns the image in the CHW format. CHW refers to the dimensions and stands for channel, height, and width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e4b2b30e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 32, 32])\n"
     ]
    }
   ],
   "source": [
    "print(image_chw.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a36612a",
   "metadata": {},
   "source": [
    "- However, the PyTorch / PyTorch Lightning model expectes images in NCHW format, where N stands for the number of images (e.g., in a batch).\n",
    "- We can add the additional channel dimension via `unsqueeze` as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "740d681f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 1, 32, 32])\n"
     ]
    }
   ],
   "source": [
    "image_nchw = image_chw.unsqueeze(0)\n",
    "print(image_nchw.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "774b7498",
   "metadata": {},
   "source": [
    "- Now that we have the image in the right format, we can feed it to our classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2be68b34",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():  # since we don't need to backprop\n",
    "    logits = lightning_model(image_nchw)\n",
    "    probas = torch.softmax(logits, axis=1)\n",
    "    predicted_label = torch.argmax(probas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "10c37394",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted label: 7\n",
      "Class-membership probability 99.99%\n"
     ]
    }
   ],
   "source": [
    "print(f'Predicted label: {predicted_label}')\n",
    "print(f'Class-membership probability {probas[0][predicted_label]*100:.2f}%')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}