pytorch-lightning_ipynb/cnn/cnn-densenet121-mnist.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d4312f94",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch            : 1.10.1\n",
      "pytorch_lightning: 1.5.10\n",
      "torchmetrics     : 0.6.2\n",
      "matplotlib       : 3.3.4\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -p torch,pytorch_lightning,torchmetrics,matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fd249fbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext pycodestyle_magic\n",
    "%flake8_on --ignore W291,W293,E703"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fed5208",
   "metadata": {},
   "source": [
    "<a href=\"https://pytorch.org\"><img src=\"https://raw.githubusercontent.com/pytorch/pytorch/master/docs/source/_static/img/pytorch-logo-dark.svg\" width=\"90\"/></a> &nbsp; &nbsp;&nbsp;&nbsp;<a href=\"https://www.pytorchlightning.ai\"><img src=\"https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/docs/source/_static/images/logo.svg\" width=\"150\"/></a>\n",
    "\n",
    "# Model Zoo -- DenseNet121 Trained on MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5445100c-532b-4bc6-b8ce-4dbcb014e70f",
   "metadata": {},
   "source": [
    "### Network Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7a7f81-98a8-418f-8983-8f99abfd9c42",
   "metadata": {},
   "source": [
    "The network in this notebook is an implementation of the DenseNet-121 [1] architecture on the MNIST digits dataset (http://yann.lecun.com/exdb/mnist/) to train a handwritten digit classifier.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d81a55-d071-4a6e-ae9e-6e22823a8ecf",
   "metadata": {},
   "source": [
    "The following figure illustrates the main concept of DenseNet: within each \"dense\" block, each layer is connected with each previous layer -- the feature maps are concatenated.\n",
    "\n",
    "\n",
    "![](../../pytorch_ipynb/images/densenet/densenet-fig-2.jpg)\n",
    "\n",
    "Note that this is somewhat related yet very different to ResNets. ResNets have skip connections approx. between every other layer (but don't connect all layers with each other). Also, ResNets skip connections work via addition\n",
    "\n",
    "$$\\mathbf{x}_{\\ell}=H_{\\ell}\\left(\\mathbf{X}_{\\ell-1}\\right)+\\mathbf{X}_{\\ell-1}$$,\n",
    "\n",
    "whereas $H_{\\ell}(\\cdot)$ can be a composite function  of operations such as Batch Normalization (BN), rectified linear units (ReLU), Pooling, or Convolution (Conv).\n",
    "\n",
    "In DenseNets, all the previous feature maps $\\mathbf{X}_{0}, \\dots, \\mathbf{X}_{\\ell}-1$ of a feature map $\\mathbf{X}_{\\ell}$ are concatenated:\n",
    "\n",
    "$$\\mathbf{x}_{\\ell}=H_{\\ell}\\left(\\left[\\mathbf{x}_{0}, \\mathbf{x}_{1}, \\ldots, \\mathbf{x}_{\\ell-1}\\right]\\right).$$\n",
    "\n",
    "Furthermore, in this particular notebook, we are considering the DenseNet-121, which is depicted below:\n",
    "\n",
    "\n",
    "\n",
    "![](../../pytorch_ipynb/images/densenet/densenet-tab-1-dnet121.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d036f2f-4246-495e-b874-449b4e811722",
   "metadata": {},
   "source": [
    "**References**\n",
    "    \n",
    "- [1] Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708), http://openaccess.thecvf.com/content_cvpr_2017/html/Huang_Densely_Connected_Convolutional_CVPR_2017_paper.html\n",
    "\n",
    "- [2] http://yann.lecun.com/exdb/mnist/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab5f0e1e",
   "metadata": {},
   "source": [
    "## General settings and hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca269eeb",
   "metadata": {},
   "source": [
    "- Here, we specify some general hyperparameter values and general settings\n",
    "- Note that for small datatsets, it is not necessary and better not to use multiple workers as it can sometimes cause issues with too many open files in PyTorch. So, if you have problems with the data loader later, try setting `NUM_WORKERS = 0` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "54fa873b",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 256\n",
    "NUM_EPOCHS = 20\n",
    "LEARNING_RATE = 0.005\n",
    "NUM_WORKERS = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eec6da61",
   "metadata": {},
   "source": [
    "## Implementing a Neural Network using PyTorch Lightning's `LightningModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1dca95d",
   "metadata": {},
   "source": [
    "- In this section, we set up the main model architecture using the `LightningModule` from PyTorch Lightning.\n",
    "- We start with defining our neural network  model in pure PyTorch, and then we use it in the `LightningModule` to get all the extra benefits that PyTorch Lightning provides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c073af3b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2:1: E266 too many leading '#' for block comment\n",
      "18:1: E303 too many blank lines (3)\n",
      "34:80: E501 line too long (80 > 79 characters)\n",
      "119:24: E225 missing whitespace around operator\n",
      "121:24: E225 missing whitespace around operator\n",
      "130:30: E261 at least two spaces before inline comment\n"
     ]
    }
   ],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "# The following code cell that implements the DenseNet-121 architecture \n",
    "# is a derivative of the code provided at \n",
    "# https://github.com/pytorch/vision/blob/master/torchvision/models/densenet.py\n",
    "\n",
    "import re\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.utils.checkpoint as cp\n",
    "from collections import OrderedDict\n",
    "\n",
    "\n",
    "\n",
    "def _bn_function_factory(norm, relu, conv):\n",
    "    def bn_function(*inputs):\n",
    "        concated_features = torch.cat(inputs, 1)\n",
    "        bottleneck_output = conv(relu(norm(concated_features)))\n",
    "        return bottleneck_output\n",
    "\n",
    "    return bn_function\n",
    "\n",
    "\n",
    "class _DenseLayer(nn.Sequential):\n",
    "    def __init__(self, num_input_features, growth_rate,\n",
    "                 bn_size, drop_rate, memory_efficient=False):\n",
    "        super(_DenseLayer, self).__init__()\n",
    "        self.add_module('norm1', nn.BatchNorm2d(num_input_features)),\n",
    "        self.add_module('relu1', nn.ReLU(inplace=True)),\n",
    "        self.add_module('conv1', nn.Conv2d(num_input_features, bn_size *\n",
    "                                           growth_rate, kernel_size=1, stride=1,\n",
    "                                           bias=False)),\n",
    "        self.add_module('norm2', nn.BatchNorm2d(bn_size * growth_rate)),\n",
    "        self.add_module('relu2', nn.ReLU(inplace=True)),\n",
    "        self.add_module('conv2', nn.Conv2d(bn_size * growth_rate, growth_rate,\n",
    "                                           kernel_size=3, stride=1, padding=1,\n",
    "                                           bias=False)),\n",
    "        self.drop_rate = drop_rate\n",
    "        self.memory_efficient = memory_efficient\n",
    "\n",
    "    def forward(self, *prev_features):\n",
    "        bn_function = _bn_function_factory(self.norm1, self.relu1, self.conv1)\n",
    "        if self.memory_efficient and any(\n",
    "                prev_feature.requires_grad for \n",
    "                prev_feature in prev_features):\n",
    "            bottleneck_output = cp.checkpoint(bn_function, *prev_features)\n",
    "        else:\n",
    "            bottleneck_output = bn_function(*prev_features)\n",
    "        new_features = self.conv2(self.relu2(self.norm2(bottleneck_output)))\n",
    "        if self.drop_rate > 0:\n",
    "            new_features = F.dropout(new_features, p=self.drop_rate,\n",
    "                                     training=self.training)\n",
    "        return new_features\n",
    "\n",
    "\n",
    "class _DenseBlock(nn.Module):\n",
    "    def __init__(self, num_layers, num_input_features, \n",
    "                 bn_size, growth_rate, drop_rate, memory_efficient=False):\n",
    "        super(_DenseBlock, self).__init__()\n",
    "        for i in range(num_layers):\n",
    "            layer = _DenseLayer(\n",
    "                num_input_features + i * growth_rate,\n",
    "                growth_rate=growth_rate,\n",
    "                bn_size=bn_size,\n",
    "                drop_rate=drop_rate,\n",
    "                memory_efficient=memory_efficient,\n",
    "            )\n",
    "            self.add_module('denselayer%d' % (i + 1), layer)\n",
    "\n",
    "    def forward(self, init_features):\n",
    "        features = [init_features]\n",
    "        for name, layer in self.named_children():\n",
    "            new_features = layer(*features)\n",
    "            features.append(new_features)\n",
    "        return torch.cat(features, 1)\n",
    "\n",
    "\n",
    "class _Transition(nn.Sequential):\n",
    "    def __init__(self, num_input_features, num_output_features):\n",
    "        super(_Transition, self).__init__()\n",
    "        self.add_module('norm', nn.BatchNorm2d(num_input_features))\n",
    "        self.add_module('relu', nn.ReLU(inplace=True))\n",
    "        self.add_module('conv', nn.Conv2d(\n",
    "            num_input_features, num_output_features,\n",
    "            kernel_size=1, stride=1, bias=False))\n",
    "        self.add_module('pool', nn.AvgPool2d(kernel_size=2, stride=2))\n",
    "\n",
    "\n",
    "class PyTorchDenseNet121(nn.Module):\n",
    "    r\"\"\"Densenet-BC model class, based on\n",
    "    `\"Densely Connected Convolutional Networks\"\n",
    "    <https://arxiv.org/pdf/1608.06993.pdf>`_\n",
    "\n",
    "    Args:\n",
    "        growth_rate (int) - how many filters to add each layer (`k` in paper)\n",
    "        block_config (list of 4 ints) - how many layers in each pooling block\n",
    "        num_init_featuremaps (int) - the number of filters to learn in the \n",
    "        first convolution layer bn_size (int) - multiplicative factor for \n",
    "        number of bottle neck layers (i.e. bn_size * k features \n",
    "        in the bottleneck layer) drop_rate (float) - dropout rate after \n",
    "        each dense layer num_classes (int) - number of classification classes\n",
    "        memory_efficient (bool) - If True, uses checkpointing. \n",
    "        Much more memory efficient, but slower. Default: *False*. \n",
    "        See `\"paper\" <https://arxiv.org/pdf/1707.06990.pdf>`_\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, growth_rate=32, block_config=(6, 12, 24, 16),\n",
    "                 num_init_featuremaps=64, bn_size=4, drop_rate=0, \n",
    "                 num_classes=1000, memory_efficient=False,\n",
    "                 grayscale=False):\n",
    "\n",
    "        super().__init__()\n",
    "\n",
    "        # First convolution\n",
    "        if grayscale:\n",
    "            in_channels=1\n",
    "        else:\n",
    "            in_channels=3\n",
    "        \n",
    "        self.features = nn.Sequential(OrderedDict([\n",
    "            ('conv0', nn.Conv2d(\n",
    "                in_channels=in_channels,\n",
    "                out_channels=num_init_featuremaps,\n",
    "                kernel_size=7,\n",
    "                stride=2,\n",
    "                padding=3,\n",
    "                bias=False)), # bias is redundant when using batchnorm\n",
    "            ('norm0', nn.BatchNorm2d(num_features=num_init_featuremaps)),\n",
    "            ('relu0', nn.ReLU(inplace=True)),\n",
    "            ('pool0', nn.MaxPool2d(kernel_size=3, stride=2, padding=1)),\n",
    "        ]))\n",
    "\n",
    "        # Each denseblock\n",
    "        num_features = num_init_featuremaps\n",
    "        for i, num_layers in enumerate(block_config):\n",
    "            block = _DenseBlock(\n",
    "                num_layers=num_layers,\n",
    "                num_input_features=num_features,\n",
    "                bn_size=bn_size,\n",
    "                growth_rate=growth_rate,\n",
    "                drop_rate=drop_rate,\n",
    "                memory_efficient=memory_efficient\n",
    "            )\n",
    "            self.features.add_module('denseblock%d' % (i + 1), block)\n",
    "            num_features = num_features + num_layers * growth_rate\n",
    "            if i != len(block_config) - 1:\n",
    "                trans = _Transition(num_input_features=num_features,\n",
    "                                    num_output_features=num_features // 2)\n",
    "                self.features.add_module('transition%d' % (i + 1), trans)\n",
    "                num_features = num_features // 2\n",
    "\n",
    "        # Final batch norm\n",
    "        self.features.add_module('norm5', nn.BatchNorm2d(num_features))\n",
    "\n",
    "        # Linear layer\n",
    "        self.classifier = nn.Linear(num_features, num_classes)\n",
    "\n",
    "        # Official init from torch repo.\n",
    "        for m in self.modules():\n",
    "            if isinstance(m, nn.Conv2d):\n",
    "                nn.init.kaiming_normal_(m.weight)\n",
    "            elif isinstance(m, nn.BatchNorm2d):\n",
    "                nn.init.constant_(m.weight, 1)\n",
    "                nn.init.constant_(m.bias, 0)\n",
    "            elif isinstance(m, nn.Linear):\n",
    "                nn.init.constant_(m.bias, 0)\n",
    "\n",
    "    def forward(self, x):\n",
    "        features = self.features(x)\n",
    "        out = F.relu(features, inplace=True)\n",
    "        out = F.adaptive_avg_pool2d(out, (1, 1))\n",
    "        out = torch.flatten(out, 1)\n",
    "        logits = self.classifier(out)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "040cf1e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pytorch_lightning as pl\n",
    "import torchmetrics\n",
    "\n",
    "\n",
    "# LightningModule that receives a PyTorch model as input\n",
    "class LightningModel(pl.LightningModule):\n",
    "    def __init__(self, model, learning_rate):\n",
    "        super().__init__()\n",
    "\n",
    "        self.learning_rate = learning_rate\n",
    "        # The inherited PyTorch module\n",
    "        self.model = model\n",
    "\n",
    "        # Save settings and hyperparameters to the log directory\n",
    "        # but skip the model parameters\n",
    "        self.save_hyperparameters(ignore=['model'])\n",
    "\n",
    "        # Set up attributes for computing the accuracy\n",
    "        self.train_acc = torchmetrics.Accuracy()\n",
    "        self.valid_acc = torchmetrics.Accuracy()\n",
    "        self.test_acc = torchmetrics.Accuracy()\n",
    "        \n",
    "    # Defining the forward method is only necessary \n",
    "    # if you want to use a Trainer's .predict() method (optional)\n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "        \n",
    "    # A common forward step to compute the loss and labels\n",
    "    # this is used for training, validation, and testing below\n",
    "    def _shared_step(self, batch):\n",
    "        features, true_labels = batch\n",
    "        logits = self(features)\n",
    "        loss = torch.nn.functional.cross_entropy(logits, true_labels)\n",
    "        predicted_labels = torch.argmax(logits, dim=1)\n",
    "\n",
    "        return loss, true_labels, predicted_labels\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"train_loss\", loss)\n",
    "        \n",
    "        # To account for Dropout behavior during evaluation\n",
    "        self.model.eval()\n",
    "        with torch.no_grad():\n",
    "            _, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.train_acc.update(predicted_labels, true_labels)\n",
    "        self.log(\"train_acc\", self.train_acc, on_epoch=True, on_step=False)\n",
    "        self.model.train()\n",
    "        return loss  # this is passed to the optimzer for training\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"valid_loss\", loss)\n",
    "        self.valid_acc(predicted_labels, true_labels)\n",
    "        self.log(\"valid_acc\", self.valid_acc,\n",
    "                 on_epoch=True, on_step=False, prog_bar=True)\n",
    "\n",
    "    def test_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.test_acc(predicted_labels, true_labels)\n",
    "        self.log(\"test_acc\", self.test_acc, on_epoch=True, on_step=False)\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n",
    "        return optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c0e755b",
   "metadata": {},
   "source": [
    "## Setting up the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7d20b72",
   "metadata": {},
   "source": [
    "- In this section, we are going to set up our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c55dbaed",
   "metadata": {},
   "source": [
    "### Inspecting the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dce74462",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "train_dataset = datasets.MNIST(root='./data', \n",
    "                               train=True, \n",
    "                               transform=transforms.ToTensor(),\n",
    "                               download=True)\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE, \n",
    "                          num_workers=NUM_WORKERS,\n",
    "                          drop_last=True,\n",
    "                          shuffle=True)\n",
    "\n",
    "test_dataset = datasets.MNIST(root='./data', \n",
    "                              train=False,\n",
    "                              transform=transforms.ToTensor())\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         num_workers=NUM_WORKERS,\n",
    "                         drop_last=False,\n",
    "                         shuffle=False)\n",
    "\n",
    "# Checking the dataset\n",
    "all_train_labels = []\n",
    "all_test_labels = []\n",
    "\n",
    "for images, labels in train_loader:  \n",
    "    all_train_labels.append(labels)\n",
    "all_train_labels = torch.cat(all_train_labels)\n",
    "    \n",
    "for images, labels in test_loader:  \n",
    "    all_test_labels.append(labels)\n",
    "all_test_labels = torch.cat(all_test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "552008d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "Training label distribution: tensor([5914, 6729, 5948, 6123, 5827, 5413, 5909, 6256, 5845, 5940])\n",
      "\n",
      "Test labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "Test label distribution: tensor([ 980, 1135, 1032, 1010,  982,  892,  958, 1028,  974, 1009])\n"
     ]
    }
   ],
   "source": [
    "print('Training labels:', torch.unique(all_train_labels))\n",
    "print('Training label distribution:', torch.bincount(all_train_labels))\n",
    "\n",
    "print('\\nTest labels:', torch.unique(all_test_labels))\n",
    "print('Test label distribution:', torch.bincount(all_test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ebda321",
   "metadata": {},
   "source": [
    "### Performance baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd70fdad",
   "metadata": {},
   "source": [
    "- Especially for imbalanced datasets, it's quite useful to compute a performance baseline.\n",
    "- In classification contexts, a useful baseline is to compute the accuracy for a scenario where the model always predicts the majority class -- you want your model to be better than that!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c9a9b881",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline ACC: 11.35%\n"
     ]
    }
   ],
   "source": [
    "majority_prediction = torch.argmax(torch.bincount(all_test_labels))\n",
    "baseline_acc = torch.mean((all_test_labels == majority_prediction).float())\n",
    "print(f'Baseline ACC: {baseline_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f760a26d",
   "metadata": {},
   "source": [
    "### Setting up a `DataModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35e59891",
   "metadata": {},
   "source": [
    "- There are three main ways we can prepare the dataset for Lightning. We can\n",
    "  1. make the dataset part of the model;\n",
    "  2. set up the data loaders as usual and feed them to the fit method of a Lightning Trainer -- the Trainer is introduced in the next subsection;\n",
    "  3. create a LightningDataModule.\n",
    "- Here, we are going to use approach 3, which is the most organized approach. The `LightningDataModule` consists of several self-explanatory methods as we can see below:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4584dd13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from torch.utils.data.dataset import random_split\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "class DataModule(pl.LightningDataModule):\n",
    "    def __init__(self, data_path='./'):\n",
    "        super().__init__()\n",
    "        self.data_path = data_path\n",
    "        \n",
    "    def prepare_data(self):\n",
    "        datasets.MNIST(root=self.data_path,\n",
    "                       download=True)\n",
    "        self.resize_transform = transforms.Compose(\n",
    "            [transforms.Resize((32, 32)),\n",
    "             transforms.ToTensor()])\n",
    "        \n",
    "        return\n",
    "\n",
    "    def setup(self, stage=None):\n",
    "        # Note transforms.ToTensor() scales input images\n",
    "        # to 0-1 range\n",
    "        train = datasets.MNIST(root=self.data_path, \n",
    "                               train=True, \n",
    "                               transform=self.resize_transform,\n",
    "                               download=False)\n",
    "\n",
    "        self.test = datasets.MNIST(root=self.data_path, \n",
    "                                   train=False, \n",
    "                                   transform=self.resize_transform,\n",
    "                                   download=False)\n",
    "\n",
    "        self.train, self.valid = random_split(train, lengths=[55000, 5000])\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        train_loader = DataLoader(dataset=self.train, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=True,\n",
    "                                  shuffle=True,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return train_loader\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        valid_loader = DataLoader(dataset=self.valid, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=False,\n",
    "                                  shuffle=False,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return valid_loader\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        test_loader = DataLoader(dataset=self.test, \n",
    "                                 batch_size=BATCH_SIZE, \n",
    "                                 drop_last=False,\n",
    "                                 shuffle=False,\n",
    "                                 num_workers=NUM_WORKERS)\n",
    "        return test_loader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "259e866c",
   "metadata": {},
   "source": [
    "- Note that the `prepare_data` method is usually used for steps that only need to be executed once, for example, downloading the dataset; the `setup` method defines the the dataset loading -- if you run your code in a distributed setting, this will be called on each node / GPU. \n",
    "- Next, lets initialize the `DataModule`; we use a random seed for reproducibility (so that the data set is shuffled the same way when we re-execute this code):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c94ae789",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(1) \n",
    "data_module = DataModule(data_path='./data')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfeec84f",
   "metadata": {},
   "source": [
    "## Training the model using the PyTorch Lightning Trainer class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "430baf56",
   "metadata": {},
   "source": [
    "- Next, we initialize our model.\n",
    "- Also, we define a call back so that we can obtain the model with the best validation set performance after training.\n",
    "- PyTorch Lightning offers [many advanced logging services](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) like Weights & Biases. Here, we will keep things simple and use the `CSVLogger`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b487c4ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_lightning.callbacks import ModelCheckpoint\n",
    "from pytorch_lightning.loggers import CSVLogger\n",
    "\n",
    "\n",
    "pytorch_model = PyTorchDenseNet121(grayscale=True)\n",
    "\n",
    "lightning_model = LightningModel(\n",
    "    pytorch_model, learning_rate=LEARNING_RATE)\n",
    "\n",
    "callbacks = [ModelCheckpoint(\n",
    "    save_top_k=1, mode='max', monitor=\"valid_acc\")]  # save top 1 model \n",
    "logger = CSVLogger(save_dir=\"logs/\", name=\"my-model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "219ad684",
   "metadata": {},
   "source": [
    "- Now it's time to train our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b52cc6f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name      | Type               | Params\n",
      "-------------------------------------------------\n",
      "0 | model     | PyTorchDenseNet121 | 8.0 M \n",
      "1 | train_acc | Accuracy           | 0     \n",
      "2 | valid_acc | Accuracy           | 0     \n",
      "3 | test_acc  | Accuracy           | 0     \n",
      "-------------------------------------------------\n",
      "8.0 M     Trainable params\n",
      "0         Non-trainable params\n",
      "8.0 M     Total params\n",
      "31.890    Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation sanity check: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1957d688968f4f6db4cc9d9bef6cf8c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training took 9.59 min in total.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "trainer = pl.Trainer(\n",
    "    max_epochs=NUM_EPOCHS,\n",
    "    callbacks=callbacks,\n",
    "    progress_bar_refresh_rate=50,  # recommended for notebooks\n",
    "    accelerator=\"auto\",  # Uses GPUs or TPUs if available\n",
    "    devices=\"auto\",  # Uses all available GPUs/TPUs if applicable\n",
    "    logger=logger,\n",
    "    log_every_n_steps=100)\n",
    "\n",
    "start_time = time.time()\n",
    "trainer.fit(model=lightning_model, datamodule=data_module)\n",
    "\n",
    "runtime = (time.time() - start_time)/60\n",
    "print(f\"Training took {runtime:.2f} min in total.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17f10432",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecab8c55",
   "metadata": {},
   "source": [
    "- After training, let's plot our training ACC and validation ACC using pandas, which, in turn, uses matplotlib for plotting (you may want to consider a [more advanced logger](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) that does that for you):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7a5c1cbf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Epoch', ylabel='ACC'>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABaU0lEQVR4nO2deXxU1d3/3yeTfd8TEiCBsAcQSNhdQJRFrahV61IVl1IUW7VPW/XXPq1t7VO7WTfEfd9wQ1GpgEgAkS3sO4QQIGFJyL5vc35/nAkOYZJMZubOTJLzfr3mNTP3nnvuJ3cm9zvnnO8ipJRoNBqNRtMaH08L0Gg0Go13og2ERqPRaGyiDYRGo9FobKINhEaj0Whsog2ERqPRaGzi62kBriQ2NlampqY6dGx1dTUhISGuFeRCtD7n0PqcQ+tzDm/Wt2XLljNSyjibO6WU3eaRkZEhHWXVqlUOH+sOtD7n0PqcQ+tzDm/WB2TLNu6peopJo9FoNDbRBkKj0Wg0NtEGQqPRaDQ2MXSRWggxE3gaMAGvSCmfaLVfWPZfAdQAc6SUWy37HgLuASSwC7hTSllnpF5N16OxsZH8/Hzq6tr/akRERLBv3z43qeo83qAvMDCQ3r174+fn51EdGu/BMAMhhDABC4DLgXxgsxBiiZRyr1WzWcBAy2M8sBAYL4RIBn4JDJNS1gohPgRuAt4wSq+ma5Kfn09YWBipqamo3xu2qaysJCwszI3KOoen9UkpKS4uJj8/n379+nlMh8a7MHKKaRyQI6XMlVI2AB8As1u1mQ28ZVlM3wBECiF6Wfb5AkFCCF8gGDhhoFZNF6Wuro6YmJh2jYOmY4QQxMTEdDgS0/QsjJxiSgaOW73PR40SOmqTLKXMFkL8CzgG1ALLpZTLbZ1ECDEXmAuQkJBAVlaWQ2KrqqocPtYdaH22iYiIoKqqqsN2zc3NVFZWukGRY3iLvrq6Opufo/7+OYe362sLIw2ErZ90rXOL22wjhIhCjS76AWXAR0KIn0op3zmvsZQvAS8BZGZmyilTpjgkNisrC0ePdZp9X0LcYIgd2GYTj+qzA0/p27dvn11TM56ewukIb9EXGBjI6NGjz9uuv3/O4e362sLIKaZ8oI/V+96cP03UVpvLgCNSyiIpZSPwKTDJQK2e48R2WPRTWPFHTyvRaDSaczDSQGwGBgoh+gkh/FGLzEtatVkC3C4UE4ByKeVJ1NTSBCFEsMXTaRrgvS4ojiIlfP0IICE3C5rqPa1I00nKysp4/vnnO33cFVdcQVlZWaePmzNnDh9//HGnj9NoHMEwAyGlbALuB5ahbu4fSin3CCHmCSHmWZotBXKBHOBl4D7LsRuBj4GtKBdXHyzTSN2KPYvh2HoYchU0VkPed55WpOkkbRmI5ubmdo9bunQpkZGRBqnSdIo1/4QP7/C0Cq/E0DgIKeVSlBGw3vaC1WsJzG/j2D8C3XfepbEWVvwBEkbAtS/CP9Pg0HIYMM3Tyrosf/piD3tPVNjc19zcjMlk6nSfw5LC+eOP0tvc/8gjj3D48GFGjRqFn58foaGh9OrVi+3bt7N3716uueYajh8/Tl1dHQ888ABz584FIDU1lezsbKqqqpg1axbjx49n8+bNJCcn8/nnnxMUFNShtpUrV/LrX/+apqYmxo4dy8KFCwkICOCRRx5hyZIl+Pr6Mn36dP71r3/x0Ucf8ac//QmTyURERARr1qzp9LXothz6Bgq2QHMjmHQMiDU6ktpTfP8slB+HmX+DgFDodzEcXKamnTRdhieeeIK0tDS2b9/OP//5TzZt2sRf//pX9u5V4T6vvfYaW7ZsITs7m2eeeYbi4uLz+jh06BA/+9nP2LNnD5GRkXzyyScdnreuro45c+awaNEidu3aRVNTEwsXLqSkpITFixezZ88edu7cye9//3sA/vznP7Ns2TJ27NjBkiWtZ3p7OKVHwNwIZw55WonX0a3SfXcZKk7Ad/+BoVdDv4vUtoHT1QiiOKddbyZN27T3S99dXkLjxo07J9DsmWeeYfHixQAcP36cQ4cOERMTc84x/fr1Y+TIkQBkZGSQl5fX4XkOHDhAv379GDRoEAB33HEHCxYs4P777ycwMJB77rmHK6+8kquuugqAyZMnM2fOHG688Uauu+46V/yp3YP6Kqg6rV6f3gMJwzyrx8vo8SOIpmYzaw4WcbzS7L6TfvMYmJth+l9+2DZohno+uMx9OjQuxzrnf1ZWFt988w3r169nx44djB492mYgWkBAwNnXJpOJpqamDs8j2xhp+vr6smnTJn784x/z2WefMXPmTABeeOEFHn/8cY4fP86oUaNsjmR6JKVHfnh9erfndHgpPd5AmCXc+84WVh5rdM8Jj2+GnYtg4nyISv1he2RfiBsKh7SB6EqEhYW1GeBWXl5OVFQUwcHB7N+/nw0bNrjsvEOGDCEvL4+cnBwA3n77bS655BKqqqooLy/niiuu4KmnnmL79u0AHD58mPHjx/PnP/+Z2NhYjh8/3k7vPYiSXPXsG6hGEJpz6PFTTP6+PkweEEt2biFSSmNTNpjN8PXDEJoIF/3q/P2DpsP6BVBXAYHhxunQuIyYmBgmT57M8OHDCQoKIiEh4ey+mTNn8sILLzBy5EgGDx7MhAkTXHbewMBAXn/9dW644Yazi9Tz5s2jpKSE2bNnU1dXh5SS//znPwD85je/4dChQ0gpmTZtGhdccIHLtHRpSiwjiLRpcGKbZ7V4IT3eQABMHRLP8r2nOXi6isGJBs5T7/pQeUtcsxACbJxn4AxY9zTkroJhrdNWabyV9957z+b2gIAA/vvf/9rc17LOEBsby+7du8+OQn7961+3e6433njj7Otp06axbdu5N7VevXqxadOm84779NNP2+23x1KSC8Gx0HcCHPgKakogONrTqryGHj/FBDBlsCrHmnWg0LiT1FeptYekMTDyJttt+oyHwAg4aDPtlEajcTUluRDdHxIsDg56mukctIEAekUE0TtUsMpIA7HuKag8CbP+Dj5tXHaTrxrqHlqupqM0PZb58+czatSocx6vv/66p2V1P0qOWAzEcPVeG4hz0FNMFkbG+bI8r5TKukbCAl0cLFN6FNY9AyNugD7j2m87aAbs+RRObofkMa7VoekyLFiwwNMSuj+NdVBRANH9IDReTTVpT6Zz0CMICyPjTDSZJetyzri+8xV/AOEDlz3WcdsBlwFCjSI0Go1xlB0FpBpBCKGmmfQI4hy0gbAwINKHsABfVu0vcm3Heetg72dw4YMQ0bvj9iGx0DtTx0NoNEbT4uIa3V89JwyHwn0qRkkDaANxFl8fwYUDY8k6WNhmEFKnMTerbK3hvWHSL+0/buAMOLEVqgxcE9FoejrnGYh0aKr9wfVVow2ENVMHx3O6op59J11U2WvbO3BqJ1z+J/APtv+4QdPV86EVrtGh0WjOp+QIBERAUJR6f9aTSa9DtKANhBWXtLi7HnTBL/e6Cvj2L9BnAgz/ceeOTRwJYb10VHU3JDQ0FIATJ05w/fXX22wzZcoUsrOz2+wjNTWVM2cMWCvraZTkqgXqluDYuCFqrVCvQ5xFGwgrEsIDGdYrnCxXrEOs+SdUn4FZT/zwBbQXIWDg5XB4FTQ1OK9F43UkJSXpwj+epiUGogW/QIgZqA2EFdrNtRVTh8TxwupcymsbiQhy0N21+DBsWAijboWk8+v72sXAGbD1LVVQqP8ljvXR0/jvI3Bql81dQc1NKs6ksySOUEa+DR5++GFSUlK47777AHjssccQQrBmzRpKS0tpbGzk8ccfZ/bscyPj8/LyuOqqq9i9eze1tbXMmTOHQ4cOMXToUGpra+2W9+STT/Laa68BcM899/Dggw9SXV3NjTfeSH5+Ps3Nzfzv//4vP/nJT2zWieixNDdC2bHzR/cJ6SrbgQYw2EAIIWYCTwMm4BUp5ROt9gvL/iuAGmCOlHKrEGIwsMiqaX/gD1LKp4zUC2odYsGqw3x36AxXjuzlWCfLfw++ATDtD44L6T8FTP7K3VUbCK/lpptu4sEHHzxrID788EO+/vprHnroIcLDwzlz5gwTJkzg6quvbjPP18KFCwkODmbnzp3s3LmTMWPsi3/ZsmULr7/+Ohs3bkRKyfjx47nkkkvIzc0lKSmJr776ClBJA1vqROzfvx8hhEPlTrsV5cdBNqspJmsS0lUcks6HBhhoIIQQJmABcDmQD2wWQiyRUu61ajYLGGh5jAcWAuOllAeAUVb9FACLjdJqzag+kYQH+rLqQKFjBuLwKjiwFKb9EcISOm7fFgGhkDJZubvO+Kvj/fQk2vmlX2tQPYjRo0dTWFjIiRMnKCoqIioqil69evHQQw+xZs0afHx8KCgo4PTp0yQmJtrsY82aNdxzzz0AjBw58mxtiI747rvvuPbaa8+mGL/uuutYu3YtM2fO5Ne//jUPP/wwV111FRdddBFNTU0260T0WFp7MLXQElFduA/6jnevJi/EyDWIcUCOlDJXStkAfAC0zkA3G3hLKjYAkUKI1nflacBhKeVRA7Wexdfkw8WD4lh9sAizuZPurs1N8PWjKo33hPucFzNoBhQf+uHLrPFKrr/+ej7++GMWLVrETTfdxLvvvktRURFbtmxh+/btJCQk2KwDYY0jWYTbcsceNGgQW7ZsYcSIETz66KP8+c9/brNORI+lxZX1PAOhPZmsMXKKKRmwTjqfjxoldNQmGThpte0m4P22TiKEmAvMBUhISCArK8shsVVVVWePTZSNFFU28NYX35IaYX8d46SCpQwq2sfu9Ec4s8753P9BNZGMBw4tXUBV5FSH/zZ3YH393ElERESb9RisaW5utqudI/zoRz/iF7/4BcXFxfz3v//l008/JTIykrq6OpYvX87Ro0epqqo6e/7Kykqqqqowm81UVlYyfvx4Fi1axMUXX8zevXvZuXMn1dXVbeqVUlJVVUVGRgb33nsv8+fPR0rJJ598wksvvcTBgweJiopi9uzZmEwm3n33XU6ePEltbS0XXXQR6enpjBo1ymb/dXV1Nj9HT32+9tJZfWk5q0nyCWBt9j4Q+3/YISUXmkI4vW0Fh6rTPKbPa5BSGvIAbkCtO7S8vw14tlWbr4ALrd6vBDKs3vsDZ4AEe86ZkZEhHWXVqlVnXxdW1MmUh7+Uz3xz0P4OqoulfCJFytevlNJsdljHeTwzRsq3rjlHnzfiKX179+61q11FRYWhOoYPHy6nTJkipZSyqKhITpgwQWZkZMi7775bDhkyRB45ckRKKWVISIiUUsojR47I9PR0KaWUNTU18rrrrpMjRoyQt912m5w4caLcvHlzm+dKSUmRRUVFUkop//3vf8v09HSZnp4u//Of/0gppfz666/liBEj5AUXXCAzMzPl5s2b5YkTJ+TYsWPliBEj5PDhw+Ubb7xhs++2rme3+/69+xMpF0y0ve/VmVK+Mt1pTdZ48/UDsmUb91QjRxD5QB+r972BE51sMwvYKqU8bYjCNogLC2Bk7wiyDhbxi2l21ode/XeoK4eZDri1tsfAGbD5ZUy95rquT43L2bXrB++p2NhY1q9fb7NdVVUVoGIZdu9W0xhBQUG88cYbdq+RWNes/tWvfsWvfnVu8akZM2YwY8aM846zVSeix1KS23bt94R02PEBSOna/+UuiJFrEJuBgUKIfkIIf9RU0ZJWbZYAtwvFBKBcSmk9vXQz7UwvGcmUQXFsO1ZKabUdcQhFB2DTy5AxBxKHu1bIoOnQ3EBk2U7X9qvR9FTMzaoWdev1hxYS0qGhUrnB9nAMMxBSyibgfmAZsA/4UEq5RwgxTwgxz9JsKZAL5AAvA2dXdoUQwSgPKI+UwpoyJB6zhDWH2gmaa25SXkuf3Qf+oTD1d64X0ncS+IcRU9x2ZK2mezJ+/PjzakJYj1Q0DlJxApob2jEQujZEC4bGQUgpl6KMgPW2F6xeS2B+G8fWADFG6muPC3pHEhXsx+oDRcwelfzDjuYmyFurMrTu+wJqisEvBK56UmVidTW+/pA2hZjD3+shbxtIo2uJe4iNGze69XzSVUkqvZ3SFg+mfrb3xw9Vz6f3wJAr3KPJS9GR1G1g8hE/uLs2NeJz9LvzjcLgmZB+rarh4BdknJhBMwnY94WKEu5ln498TyEwMJDi4mJiYmK6pZFwF1JKiouLCQwM9LQU42krBqKFgFCI6qddXdEGom2am7gxKofx9e9h/tdcfOpK3GsUrBlwuXo+tEwbiFb07t2b/Px8ioraz59VV1fn1Tc/b9AXGBhI79521Czp6pTkqiwF4cltt9HFgwBtIM6l1fTR5JpiRpkCOBx6CYNn3+5eo2BNWAIVYQMIP7gcLv6N+8/vxfj5+dGvXxtTBVZkZWUxerSDebHcgLfr61aU5KpgVp92YpwShquMCA01nUvV383QBsLcDEfWMOjAQth013nTR3euDKFeBPD50MkelVkSnUn40UVQXQwhHlua0Wi6PiV5agqpPRLSQZqhaB8kZ7hFljei0303N8Kin5JwerVKkPeTd+C3h+H612Doj5g8pA8788sorqr3qMzimExAQs43HtWh0XRppDw/zbctzqbc6NnTTNpA+AXCHUtYN/mts0bBehppyuA4ZEfurm6gMiwNQuJ1ESGNxhmqCqGxumMDEdUP/IK1gfC0AK8gOQOzKcDmrhHJEcSE+JN1wLMGAuGjigjlfKPWSjQaTefpyIOpBR8fiB+mDYSnBXg7Pj6CSwYrd9fmzmZ3dTUDp6t0Hvk6ZYJG4xBnDUTHjg3Kk2m3mpbqoWgDYQdTBsdTVtPI9uNlnhWSNhV8fFWNCI1G03lKj4AwQUSfjtsmDIfaUqg82XHbboo2EHZw8cBYfASsPlDoWSGBEdB3oqoyp9FoOk9JLkT2URkKOkIvVGsDYQ+Rwf6M6RvFKk+vQ4AqIlS4F8qOd9xWo9Gciz0eTC3o4kHaQNjLlMFx7Coop6jSs+6uDLSkcdbeTBpN5+mMgQiKVFNRegSh6Ygpg+MBWH3Qw6OI2IEqCvSgnmbSaDpFTYly8ugoSM6aHp5yQxsIO0lPCicuLIBVnl6HEEKNIo6sgcZaz2rRaLoSbdWhbo+EdDhzEJo8PHPgIbSBsBMhBFMGxbH2YBFNzWbPihk0HZpq4chaz+rQaLoS9sZAWJOQDuYmZSR6INpAdIKpQ+KpqGtim6fdXVMuVFGeeh1Co7GfklxAqClae+nhxYO0gegEkwfEYvIRrNrv4Wkmv0CVN+rg8u4RxJP9mo7t0BhP6REIT1L/P/YSnQamgB7ryWSogRBCzBRCHBBC5AghHrGxXwghnrHs3ymEGGO1L1II8bEQYr8QYp8QYqKRWu0hIsiPjJQoz6fdABVVXX4MivZ7WolzNNbC14/Ct497Wommu9MZD6YWTL4QP0SPIFyNEMIELABmAcOAm4UQw1o1mwUMtDzmAgut9j0NfC2lHAJcgKpr7XGmDo5n78kKTlfUeVbIwOnquav/8j76PTTVwamdKpGaRmMUJbn2pdhoTcJwbSAMYByQI6XMlVI2AB8As1u1mQ28JRUbgEghRC8hRDhwMfAqgJSyQUpZZqBWu5kyOA6A1Z4eRUQkQ8KIrh9Vffhbq9erPKdD072pr4Tqos6PIEAtVFedhiovmDlwM0YWDEoGrMN984HxdrRJBpqAIuB1IcQFwBbgASlldeuTCCHmokYfJCQkkJWV5ZDYqqoqu46VUhIVIPjwuz3EVx926FyOYEtfv4DB9D36KetWfEmTX6jbtNjC3uvXmrE7ltAQOYKQ6qOUfP8e+0sTXC8Ox/W5C63POTrSF1qZSyaw50QNRZ38OyJLmxgFbF/xHmVRFxiiz2uRUhryAG4AXrF6fxvwbKs2XwEXWr1fCWQAmSgjMd6y/WngLx2dMyMjQzrKqlWr7G778Mc75PA/fC0bmpodPl9nsanv6AYp/xgu5a6P3aajLTpz/c5Slq/0f/eUlB/dJeU/BkjZbMw1dUifG9H6nKNDfbsXq+/aiR2d77yqSB37/XOOSJNSevf1A7JlG/dUI6eY8gHrlIm9gRN2tskH8qWUGy3bPwbG4CVMGRxPZX0TW46WelZI70wIiu66UdUt00tp0yDtUqgu7LHeIhqD6Uya79aExEJoQo9chzDSQGwGBgoh+gkh/IGbgCWt2iwBbrd4M00AyqWUJ6WUp4DjQojBlnbTgL0Gau0UkwfE4GcSno+q9jHBgMsgZ4Wqrd3VOLwSQhPVHG/apT9s02hcTUmuqsgYEObY8S21IXoYhhkIKWUTcD+wDOWB9KGUco8QYp4QYp6l2VIgF8gBXgbus+riF8C7QoidwCjg/4zS2lnCAv3ITIn2/EI1qOyuNcVQsNXTSjqHuVktSqddqtKHhPeC+PRzF601GldRmufY6KGFhHQo3N/jqjkauUiNlHIpyghYb3vB6rUE5rdx7HbUWoRXMnVIHP+3dD8nympJigzq+ACjSLtUlSM9tAz6jPWcjs5yYhvUlcGAaT9sS5sKm16ChmrwD/GYNE03pCQX+l3s+PEJw6G5HkoOQ9zgjtt3E3QktYNMtWR39XjQXHA09B7b9VxEc1YCAvpP/WHbgGnQ3AB533lMlqYb0lgLFQWOubi20ENrQ2gD4SAD4kNJjgwiy9PrEKDSbpzYqsojdhUOr4SkURAS88O2vpPAN0hPM2lcS2meenbGQMQOUuV+e9hCtTYQDiKEYMrgONblnKGhycPZXftPAWnuOr+8a8sgP1t5L1njFwipky2jC43GRbSk+e5MHYjW+AYoI6ENhMZepgyOp7qhmey8Es8KSc4EvxDIzfKsDns5shpk87nrDy2kXQrFh6DsmPt1abonzri4WtMDiwdpA+EEk9Ji8Df5eN7d1dcfUi/sOgYiZyX4h6m1k9a0jCr0KELjKkpyITBSrdc5Q0I6lB9XI+AegjYQThAS4Mv4/tGs8vRCNahppuIcKDveYVOPIqVaY+h/CZj8zt8fNxjCk/U6hMZ1OJLF1RYttSEKvSYky3C0gXCSSwbFkVNYxfGSGs8K6T9FPXv7KOLMIfUrrCUwrjVCKHfX3NU9zudcYxClR5yfXgIrT6aeM82kDYSTTB1icXc96OFRRPxQFSnq7QaiJVLa1vpDC2nToL4cCra4R5Om+9LUoNazXDGCCOsFQVFwapfzfXURtIFwkv6xISRHBrH+8BnPChFCjSJys8DsYa+q9shZqap0tVf2sf8UFfynp5k0zlJ+XHn4ucJACNHjakNoA+EkQggyU6PIzittyUjrOdKmQs0Z750jbaxTrrjtjR5ALSYmjdF5mTTOc9aDyQUGAiwpN/Z6948wF6INhAvITI2msLKe4yW1nhXS7xL17K3TTMfWQ1Pt+fEPtki7VE0xdaXgP4334YoYCGsS0qGxRq1r9AC0gXABmSlRAGQf9XA8RESyCubJ9dK0G4dXgo+fcsntiAHT1NSAtxo7TdegJFfFCIXGu6a/HrZQrQ2ECxiUEEZYgC/Znq4PAWr+/uj30FTvaSXnk/Mt9J0AAXZUv0vOhIBwvQ6hcY4WF1chXNNf3FBAaAOhsR+Tj2BMSpTnI6pBGYjGGsjf7Gkl51JxEgr3dLz+0ILJV2XfzPlWxU5oNI5QkusaF9cW/IMhJq3HJO3TBsJFZKZEcfB0FeU1jZ4VknohCJP3Tc1YV4+zlwHToCIfzhw0RpOme2NuhrKjrjUQ0KNSbmgD4SIyU1UY/9ZjHp5mCoyA5AwvNBArVZxGSzSqPbQYEz3NpHGEigKVPt5VHkwtJAxXi9T1Va7t1wsx1EAIIWYKIQ4IIXKEEI/Y2C+EEM9Y9u8UQoyx2pcnhNglhNguhMg2UqcrGNUnEl8fwWZvmWYq2OI9OWOsq8f5dOIrF5UCMQN0XiaNY7jaxbWFloXqwn2u7dcLMcxACCFMwAJgFjAMuFkIMaxVs1nAQMtjLrCw1f6pUspRUkqvrSzXQpC/ifSkcO9ZqPam9N8nt0Ntif3rD9akXar+jsY6l8vSdHOMNhA9YB3CyBHEOCBHSpkrpWwAPgBmt2ozG3hLKjYAkUKIXgZqMpTM1Gh2HC/zfH2I3mO9K/13jmWKyLp6nL2kTVOxE8fWu1aTpvtTcgRMARCW5Np+I/qqbMQ9YB3CyJrUyYB1atF8YLwdbZKBk4AElgshJPCilPIlWycRQsxFjT5ISEggKyvLIbFVVVUOH9tCYFUT9U1m3vpyFQMiTU711ZrO6hsRNoSg3UvZFHKVS3W0RXv6Rm37FFNoGluyO/8PZWqSTBa+5Ge9Se5xx10VXfH5GonW5xy29KUf3ERwQByb16xx+flGB/ZGHlzH9pCsDtu2pa9LIKU05AHcALxi9f424NlWbb4CLrR6vxLIsLxOsjzHAzuAizs6Z0ZGhnSUVatWOXxsC6cramXKw1/Kl1Yfdrqv1nRa3/fPSfnHcCnLjrtciy3a1FdbJuVjUVKueMzxzl+/UsrnJzl+vHTN52skWp9z2NS3YKKU7/7EmBN+8ZCU/9dHSrPZrubefP2AbNnGPdXIKaZ8oI/V+97ACXvbSClbnguBxagpK68mPiyQlJhg71moBs9PMx1Z03b1OHtJu1TN91aecp0uTfdGSkuabxevP7SQkK4yDpfnG9O/l2CkgdgMDBRC9BNC+AM3AUtatVkC3G7xZpoAlEspTwohQoQQYQBCiBBgOtAlVoQyUqLYctQLEvfFD4OQOM8biJyV4B8KvZ2w7y3G5bCXphDReB9Vp1XAqKtjIFpocdfu5usQhhkIKWUTcD+wDNgHfCil3COEmCeEmGdpthTIBXKAl4H7LNsTgO+EEDuATcBXUsqvjdLqSsamRlNc3cCRM9WeFWKd/ttTxkpKFf/Q72JVFtVREkYoY6ezu2rsxVV1qNsifqh67uaeTEYuUiOlXIoyAtbbXrB6LYH5No7LBS4wUptR/JC4r5T+cXbkHDKS/lNh10cqPXGLa547KT6sirVM+qVz/fj4qL/l8EqVZrkzsRSanolRLq4tBIZDZIoeQWg6R1pcKJHBfmzJ84Z4CA+n/7anepy9DJgGNcVwaofzfWm6PyW54OOrXFKNogcUD9IGwsX4+Agy+kax2dOpvwEiekPMQM/N3eesVHn4XfErriWGQqfdcD1Zf4dvHvO0CtdScgQi+qikj0aRkA7Fh7p1EKc2EAaQkRpFblE1xVVekHK7/xQ4uk7V5nUnTfWQtxYGXOaa/sIS1FpEjjYQLqW8ANb8E757qnv9Gm5J820kCekqY0HRfmPP40G0gTCAsZbEfVu8Ie1G2lTPpP8+tkGd1xXTSy0MuBSOb4D6Stf12dPZuFDd5PxDIOsJT6txDVKqEYThBqL7ezJpA2EAI5Ij8Df5eIeBSL0QhI/71yHOVo+7yHV9pk0Dc5P35Jjq6tSVQ/YbkH4NTJwP+5bAyZ2eVuU8NSUqRsFoAxHdD3yDtIHQdI5APxMjekd4R+I+T6X/7kz1OHvpOwH8gnV2V1eR/To0VCovswn3QUAErP67p1U5T0u9aKNcXFvwMSl3127s6qoNhEFkpkSxK7+cusZmT0v5If13Xbl7zld5Gk7vUhHQrsQ3QI2IdDyE8zTVw4aF6ruRNAqCItUoYv+XcGKbh8U5idEurtYkpCsD4enAWIPQBsIgMlKiaGg2s6vATTfl9ug/RaW7cNfUTIunkSvXH1pIm6ZuACVHXN93T2LXR1B16twYlQnzIDCy669FlOQCQsUpGE3CcOV+XVVo/Lk8gDYQBpFhCZjzirxMvcepqRl3TTPlfKMinxNGuL7vs2k3tDeTw5jNsO4Z9flYj/ICI2DSL+Dg15C/xXP6nKUkV7l4+wUaf65uXhtCGwiDiAkNoH9ciHcEzPn6Q8pk9xgIsxlyHageZy8xA1TwkzYQjnNoOZw5AJN/qVKyWDP+5xAUDVl/84w2V1ByBKJS3XOuswaiey5UawNhIJkpUWw5VorZ7AXzk/2nwJmDyu/dSE5uV0PuNAOml0Dd0NKmQu5qaG405hzdnXVPqyCy9GvP3xcQpgxHzgo4vsn92lyBO2IgWgiOVgWJerKBsGRX9bG8HiSEuFoI4WestK5PZmo0ZTWNHC7yguLmLem/j6w29jwtC8iuXqC2ZsA05X2T7/Wlyr2P45vh2PdqQdrUxr/w2J9BcCys+j/3anMFdeVQc8Z9BgLUKCJ/s6q93s2wdwSxBggUQiSjivrcCbxhlKjugnXiPo/Tkv7b6LQbOd9C4kgIjTPuHP0uUbEd2pup83z/tFqIHn1b220CQmHyA2qq8GgXK/Xa4rzgTgNxwU1QchjWL3DfOd2EvQZCSClrgOtQVeGuBYYZJ6t70C82hJgQf+9YqPbxMT79d10F5G8yxnvJmqBISM7U8RCd5UwO7PsSxt7TcXzK2HsgJB6yutgowl0xENYM/zEMuQq+fRyKDrjvvG7AbgMhhJgI3IoqEwoGpwrvDgghzhYQ8gr6T4HqQijcZ0z/R9aoSGej1h+sGTBN+evXeIHx7SqsfxZM/mohuiP8g+HCh9RnemSt8dpcRUsMRJQbDYQQcNV/VLqSz+6F5ib3ndtg7DUQDwKPAostRX/6A7q8lx1kpkZxtLiGwkovyPjYz+D034ct1eP6jDemf2vSpgFSTYNoOqaqELa/D6NuhtB4+47JvBNCE5VHU1cJBCvJhdAE10bw20NoPFz5LxWQuv5Z957bQOwyEFLK1VLKq6WUf7csVp+RUjpZBaZnkNmSuM8b3F0j+yg3USMMhJRqyif1Iueqx9lL8hjlt98dsrtWnsavoczYc2x8EZobYOIv7D/GLwgu+hUcXUdkWRfJ0eSOJH1tkX4dDL1aLe4bNUp3M/Z6Mb0nhAi31IfeCxwQQvzGjuNmCiEOCCFyhBCP2NgvhBDPWPbvFEKMabXfJITYJoT40t4/yNsYnhRBgK+PdyxUg5pmyvvO5em/g2pPQtlR49cfWvAxqb/l8Mqu8+vWFtXF8OLFZGY/BBUnjDlHfRVsfgWGXgWxAzp37Jg7ICyJfkfe7xrXueSIe6eXrBECrnxSuQp3k6kme6eYhkkpK4BrUCVE+wLtuEGomzuwAJiFWtC+WQjRemF7FjDQ8pgLLGy1/wFUPesui7+vDxf0ifQiAzEVGquhwLUuotEllvw9Rrq3tiZtGlSe7Lr5+KWEJfdDbQmm5hr44BZorHX9eba9DXVlMOmBzh/rFwgX/w8RFfu8PzixoQYqT3huBAHKe+/Kf6v1sXVPeU6Hi7DXQPhZ4h6uAT6XUjYCHf2cGAfkSClzpZQNwAfA7FZtZgNvScUGIFII0QtACNEbuBJ4xU6NXktmShR7CsqpbfACP2mD0n9HlW5T0asxaS7tt11ajFFX9WbKfg0OLIXL/sS+ob+CE9vh8/mu/aXe3KjcL/tOgj5jHetj9G3UBVjiIrx5FFGap57d6cFki/Rr1SPriS4fQGevJ9KLQB6wA1gjhEgBKjo4Jhk4bvU+H2i9emmrTTJwEngK+C0Q1t5JhBBzUaMPEhISyMrK6kCWbaqqqhw+tiP8K5poMkve+CKLoTEmh/pwpb4xoQOQ2z5nm5jkkv6EuZHJpTspSLyUQwZdw7YYG9yb+s0fsbNheLvtjPx8HSG4+hgZWx6hLHoMu+qGUBVYQ3D/20jb/RZHqgI5mnqjS84Tf3o1w8qPs6vPHRQ78fdHJ85m5NFX2fnpvymJyXSJNldSVVXF7jUbGA5sOVJKZXGWR/X4RVzHWNO31L/9U7aO+SdVNXVe9f2zGymlQw/At4P9NwCvWL2/DRVDYd3mK+BCq/crgQzgKuB5y7YpwJf2aMrIyJCOsmrVKoeP7Yiy6gaZ8vCX8plvDjrch0v1ffNnKR+LkrK23DX95a6W8o/hUu770jX9dYalD0v5l3gpG2rabWbk59tpGmqlfH6SlH/vL2XlaSmlRZ/ZLOUnc9W13PO58+cxm6V8frKUz46VsrnZqa6yVi6X8j/DpXzxEtWvl7Fq1Sop1z2jrl1NiaflKPYuUXpWPeFd379WANmyjXuqvYvUEUKIJ4UQ2ZbHv4GQDg7LB/pYve8NtF6Fa6vNZOBqIUQeamrqUiHEO/Zo9UYigv0YlBDqResQU1yb/jtnJWZhcm31OHsZMA2a6uDo9+4/t6Os/JPK/nnNwnNdToWAHz0NvcfC4p/DyR3OnSd3larLMfmXTidOlD5+cPFv1dz6wa+d02UUJbkQFKUe3sDQH8GIG2DNPwitzPW0Goew91vzGlAJ3Gh5VACvd3DMZmCgEKKfEMIfuAlY0qrNEuB2izfTBKBcSnlSSvmolLK3lDLVcty3Usqf2qnVK8lMjWbrsVKavSFxXx8Xpv8+vRd2fUxF+BAIDHe+v86SMlkFf+3/quO23sChFbDheRg/DwZNP3+/XyD85F11k3v/FlV8yVHWPa3iGEbc4Hgf1lxwk1pn8ta1CHcm6bOXWf+AoGiG7H/G5Z6D7sBeA5EmpfyjVAvOuVLKPwHtfhJSyibgfmAZyhPpQ6mC7OYJIeZZmi0FcoEc4GXgPof+ii5AZkoUlXVNHDxd6WkpqjJbyiTnDITZDOufh5emQHM9R/o5br+/3n2KLUcdjIj2D1Y3wOxXVSCYN1NVqNwf49Phsj+13S4sAW5+H2pLYNGt0OhAkOWJ7erznXCv+rxdgckPLnkYTu30ToPsjQYiOBp+9BSh1Udg7b88rabT2GsgaoUQF7a8EUJMBjr0x5NSLpVSDpJSpkkp/2rZ9oKU8gXLaymlnG/ZP0JKeZ7vpZQyS0p5lZ06vZbMFBUw51XTTGcOOOZ7X14Ab18Dyx5VUzz3rqc80rHUXGaz5Dcf7+CxJXsdOh5QaQ76XaI8gPYvdbwfI5ESPrsP6ivhx690XMym1wVw7QsqS+gXD3T+F/v3z4J/mIqGdiUjboToNBVdbTa7tm8nEOZGKM/3XAxEewy5klMJU2DNv5Th7kLYayDmAQuEEHmWdYHnADsSumha6BMdRHxYANnekLgPfkj/ndvJ9N+7P4WFk9SN60dPw03vOZW59WBhJZV1TewqKKegzMEYAN8ApSNpFHw0xztzB218UdVYmP44JNhpTIfNhqm/g50fqOkieyk9CnsWQ+YcFW3uSky+ahRxejfsaz1j7DkC6wpBmr1vBGEhZ8DPVDblz+5V9cC7CPam2tghpbwAGAmMlFKOBtwYEdX1EUKQmRpFtjek3AA1zREca38uo7py+HQufHynStcx7zvImHN+RbJOstnqeizfc8rxjgJC4daPlQ/8+zdDwVandLmUU7thxf/CoFkqS2pnuPg3KoXDN4/ZPzra8LyKdRl/b6el2sWI6yFmoPLz95JRRFDtSfXCSw1Ek18oXP0MFO6F1f/wtBy76ZRrg5SyQqqIaoBfGaCnW5OZEk1BWS0nyw2Ilu0snUn/nbcOFk6GXR/DlEfhrmUuC4jbkldCXFgAgxJCWeaMgQA133vbYgiOgnd+7B2plxtr4ZO71aLz7Oc6b1CFgGueV6OjT3/WceBVTQlsfQtG3ggRyQ7LbhcfE0x5BIr2wd7FzvVVmgdb3oCdHzpVIdDbDQQAg2bAqFvhu/+opH5dAGd835z76dgDyUy1FBDyllFE/ylQdbrtVBVN9bDiD/DGlWqB8u7l6sZgcl2m9815pYxNjWJGeiKbjpRQUu2kp0d4Etz2Gfj4wtvXQtnxDg8xlOW/V9f32hcgJNaxPvyC1BSafyi8fxNUn2m77eZXoLEGJnUiKZ8jpF8LcUMso4hOZAioKYE9n8EXD8LTF6jHFw8o47dgHOxY5FBltqDaU+r6OHqN3cWM/1PZZj+7zzHnAzfjjIHwQj8372Zor3CC/EzeVR8CbHszFe6DV6apue+MO+Dna6G3ayNoT5bXUlBWS0ZKNDPSEzFL+GafE26dLcSkqZFEQ5VaTK8qcr5PR9i/VN2wJ97vfI6q8CS4+T3lCbXop7ZdJhtr1VrHwBkQP9S583VEyyjizEHY/Unb7Rrr1DrXN48pj7d/9IeP7lCj0fhhyg10/ia4+QNVT2HxXHh+olpD6cT0VVDtSTW96OSUp+EERcLVz6ofDVl/87SaDmnXQAghKoUQFTYelUCSmzR2G/xMPozuG+kdFeZApf+OTjvXQJjNsOEFePESqDgJN72vFqMNyK/fMpIamxpFelI4yZFBzq1DWJM4HG75SHlcvXMdpqZq1/RrLxUnlVdV4kiY9gfX9JmcAbMXwLH18NVD508Nbn9P1WOe7EBSPkcYOlutZWU98UPmUrNZBfite1qN4P6eCm9drbyqfAOVUblrOTx8RLnyjv85xA2GwbNg7hq48S11k/9oDrx4kXKntcODSxkIL55esmbgZark6/fPeH1d9XbnCqSU7eZB0nSezJQonluVQ1V9E6EBXlCUr/8U2LlIzf9WF6mhb+4q9St09nP2F5dxgOy8EoL9TQzrFY4QgunpCby78RjV9U2EuOLa9B0PP3kH3r+JEbWPwyWXqukaozGb4bN5KsL7+tdcF4cAaoG4aD+s+af6BT5xvuWczbD+OWVEUlyTY6tDfHxg6qNqRPPf30JtKRxZDTXFan/cEOXI0H8KpE5WabA76m/YbFW+c/en6hf2B7dA0mjlzTXgMtsjBHOz8mLqKgYCYMZfVX34z+6Fn69xz/fSAZyLv9d0mszUaMwSth8r87QURdpUNRXz7eNqaH98o4oruGWRocYB1PrDqD6R+JrU13BGeiINTWZWH3ThlNDAy+C6F4ko3wcf3uHUQqjdrH9Ojcpm/g1iB7q+/yn/T6VxWP57FZkNsP9LFSg2+QH3TrMMuUrFbGS/CkfXwYDL4ZoX4Ff7Yf5GmPUEDJ7ZsXGwxscEI29QU0+zFyiD8+718NoM227Z5fn4yCbvjIFoi8AImP2smqJb9VdPq2kTbSDczOi+kfgIvGeaqSX997qn1C+wn6+FzLsMv8lU1jWy/1TF2Yp7AGNTo4kO8Xfem6k1w3/MwUHz4NAyNUIy0jXzxHZY+Wd1Ax9zhzHn8PGBa19U0zsf3wWF+9WUTnR/dcN2J0LATxfDfRvhfw7AdS+qsqbhvZzv2+QLo38K929RP1rKjqvpqjeugqPrf2jXUoe6K40gQK1LZdwJ3z8HxzZ6Wo1NtIFwM2GBfgxODPeeheqgKLjof2Dq75WXUmcrjjnItmNlmKVaf2jB5CO4bGg83+4vpKHJtTfxk0kzYdofYdeH8PXDxuQSaqhWLq0hcfCjZ4w1sv4hag7fN1B5mRVsUYvhPo6lk3eKkBiIH2Lc3+vrr360/HIbzPy7cl9+fSa8fZ36u7uqgQCY/heI6KPii7zQ9VUbCA8wNjWKrcdKaWr2jiAjLv09XPIb5crqJrKPluIjYHTfczNvzkhPpLKuifW5xa4/6YUPwaRfwqaXjPEg+foRKD4M172kYjKMJrIP3PQu1FeooMdRtxh/Tk/iFwgT5sEDO+DyP6vMsi9fCll/o9nHH8JcMGpxNwFh8JO3AQGvzYRNL3tVIkRtIDxARkoUNQ3N7D/lBYn7PER2XglDe4Wft1A/eUAsIf4mvt7t4mkmUL9wL/+z8iBZ/XeVbNBV7P1cBahd+BD0c2Pa8z7j4I4vlZuoly50uhz/YLXW8uBO9eOmuYGq0H5OpzT3GEmjYN5atZi/9NdqNFHXUT0299BFr2jXZqxl3t1r8jK5mcZmM9uOlZ29DtYE+pmYMjieFXtPG5MavaXmwrDZKtmgKzLAlufDkl9C0hiY+v+c76+z9B3veDnRrkxAmEpF8tBedo78o6fVOEdwNNy8CC57DPYuUTEjp3Z5WpXdJUc1LiQpMoikiEA2Hy1lzuQu5HnhIvadrKC2sZmMFNuFXaanJ/DVrpNsO1Z6ziK2y/AxwXUvq19pn89XdSyGXHl+OymV62bVafWoPP3D66pCqDqlnssLAKmytLpxmk5jISCUZt+O6pd1AXx81Ai09zjlfPDKZXDFP9WI10MBgNpAeIiM1Gg2HSlGSonw9uhPF9OSoC8z1baBmDokHj+TYNmeU8YYCFCxCT95B96aDR/dqQK26ivUDb/ScuOvOg1mG26xvkGqZkNognJjTb0Qhv/YZfmpND2c1MlqyumTe2DJL1S1xCv/rRwT3Iw2EB5ibGoUX+w4QX5pLX2igz0tx61k55XQOyqIXhG258zDA/2YlBbLsj2n+X9XDDXOgAaEwq0fqYjf759VeXxCE1X8R9yQH4xAaLzl2bIvIMz7Uzpoujah8SpdzOp/qPWyE9tUlHncYLfKMNRACCFmAk8DJuAVKeUTrfYLy/4rgBpgjpRyqxAiEFgDBFg0fiyl7OKTjOfSMr2y5WhpjzIQUkqyj5YyOS2m3XYz0hP5f4t3sf9UJUN7GVjKNDga5mapSGQXJiHUaJzGx6Qi1fuOh09+Bi9NhR89pTL1ukuCUR0LIUzAAmAWMAy4WQjRulLKLGCg5TEXWGjZXg9caqlBMQqYaalZ3W0Ykqg8eLIdLbXZRTlWUkNRZX2HU0eXD0tACFwfNGcLIbRx0HgvaZeq+iu9LlBZb794wG2ZYI30YhoH5FhqWDcAHwCzW7WZDbxlKT26AYgUQvSyvK+ytPGzPLzHOdgFmHwEo/tGek/qbzex+WyCvvYNRFxYABl9o1i2xwXZXTWark54L7jjC5j8oKqf8eplKubGYIz82ZQMWCfjzwfG29EmGThpGYFsAQYAC6SUNmPRhRBzUaMPEhISyMrKckhsVVWVw8c6Sqxs4LtTjXy1YhUhfu3PaXtCX2ewV98Xu+sJ9oWCfdmc3N/+35wW2Miiow18tPRb4oKd+y3TXa6fp9D6nMNl+vymEjM8lCH7n0I8fyH7h/yCM3EGJmeUUhryAG5ArTu0vL8NeLZVm6+AC63erwQyWrWJBFYBwzs6Z0ZGhnSUVatWOXyso6w7VCRTHv5Srtp/usO2ntDXGezVN+3fWXLOaxvtanv0TLVMefhL+fKaw04oU3SX6+cptD7ncLm+0qNSvjRVyj+GS7n0YSkb6x3uCsiWbdxTjZxiygf6WL3vDZzobBspZRmQBcx0uUIPM6pvJCYf0WOmmUqrG8gprLLbdbVvTDBDEsPcsw6h0XQlIvvCnV/DhPtg40KVm6q+quPjOomRBmIzMFAI0U8I4Q/cBCxp1WYJcLtQTADKpZQnhRBxQohIACFEEHAZ0EZdzK5LsL8vw3qF95iF6pYEhZltBMjZYkZ6ItlHSymqrDdKlkbTNfH1Vynlb3wLkjMNKeplmIGQUjYB9wPLgH3Ah1LKPUKIeUKIeZZmS4FcIAd4GbjPsr0XsEoIsRNlaFZIKb80SqsnyUyNYvvxMhq9JXGfgWw+WoKfSXBBn0i7j5mRnoh0VSlSjaY7Mmw2XPEPQ7o21LdPSrkUZQSst71g9VoC820ctxMYbaQ2byEzJZrX1+Wx50QFozpx4+yKZOeVMiI5gkA/+1NSD+0VRt/oYJbtOcXN4/oaqE6j0bRGO397mJZ0E8+uPMTYftFEBPmd9wgP8sPsRSmAHaGusZld+eXMmZzaqeOEEMxIT+DN749SWddIWKDOdaTRuAttIDxMQnggFw2MZc2hIlbuL2yznQDC1yy3aTwig/24ZlQygxO9t4T4roJyGprNnVp/aGFGeiIvrz3CqgNFXH1BkgHqNBqNLbSB8ALevns8UkpqG5spr21Uj5rGH17XNrJjfw5R8UnnbDtRXktFbSOlNY1sOlLCJ/e6qVi9A7SUWG0rg2t7jOkbRWxoAMv2nNIGQnMeL64+zIY99UyZ4mkl3Q9tILwEIQTB/r4E+/vaTGKX1XyMKVOG2zz2lbW5PP7VPvacKCc9KcJoqQ6xJa+U/nEhxIQGdPpYHx/B5cMSWLK9gLrG5k6tYWi6N/VNzTyfdZiK2iZOlte2mQBS4xi6YFA34IaMPgT6+fD2+qOelmITs1kl6Bub4njq7hnpCVQ3NPP94TMuVKbp6qzaX0h5bSMS+HLHSU/L6XZoA9ENiAj2Y/YFyXy2vYDyGhv1CzxMTlEV5bWNbdZ/sIdJabGEBfiybLd2d9X8wKdbC4gNDSAl3IfPthd4Wk63QxuIbsJtE1OoazTz0ZbjHTd2M9lnCwQ5PoLw9/Vh6pB4Vuw7TVMPiBnRdExpdQOrDhRyzagkJiX5sudEBTmFPbfOuxFoA9FNGJ4cwZi+kbyz4ShmI2o5O0F2Xgmxof6kxjhX92JGeiIl1Q1kH+0ZqUk07fPlrpM0NkuuHZPM+EQTQsCS7a2z+WicQRuIbsTtE1PJK65hbY53zdNvPlpCZkq005XhpgyOw9/XR+dm0gDw6dZ8BieEMaxXOJGBPkxKi+HzHSdaknxqXIA2EN2IWSMSiQnx5+31eZ6WcpbTFXUcL6l1av2hhZAAXy4aEMvyPaf1TaCHc+RMNduOlXHdmOSzPzxmX5DM0eIaduSXe1hd90EbiG5EgK+Jm8b1YeX+Qo6X1HhaDuCa9QdrZqQnUlBWy54TFS7pT9M1WbytACFg9qjks9tmDE/E3+TD53qx2mVoA9HNuGV8CgJ4d+MxT0sBIPtoCYF+PqQnuaau9LSh8fi4qxSpxiuRUrJ4Wz6T02JJjAg8uz0iyI+pQ+L4YsdJmr1sHa6rog1ENyM5MojLhiawaPMx6hqbPS2H7LxSRvWJxM/kmq9aTGgAY1OjtYHowWQfLeV4SS3Xjk4+b9/sUcmcqapn/eFiDyjrfmgD0Q25fWIqpTWNfLXTs4FD1fVN7D1Z0WH96c4yIz2Rg6erOHKm2qX9aroGn24tIMjPxMzhieftu3RIPKEBvnqayUVoA9ENmTwghv5xIby1wbOR1duPl9Fslg7lX2qP6ekJgJ5m6onUNTbz1c4TzByeSEjA+ZmCAv1MzEhP5Ovdp7xiBN3V0QaiGyKE4LYJKew4XsbO/DKP6dicV4IQMMbFBqJ3VDDDk8O1geiBfLu/kIq6JpvTSy1cMzqJyvomsg60nR1ZYx/aQHRTfpzRm2B/E295MD9Tdl4pQxLDCTeghsOMYYlsO1bG6Yo6l/fdWXYXlLPnhHatdAefbi0gPiyAyQNi22wzsX8MsaEBfK6D5pzGUAMhhJgphDgghMgRQjxiY78QQjxj2b9TCDHGsr2PEGKVEGKfEGKPEOIBI3V2R8ID/bhmdDJf7DhBaXWD28/f1Gxm27FSh+o/2MMMy/zz8r2ezc205Wgp17/wPVc9+x2//2wX5bXelwuru1BS3UDWgUKuGZ2MyaftoEtfkw9XjezFyv2FVNTpz8MZDDMQQggTsACYBQwDbhZCDGvVbBYw0PKYCyy0bG8C/kdKORSYAMy3caymA26fmEJ9k5kPs92fn2n/qUqqG5pdEiBni4HxofSLDWG5B6eZcgqruPvNzSSEB3L7hBTe23iMy55czRIdzWsIX+w4QZNZtju91MLsUUk0NJn5ereehnQGI0cQ44AcKWWulLIB+ACY3arNbOAtqdgARAohekkpT0optwJIKSuBfUDH3wrNOQxJDGdcajTvbDzqdr/wlgJBrvZgakEIwfT0BNYfLvZIBtvTFXXc8domfH0Eb901jj/NHs6S+y+kV0Qgv3x/G7e/tok87WXlUj7dVsDQXuEM7dVxTM2oPpH0jQ7WuZmcRBj1S0cIcT0wU0p5j+X9bcB4KeX9Vm2+BJ6QUn5neb8SeFhKmW3VJhVYAwyXUp4XPiuEmIsafZCQkJDxwQcfOKS3qqqK0NBQh451B47q23iyiYU76nlwTACj4o2rD9Va3/Pb68gpM/PkFOcS9LVHTlkzj2+oY+7IACYltf+3ufLzrWmU/G1THUU1Zh4ZF0hqxA8FjMxS8u2xJj4+2ECThB/19+OK/n74tTMl4mp9RuBpfSerzDz6XS0/GezPrH7nr2nZ0vfJoQa+PNzIf6YEERno2eVWT1+/9pg6deoWKWWmzZ1SSkMewA3AK1bvbwOebdXmK+BCq/crgQyr96HAFuA6e86ZkZEhHWXVqlUOH+sOHNVX39gsMx9fIee8ttG1glphrc9sNstxf10hf/HeVkPP2dxslmMfXyF//lZ2h21d9fnWNTbJm15cL9Me/UquPlDYZrtT5bVy/rtbZMrDX8qp/1ol1+UUuUWfUXha37+W7Zf9HvlSni6vtbnflr5DpytkysNfylfX5hqsrmM8ff3aA8iWbdxTjTSr+UAfq/e9gdbjvTbbCCH8gE+Ad6WUnxqos1vj7+vDzeP6knWwiGPF7snPlF9ay+mKesYatP7Qgo+PmmZafbDILT7vZrPkfz7cwfrcYv55w0guHhTXZtuE8ECeu2UMb9w5lqZmyS0vb+RXH26nuKrecJ3dDbNZ8unWAiYPiCU+PLDjAywMiFeZXj/foaeZHMVIA7EZGCiE6CeE8AduApa0arMEuN3izTQBKJdSnhQqPeOrwD4p5ZMGauwR3DKuLz5C8M5G97i8Zh9V6w8ZTpQYtZcZ6YnUNjaz5mCRoeeRUvL4V/v4cudJHp01hGtH97bruCmD41n+0MXcP3UAX+w4waX/Xs0Hm455Xc0Ob2ZzXgkFZbX8eIx919ya2aOS2HG8TK8HOYhhBkJK2QTcDyxDLTJ/KKXcI4SYJ4SYZ2m2FMgFcoCXgfss2yejpqQuFUJstzyuMEprdycxIpAZ6Ql8mH3cLb+0N+eVEhbgy+DEMMPPNaF/DOGBvjz7bQ5Hi427Cby8NpfX1h3hzsmpzL24f6eODfQz8esZg1n6y4sYnBjGI5/u4sYX13PglK5+Zg+LtxUQ7G86G0HfGa4elaQKCelRhEMYunIjpVwqpRwkpUyTUv7Vsu0FKeULltdSSjnfsn+EtCxOSym/k1IKKeVIKeUoy2OpkVq7O7dNSKWsptEt/yjZeSWMSYlq11fdVfiZfPi/60aQd6aaGU+t4ZW1uS732Fq8LZ//W7qfK0f24n+vHOZw4aOBCWEsmjuBf1w/ksNFVVz5zFr+9t991DQ0uVRvd0Kl1jjJzOGJBPt33smiV0QQ41Kj+Wx7gXY9dgAdSd1DmNA/mkEJoby9/qih/yjlNY0cPF1lWICcLa4amcTyX13MpLRYHv9qH9e/8D2HTrvm1/naQ0X85qOdTOwfw5M3XoCPk0ZPCMGNmX1Y+T9TuHZ0Mi+uzuXyJ9ewv0TnDbLFN/tOU1nf5ND0UguzRyWTW1Sta4g4gDYQPYSW/Ey7CsrZfrzMsPNsOabWH1xVIMheekUE8eodmTx90yjyzlRz5TPf8ezKQzQ2mx3uc3dBOfPe3sKA+FBevD2DAF9TxwfZSXSIP/+84QIWzZ2Ar0nw4o56Gpoc19pdWby1gMTwQCb0j3G4j1nDE/EzCT3N5ADaQPQgrh3Tm9AAX942MD/T5rxSfH0Eo/pEGnaOthBCMHtUMit+dQnT0xP494qDXP3cOnYXdD5P0rHiGua8vonIYH/evGucIfmkAMb3j+Gxq9MprZd8uVPfwKw5U1XP6oNFzB6d5NR0ZVSIP5cMimPJ9hPaOaCTaAPRgwgN8OW6Mcl8ufOkYe6WW/JKSU+OIMjfdb+2O0tsaADP3TKGF2/LoLiqntkL1vHRgQa7F+iLq+q5/bWNNJklb941joROuFY6wpRBcSSFCl5ak6vnya1oSa1xnZ0eY+1x9ahkTlXUsckS4a+xD20gehi3TUihodnMIgPyM9U3NbM9v4yxblx/aI8Z6YmseOgSfjwmma+ONHLFM2vJ7uAGUV3fxF1vbOZURR2v3jGWAfHGR78KIZiV6sf+U5V8l3PG8PN1FRZvKyA9Kdwl3nCXDY0n2N+kCwl1Em0gehgDE8KY2D+Gdzccc7m3z+6CchqazG5ff2iPiGA//nH9Bfw6M5D6RjM3vLiex5bsobr+fM+hxmYz89/byq6Ccp69eYzLCx21x4QkX+LCAnhpTa7bzunN5BRWsTO/3K7EfPYQ7O/L9GEJLN11ivom7RBgL9pA9EBun5hCQVkt3+53bUGV7LxSALfeWO1leKyJ5Q9dzB0TU3lzfR4znlrD2kM/BNdJKXn0011kHSjir9eO4PJhnfe5dwY/H8GcSamsPXSGvdrbhsXb8vERKo7BVcwelUx5bSNrDupRmr1oA9EDuWxYAgnhAby1Ps+l/W7OK6VfbAhxYQEu7ddVhAT48tjV6Xz084n4+/pw26ub+O3HOyivbeRfyw/w8ZZ8HrxsIDeP6+sRfbeO70uwv4lX1vbsUYTZLPls2wkuHhRHfJjr1n8uHBhLdIi/nmbqBNpA9ED8TD7cMi6FtYfOkFtU5ZI+pZRsOVri1vgHR8lMjWbpLy/ivilpfLK1gIv/sYoFqw5z87i+PDBtoMd0RQb7c2NmH5bsOMHJ8lqP6fA0G4+o1Bquml5qwc/kw5UjevHNvtNU2Zhi1JyPNhA9lJvH9cHXR/DOhmMu6e9ktaS0ptGwAkGuJtDPxG9nDuHz+ZNJjQ3hRxck8ZfZ6Q5HSbuKuy/sh1lK3liX51EdnmTxtnxCA3yZPizR5X3PHpVEXaOZFXt1ISF70AaihxIfHsjM4Yl8tOW4S1I9HCpVC3/etEBtD8OTI/h8/mSevXk0vibP/zv0iQ7mihG9eG/jMSp7YLnM2oZmlu46xazhiYa4So/pG0VyZJCuV20nnv+P0HiM2yemUlnX5JJ/lkNlZqJD/OkfG+ICZT2buRf3p7K+iUWb3V8q1hEOna48W0HQWVZYpn+uHWNMAUkfH8HVo5JYe+iMTr1uB9pA9GDGpkYxJDGMt1yQn+lQaTMZKVEen6LpDozsHcn4ftG89t0Rp1KFuIO8M9Vc/8J6bnhhPbe8vMHpNC6Lt+aTFBHIhH6Op9boiNmjkmg2S5buOmnYOboL2kD0YIQQ3DYxhX0nK/j+cLHDRqKosp7TNdLwAkE9ibkX9+dEeZ1X38TKaxu5+83N+Aj49fRBHDhVyTUL1vHzt7MdSpZYVFnPmkNnmD062emkiO0xJDGcwQlheprJDowrUqzpElwzKpm//3c/t76ykbAAX1JjQ0iNDaFfTLDV6xCiQvzb7GPLUc8k6OvOTB0cT1pcCC+tyeXqC5K8bmTW1GzmF+9v42hxDe/cM54J/WOYM7kfr649wstrc1mxdw3XjenNg5cNpHeUfXXJl+w4QbNZcp2LvZdscfWoJP657ADHS2roE21c3fSujjYQPZyQAF8Wz5/MmoNFHDlTzZEz1Ww/XspXO09gHWgdEeRHv9gQ+sWGkBoTQmpssHodG8LmvFL8fGB4UoTn/pBuho+P4GcX9eeRT3ex/nAxkwbEelrSOfx16T7WHCzib9eNOJtpNTTAlwcuG8htE1NYmJXDm+uPsmT7CW6d0Jf5UwcQG9p+fMzibfmMSI5gYILxhaauvkAZiC92nuC+KQMMP19XxVADIYSYCTwNmIBXpJRPtNovLPuvAGqAOVLKrZZ9rwFXAYVSyuFG6uzppMWFkhZ3bs6h+qZmjpfUknemmrzianLPVJN3ppqNucUs3nZuoJGPgIGRPvj76hlLV3LN6GT+tfwAL63N9SoD8f6mY7y+Lo87J6faDCqMDvHnd1cO487J/Xhm5SHe/D6PDzcf5+6L+vOzi/oRZiMz7sHTlewuqOAPVw1zx59An+hgMlOiWLJdG4j2MMxACCFMwALgciAf2CyEWCKl3GvVbBYw0PIYDyy0PAO8ATwHvGWURk3bBPiaGBAfajNZXV1jM0eLazhiMR5Hi6tJNhtbE7onEuhn4o6Jqfx7xUEOnKp0SwnXjtiQW8z/frabiwfF8bsrhrbbNikyiCd+PJKfXdyfJ5cf5JmVh3h7fR7zpw7gpxNSCPT7wY31060FmCweRu5i9qgk/vfzPew/VcGQxHC3nbcrYeRPvnFAjpQyV0rZAHwAzG7VZjbwlqX06AYgUgjRC0BKuQbQuXm9kEA/E4MTw5g5PJF5l6Txt+tGMjzWc+m9uzPqRurjFek3jhXXcO87W0iJCea5W+yPG0mLC2XBrWP44v4LGZ4cweNf7WPqv7JYtPkYTc1mzGbJ59sLuGRQXIfTUK7kihG9MPkIvVjdDkYaiGTA2pE737Kts200mh5LVIhKv/HZ9gJOV9R5TEdtk+TuNzdjlvDKHWMdKqA0oncEb989nvd+Np6E8EAe/mQX059aw79XHOBkeZ3LU2t0RExoABcNjNWFhNrByDUIW24XrT8Fe9q0fxIh5gJzARISEsjKyurM4Wepqqpy+Fh3oPU5R1fWN9zPTFOz5M/vr+GGwW17kxmFWUqe21JNbpngfzIDObp7M87WJHxgmGRrbACfHKphwarDBPlCwJkDZGUddKg/Rz/fQQFNZJXV8+rn3zIwyrhRsLd//9pESmnIA5gILLN6/yjwaKs2LwI3W70/APSyep8K7Lb3nBkZGdJRVq1a5fCx7kDrc46uru/ed7LliD9+LSvrGt0jyIrHv9wjUx7+Ur69Ps/lfTc1m+Vn2/Llij2nnOrH0c+3sq5RDv79Uvm7xTudOn9HePP3D8iWbdxTjZxi2gwMFEL0E0L4AzcBS1q1WQLcLhQTgHIppfdGBmk0HuJnF/Wnoq6JD92cfuPD7OO8vPYI0/r68tMJKS7v3+Sj6ohf5ub6Gy2EBvgya3gvPszOPxvPo/kBwwyElLIJuB9YBuwDPpRS7hFCzBNCzLM0WwrkAjnAy8B9LccLId4H1gODhRD5Qoi7jdKq0Xg7o/tGMTY1ile/O0KTm9JvbDpSwu8W7+LCAbHcMsT9U1vu4g9XDSM5Moh73swm70y1p+V4FYY6rkspl0opB0kp06SUf7Vse0FK+YLltZRSzrfsHyGlzLY69mYpZS8ppZ+UsreU8lUjtWo03s7PLupPQVkt/91tfKrq4yU1zHtnC32igllwyxhMBqa+8DRRIf68PmcsAHe+sZnS6gYPK/IedGSTRtNFuGxoAv1iVfoN6WRyxfaoqm/injezaWo288odmUQEd95jqauRGhvCy7dnUlBWy9y3s6lr1HWrQRsIjabL4OMjuOeifuwqKGfjEWPmy5vNkgc/2EZOURULbh1D/7jzAyW7K5mp0Tx54wVszivltx/v1K6vaAOh0XQpfjymN9Eh/ry8xpjAuX8s2883+wr5w1XDuGhgnCHn8GauGpnEb2cOZsmOEzy5wjGX2+6ENhAaTRci0M/E7RNTWLm/kJzCzqfUbo9PtuTz4upcbh3fl9snut5jqatw7yVp3DS2D8+tynG715i3oQ2ERtPFuG1CCgG+Pryy9ojL+txytIRHP93FxP4xPHa152tzexIhBH+5ZjgXDYzl/y3exXeHznhaksfQBkKj6WLEhAZwfUZvPt1aQGGl8+k38ktr+PnbW+gVGcjzt47Bzwtqc3saP5MPC24dw4D4UO59ZwsHTrl2tNZV0N8EjaYLcs9F/Wk0m3l7vWNJL85U1bNo8zHufmMzl/57NfWNZl69I7PdwlA9jfBAP16bM5YgfxN3vbGZQg/mwvIUumCQRtMF6RcbwvRhCby94Sj3Tkkj2L/jf+UjZ6pZsfcUy/ecZsuxUqSE5Mggbh3fl5vG9mVAvOfTiXsbSZFBvDZnLDe+uJ6738xm0c8n2HWtuws95y/VaLoZcy/uz7I9p/koO587JqWet99sluwsKD9rFA4VVgEwrFc4D0wbyPRhiQztFdaj1xvsYXhyBM/ePJqfvZXNL9/fxou3ZXbrwEFrtIHQaLooGSnRjOkbySvf5fLTCSmYfAQNTWbW5xazYu8pVuw9zemKekw+gvH9orllfF8uH5Zgd41ozQ9MG5rAH3+Uzh+X7OEvX+7lsavTPS3JLWgDodF0YeZe3J9572zlif/u41RFPVn7C6msbyLIz8SUwXFcPiyBS4fEExms1xac5Y5JqRwrqeHV747QNzqYuy7s53YNdY3NFFXWU1hZT1Fl3dnXZin5zYwhLj+fNhAaTRfm8mGJpMYE8/LaI8SG+nPFiF5MT09g8oDYc0p6alzD/7tiKPmlNfzlq730jgpienqi031KKamobaKoqo7Cipabfz2FlXVWr+sprKijoq7pvON9BKTGhGgDodFozsXkI3jrrvEUVdUzqk9kj5kb9xQmH8FTPxnNTS+t54EPtrPo5xMY2TvS7uOllJwor2NXfjm7C8rZVaCei20kCAz08yE+LJC4sAAGxocyOS2GuLAAtS08gLjQAOLDA4gJCTDsc9cGQqPp4vSNCaZvjF5XcBdB/iZeuWMs1yxYx11vZPPZ/Ek213WklOSX1rK7oJyvDjbwWu4mdheUU2IxBiYfwcD4UC4dEs+ghDDiw9XNPz48gLiwAMICfD3uQKANhEaj0XSSuLAA3rhzLNct/J673tjMR/MmUVHbyC6rUcHugnJKaxoBMAkYlFjPZUPjGZEcwfDkCIb2Cvf6aUBtIDQajcYBBiaE8eJPM7j9tU1kPr6CxmaV/dXXRzAoIYzpwxIZ3juCEckRFB7cxvRpF3lYcefRBkKj0WgcZNKAWF6+PZOV+08zJDGcEckRDE4MO29kkHW4a64NGWoghBAzgacBE/CKlPKJVvuFZf8VQA0wR0q51Z5jNRqNxhuYOiSeqUPiPS3DEAzLxSSEMAELgFnAMOBmIcSwVs1mAQMtj7nAwk4cq9FoNBoDMTJZ3zggR0qZK6VsAD4AZrdqMxt4y1KbegMQKYToZeexGo1GozEQI6eYkgHrahv5wHg72iTbeSwAQoi5qNEHCQkJZGVlOSS2qqrK4WPdgdbnHFqfc2h9zuHt+trCSANha1WmdZHXttrYc6zaKOVLwEsAmZmZcsqUKZ2Q+ANZWVk4eqw70PqcQ+tzDq3PObxdX1sYaSDygT5W73sDJ+xs42/HsRqNRqMxECPXIDYDA4UQ/YQQ/sBNwJJWbZYAtwvFBKBcSnnSzmM1Go1GYyCGjSCklE1CiPuBZShX1deklHuEEPMs+18AlqJcXHNQbq53tnesUVo1Go1Gcz6GxkFIKZeijID1thesXktgvr3HajQajcZ9CHWP7h4IIYoAx4r0QixwxoVyXI3W5xxan3Nofc7hzfpSpJRxtnZ0KwPhDEKIbCllpqd1tIXW5xxan3Nofc7h7frawshFao1Go9F0YbSB0Gg0Go1NtIH4gZc8LaADtD7n0PqcQ+tzDm/XZxO9BqHRaDQam+gRhEaj0Whsog2ERqPRaGzSowyEEGKmEOKAECJHCPGIjf1CCPGMZf9OIcQYN+vrI4RYJYTYJ4TYI4R4wEabKUKIciHEdsvjD27WmCeE2GU5d7aN/R67hkKIwVbXZbsQokII8WCrNm69fkKI14QQhUKI3VbbooUQK4QQhyzPUW0c2+731UB9/xRC7Ld8fouFEJFtHNvud8FAfY8JIQqsPsMr2jjWU9dvkZW2PCHE9jaONfz6OY2Uskc8UCk7DgP9UckAdwDDWrW5AvgvKpvsBGCjmzX2AsZYXocBB21onAJ86cHrmAfEtrPfo9ew1ed9ChUE5LHrB1wMjAF2W237B/CI5fUjwN/b0N/u99VAfdMBX8vrv9vSZ893wUB9jwG/tuPz98j1a7X/38AfPHX9nH30pBGEMwWM3IKU8qS0lFyVUlYC+1C1MboSHr2GVkwDDkspHY2sdwlSyjVASavNs4E3La/fBK6xcahbimbZ0ielXC6lbLK83YDKpuwR2rh+9uCx69eCEEIANwLvu/q87qInGYi2ihN1to1bEEKkAqOBjTZ2TxRC7BBC/FcIke5eZUhguRBii1DFmlrjLdfwJtr+x/Tk9QNIkCprMZZnWwWNveU63oUaEdqio++CkdxvmQJ7rY0pOm+4fhcBp6WUh9rY78nrZxc9yUA4U8DIrQghQoFPgAellBWtdm9FTZtcADwLfOZmeZOllGNQ9cLnCyEubrXf49dQqBTxVwMf2djt6etnL95wHX8HNAHvttGko++CUSwE0oBRwEnUNE5rPH79gJtpf/TgqetnNz3JQDhTwMhtCCH8UMbhXSnlp633SykrpJRVltdLAT8hRKy79EkpT1ieC4HFqKG8NR6/hqh/uK1SytOtd3j6+lk43TLtZnkutNHGo9dRCHEHcBVwq7RMmLfGju+CIUgpT0spm6WUZuDlNs7r6evnC1wHLGqrjaeuX2foSQbCmQJGbsEyZ/kqsE9K+WQbbRIt7RBCjEN9hsVu0hcihAhreY1azNzdqplHr6GFNn+5efL6WbEEuMPy+g7gcxttPFY0SwgxE3gYuFpKWdNGG3u+C0bps17TuraN83q66NhlwH4pZb6tnZ68fp3C06vk7nygPGwOorwbfmfZNg+YZ3ktgAWW/buATDfruxA1DN4JbLc8rmil8X5gD8orYwMwyY36+lvOu8OiwRuvYTDqhh9htc1j1w9lqE4CjahftXcDMcBK4JDlOdrSNglY2t731U36clDz9y3fwRda62vru+AmfW9bvls7UTf9Xt50/Szb32j5zlm1dfv1c/ahU21oNBqNxiY9aYpJo9FoNJ1AGwiNRqPR2EQbCI1Go9HYRBsIjUaj0dhEGwiNRqPR2EQbCI2mEwghmsW5GWNdliVUCJFqnRVUo/E0vp4WoNF0MWqllKM8LUKjcQd6BKHRuABLbv+/CyE2WR4DLNtThBArLYnlVgoh+lq2J1hqLeywPCZZujIJIV4Wqh7IciFEkMf+KE2PRxsIjaZzBLWaYvqJ1b4KKeU44DngKcu251Dpz0eikt49Y9n+DLBaqqSBY1DRtAADgQVSynSgDPixoX+NRtMOOpJao+kEQogqKWWoje15wKVSylxLwsVTUsoYIcQZVCqIRsv2k1LKWCFEEdBbSllv1UcqsEJKOdDy/mHAT0r5uBv+NI3mPPQIQqNxHbKN1221sUW91etm9DqhxoNoA6HRuI6fWD2vt7z+HpVJFOBW4DvL65XAvQBCCJMQItxdIjUae9G/TjSazhHUqgj911LKFlfXACHERtQPr5st234JvCaE+A1QBNxp2f4A8JIQ4m7USOFeVFZQjcZr0GsQGo0LsKxBZEopz3hai0bjKvQUk0aj0WhsokcQGo1Go7GJHkFoNBqNxibaQGg0Go3GJtpAaDQajcYm2kBoNBqNxibaQGg0Go3GJv8fvwdHHPmZkDsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA3M0lEQVR4nO3deXxU9bn48c+TfV+AJCxBQEVZE3bcqlBai61WW7HiLq11qbZWW6+29l7b29Yut9dWe62WqrVSK1Jc6q9FbVVSWkUFBJFdFoEQSEIgzIRkJpnk+f1xTkIIM9kmk4mZ5/16zWtmzjLnyWE4z3zXI6qKMcYY01ZctAMwxhjTN1mCMMYYE5QlCGOMMUFZgjDGGBOUJQhjjDFBJUQ7gJ40aNAgHTlyZLf2PXr0KOnp6T0bUA+y+MJj8YXH4gtPX45vzZo1B1U1L+hKVY3IA3gCqAA2hFgvwEPAdmA9MKXVurnAVnfdPZ095tSpU7W7li9f3u19e4PFFx6LLzwWX3j6cnzAag1xTY1kFdOT7oU+lAuA0e7jRuARABGJBx52148DrhCRcRGM0xhjTBARSxCqugI41M4mFwNPuUnsbSBHRIYAM4DtqrpTVeuBxe62xhhjelE02yCGAXtbvS91lwVbPjPUh4jIjTglEAoKCigpKelWMDU1Nd3etzdYfOGx+MJj8YWnr8cXSjQThARZpu0sD0pVFwILAaZNm6azZs3qVjAlJSV0d9/eYPGFx+ILj8UXnr4eXyjRTBClwPBW7wuBMiApxHJjjDG9KJrjIF4CrhXHGcARVd0PrAJGi8goEUkC5rvbGmOM6UURK0GIyDPALGCQiJQC9wGJAKr6KLAM+CxOV9ZaYIG7LiAitwGvAvHAE6q6MVJxGmOMCS5iCUJVr+hgvQK3hli3DCeBGGPMx1Zjk3K0PsAhXxMflnup8Qeo8Qc46g9Q42+kxtfA0fpG6gNNJCXEkRgvJMbHua/jSIqPa/VenPcJrZc726ckxlOQldLj8ferkdTGmN6jqlR6/Wzc76Gqpp44ARGIE6efSZxIy3sBRMTdRlq2FXddZkoCw3PTyMtMRiRYP5Xe/bv8gSbnYu5zLuhe97nG30CNL4C31brW74/Wt1ruD1Bb33jsg0tWRCzmQRlJrP7ep3v8cy1BGGM61KTK9ooaNpYdYdN+D5vKPGze7+FgTX2PHic5IY7C3FSGD0hjeG4awwekus/OIzs1sVufW1sfoMLjp8Lrp9zjo8Lrp8Lrc5f5OHS0oeXiX+MP0NDY8Y3UEuKEjJQEMpKPPQakJzF8QBqZyQmkt1q+b/cOphSNP2F5RkoC6cnxJMXH0dCoNDQ2UR9ocp4bm2ho1OPeN792tlNnm0ATCfGRSaqWIIyJoobGJso9PqrqnP/0ifHRnz+ztj7AlgNeNpV5WpLBpn211L/6TwAS44XTCjKZfXo+44ZmMW5IFkOyU1EUVSeZNCmA89y8rPWztqxznj11Dew9XMveQ7XsOVTL3kN1rNl9GK8vcFxsWSkJxycPN3FsqWrkyLp9LRf8cve5wuunwuOnxh844e9Mio8jLzOZgqxkhuWkkJmS2XLRzkhOIDPl+At5ZnLiceuSE+I6XdopKdnDrOKh7W6TlCAkJcSRntypj+wVliCMiZDGJuVgjZ+y6jr2H/G1PO8/UkdZtfNc4fXTfNffb694mfzMZAZnpzI0O4XB2SkMzU51nnNSGJydSkFmMglhJBF/oBFPXQCvrwGPz32uC7DnUK2bDI6w6+BR9wLvXJDHDc1i1vAEzp8xnnFDsjg1P4OkhN5JZEdqjyUO57mOvYdr2Vbh5Y2tFdQHmo5tvGod4JRCCrJSyM9MZszgTM4dnUd+VjL5mSkUuM/5mcnkpCVGvTqrr7MEYUw3NTYppYdr2VFZw95DdZQdqWN/9bEEUO7xEWg6vqoiNTGeITnOhf/c0XkMyXGSwdatW8kcPIIDR5wk8mFFDSu2VXK0dR02ECeQl5nMkOxUhmSntDwnJ8bhqWvA6wvgcS/6njZJwONrOP6C2sawnFTGDc3iwqKhjBuaxfihWQzLSUVEnIFeUwsjch7bk52WSHZaNhOGZZ+wrqlJqazxs/dQLe+uWcunz5lBflYKWSkJduHvIZYgjOlAXX0jOw/WsKPyKNsrathRWcOOihp2Hjx63AU3KT6OwdkpDMlOYcaoAc4F3E0AQ7JTGZqTQnZq8F+tJbU7mTXrtOOWqSpef6Al6TilDx/7q+s44PGxrdzLP7dVHtcQmpwQR1ZqIpkpCWSlJJKdmkhhbipZKYlkpSQcty6z1fshWalkp3Wvfj9a4uKEgqwUCrJSqPkontEFmdEOqd+xBGGMq6rGf1wSaH7eV13XUg0UJzB8QBqn5GVw7ml5nJKXzil5GYwYmM7A9CTi4nrul6uIOBf2wYmcPjj4xU9V8fgCNDQ2ufXi8T12fNOHqcKRvbD3XShdDb4j8IVHevwwliBMzPEHGvmwvKalAfbw3k0cPVhKuS+BWpKp1RQaE1MZMmggU07K5bKpwzk1P4NT8tMZOTCdlMS+cxEWkW737OkX6muhajuJ9Z5oRxJZDXVQtg5KV0Hpu7B3FdQccNYlpMLwGdDUBHE92zZkCcL0Kq+vgdLDde6jtuU5IT6OYTmpDMtJZaj7PCwnlazU8OqTj9Q2sGm/h1c/auClJevYVOZhe0UNA5uquCh+JZclvMV42eVs3Lb3yGHgSALsTofEdEhKg6Rgr9Nh0Glw+gWQM7xtCCZcquA9AAe3OY+q7e7rD51f0cCMhEyYMQlyTopurD1BFar3uMlglVNKOPABNDU463NHwqhznaRQOA0KJkB8ZH4kWIIw0BgAbYKEpLA/qsYfcC78h1ongDpKq50eKEfqGo7bPjUxnmG5qTQ2Kf/YVH5CI2pGcoKbNFIYlnt88hiWm0p+ZgrxcYKqUnq47li3TPd5X3Vdy2eNytjN1dnv86lB/+QkzxoERYdOhok/gYJxzq/R+qPQcLTj17VVUL3XeV1fA75qePkuGFwEYy6EMZ91/uP2RmOp3+v8+6Wc2JAbMQ11TtXG7jedx9EqSM2BlJzOP7f9vjX44NBONwl86CSAg9vg4Hao9x7bLjEdBo2Gk86EQddC1lDkr3fBkmthwSuQ2PMjiiOqoQ7K1h5LBqWroKbcWZeYBkOnwFm3QaGbEDLyey00SxCxbtcKWHIdaCOMuwQmXgYjzu6wqFpbH2BjmYf391azvvQI63fVcWTF3zlce3wCSEmMozA3jcLcVCYPz6UwN7XlfWFuKgPSk1pKCE1NStXRevZV11FWXce+w3Xsq65reb92bzXVbT4/IU4YnJ2Cp87psQPONfnkQelMGZHLddMHczbvkbvxjww98h5U+SF3FJz3HzDxMmTQ6J45jwe3w9a/wZa/QclPoOR+59fs6Z9zksVJZ0F8D/x3a2pyLp7NF5LSVVCxGVAYcDIMmQRDJznPQ4qdi3EPiA/UwvbXYPdbzmPfGmisBwQGT4DcEU49+OGPYH811FU7CbU9iWnHEkZDHVTvdhJds6xCJxFMusIpoQ0a7TxnDjkh8W7ZXsqEjT+BV+6Gix7skb85IlSdv3PvqmPVRQc+gCZ3nEbuKDh5FhROdx4FE3rme9NNliBilSqsegxevhsGnupcVD5YCu/9AbKGwYRLoehLUDCBhiZl6wEv75dWs37vEd4vrWZbubelr/zQ7BQGJAlnjx1CoTuAqTkJDGyVADoSFyfkZSaTl5nMpOE5Qbc56g9QVl1HaaskUlZdR1pyAuOGOF0zTy9IJ63sbVj/OLz7EviPUJ+YDdMWwMQvwbApPf/LftCpMOh2OPt2qKmAba84yWL1E/DOI86F8LS5TrI4ZQ4kZ3Tuc31HnF/qzcmgdJWzDJwSw7BpMO5iiIs/Vke98flj++eOcv5th07uWtKoOwx73nZKBx+9yTll6+DfTSDxzmfNvNn5IXHSGaE/L1DvxOqrdhJGe8/xic73rTkRDDzVqbrrpIN5Z8A5d8C/f+n80p58Vaf3jaj6Wihby/A9z8EzC51/n6MVzrrENBg2Fc76xrGEkJEX3XjbsAQRiwL18PJ/wJrfw+jPwKWPQUoW1NfStGUZde89Q+rK3xD31kPsiT+JpfVn8nzgLEo1j5y0RIoLczh/XAFFhTkUDc8mPzPFvSHKxIiHnp6cwOiCzBO7NKrCgfWwfiH8+XnwlkFShlPdU3QZK/cI531yTsTjA5wqgCnXOg9/Dex4A7Yuc5LG+sUQn+z8ShzzOafdornKoKnJqVIpffdY75TKLbTcRyt/rFPKK5zu1D8PHB28pHf0IOxf5ySM/eugdA1sfOHY+uak0VLaKHaqGfe4pYOP3oTyDc5x45Ng2DT2nHQpI8670rn4dja5JSQ5F7zeuujN/p5TsvnbnTB4Igwp6p3jNlN1SlAtVUXvwoENoI2cAk4J75RPwvDpznnMHxfV0kFn9O3oTM87etCpq939Jk1nfZOdRXey7cOjfLCvjPWl1awvzcTr+zK5zOOSpFVcnriSO+Of5c74Z/ENmU7y5PnI+KmQPjDaf4kj4HdKQmv+AAe3QlwCnPpp+MyP4LQLnMZkQEtLohNfcgaM+7zzaAzAnpVOstjyV/jwVfh/AoXTKTraAG/vbFU6yHESwYQvOvXOw6Z2vo0hfRCc+inn0exoFexfGzppNGvuETP7uzDiLKeEkpjCrpISRpwyK7xzEWnxCXDpE/Dbc+HZq+Gmf0JqbuSPu+MNePd3bumg0lmWmO6UVM/5JhTO4M3dfs4+/+LIx9LDLEHEgKYmZe/hWsq2rmb8P28ipb6KB9K+xRP/nE79G/8GnLr8MUMyuah4KJPcksGpeZc70zoc3g0blpKy/s+w7FtOPe8pc5wqgdMv6FJVQI9RdS60r94Lh3fB8JnwuQdg/BcgbUDvx9MZ8Qkw6hPO4zP3Q/lGpxpq6zKS6qud2AtnOIlh4Kk922UxfWCIpLHOeUicU2U0ZFKPdFaImow8+NJT8PsL4Pmb4IrFPd718zjvL4YXvwaZg51z21y6yxt7XOmgYX9J5GKIIEsQ/YiqUnbEGWG77YCXbeU1bCv3sr2ihnMbV/JA4iN4SeOO5B/RlDeZBRMyOa3AeYwuyAjdvz93BHziW3DOnU7Vw/olsOE5eO5V55fS2AvJkQmg5/VOr53yTfDqd2BnCQw6Ha5+7vgL38eBuI27gyfArLtZHY17FqcPhFPnOI/+ZPh0mPsTWPZt+NcvnA4JkfDOb52q2lHnwvw/QXL/G8ltCeJjbnuFl2fe3cvaPYf5sLwGb6tZK/MzkxlTkM4jhf9g1v7HOJo3iYzL/8jjg7rZV1/EqdsdPBE+9QOnznr9Etj0IpN8z0L5EjjjVqeBOxK/QmsPwfL7nYbf5AyY+zOY/pWI9QE3H2PTb3DaAZbf71T19OQPCFX458+dnmpjLoRLH//4da3tJEsQH0MNjU38Y1M5i1buZuXOKpLi45h8Ug5fmDKspURwWkEGOQkN8OItsOkvUDSf9Ise7LkvclwcjDzHeVzwM7b++Yecfvh1ePFmeO37MOMGmPrlnmmraGxwksLy+8HvgWlfhlnf7TvtIKbvEYGLfuWUeJ+7AW5a0TOD6Jqa4NXvOj3Tiq+Ez/+6zzc0h6P//mX9ULnHx5/e2cMz7+6hwutnWE4q/zH3dL40bTiDMtoMA67eA3+4Eio2wvk/gjNvi1z1T2Iq+4eez+lX/Bh2vA4rH4Y3fgQrfgHFV8AZX4O80zr+nGB2vAGvfMfpzTPqXJj7UygY37Pxm/4pKR0u/yMsnNUzg+gaA/DS1+H9P8HMW5x2pEi2b/QBliD6OFVl5c4q/m+tj7V/f4MmVc47LY+fnDGCWafnEx9scrjdK51eHI31cOUSGN3ztyIMSuRYQ2jFZnj7N7DuT2532vOdRHHyrM4lqqodTgP0tpedqQUuf9rpFmrTOJuuGHgKXPIIPHtVeIPoGnyw9MvOYMjZ98K5d8XEd9ESRB/l8TXw/JpSFr29mx2VR0lPhK+cczJXzTyJEQPb6TW05g/wt285xekrFnf/l3u48sc6xe9P/pdTPbTqd7DoEsgfD2d+zRmxnRDk1lk+D6z4H3j7EWf9p77vJJZg2xrTGWMvDG8Qnd8Lz1wBH/0LLvg5zLwpMnH2QZYg+phNZR4Wvb2bv6zbR219I8XDc/jFZcVkVX/I+XPGht6xMeDUjb77W2cwzrwneqcPeEcy8mDW3c4I4w1Lneqnv9wKr/3AaUic/hWn335TI6x7Gl7/b2esxqSrYM5/Ot0HjQlXdwfR1R6CP14K+9+HLyyE4ssjG2cfYwmiD/AHGnllwwEWrdzN6t2HSU6I4+JJQ7n6jBEUFeYAUFKyPfQH1B6CP18Pu/7ptDV86gd9r+EsMQUmX+1c+HeWOImi5H741/9C0WWwf70zEnr4TKdabNiUaEds+pPuDKLzlMGiL8ChXTD/aWfMT4zpY1eR2LO76ijXPP4uew7VMnJgGt/73FjmTS0kJ60T3UQbA87UDcvvd0ZwXvybvjMHTSgicMps51G51WmneH8xpA10ugtOuDQm6nZNFHRlEF3VDnjqEmdOqqufcwY3xiBLEFG0vaKGqx57G3+giSeun8as0/I7d0cyVWcE7uv/7UwvMXSK88UvnBb5oHtS3ulOo+FnfuKMZbDxDCbSOjOI7sAHsOiLzgyr170U06VZSxBAy/0ke9GmMg/XPP4OIsKzN54Z8paSJ9j1L2ecwb7VzmRtX1oEYy/6eP/qdudLMqZXtDeIbs878KfLnIker/+r8yMmhlmCUIXHz+dUzYcJw5yphiNs3d5qrn38HdKTE3j6hpmcnNfx7JgZ3h2w6CFnnEHWMKeHUPGVfa+twZi+LtQguu2vweKrIWsoXPti/7g7XZgiOspDROaKyFYR2S4i9wRZnysiL4jIehF5V0QmtFp3h4hsFJENIvKMiERmLHt9DeSOZGjZK/B/05xGqS3LnF41EfDOziqufuwdctKSWHLTmR0nh6od8OcFTFtzJ5S95wx6+/oaZyppSw7GdE/zILqmRmcQ3fvPwp/mO/f1+PIrlhxcEUsQIhIPPAxcAIwDrhCRcW02+y6wTlWLgGuBB919hwHfAKap6gQgHpgfkUCTM+HS37HyzMedrnAVW2DxFfDgJPj3r5weQj1kxbZKrvv9uxRkJbPkpjMZPqCdqhXPfvjrHfDwDNj2CrtPugxufx/O+jokpvZYTMbErOZBdGVr4YUbnTa86//Wq7f07OsiWYKYAWxX1Z2qWg8sBtpOiD4OeB1AVbcAI0WkwF2XAKSKSAKQBpRFMFYaknLgvLvgm+udBt/cEfDaffDAWHjxVudLFIZ/bCrnhj+sZtSgDJ696UwGZ4coENUddtoYHpoM7z0FUxfAN9ax6+Sre/eew8bEgrEXOqXy4ivh6uft/1gbohFqoBWRecBcVb3BfX8NMFNVb2u1zf1AiqreKSIzgLfcbdaIyO3Aj4E64O+qGrT/pojcCNwIUFBQMHXx4sXdirempoaMjOOre9JrdjO0bBmDDywnvsnPkazT2Tfss1TmnY3Gdb7HzTv7A/x2vZ+RWXHcOTWFjKQTG5TjGv0M2/dXTtrzHImBo5Tnn8uuUVfiSx0SMr6+xOILj8UXHouv+2bPnr1GVYN3gVTViDyAy4DHWr2/Bvh1m22ygN8D64BFwCqgGMgF3gDygETgReDqjo45depU7a7ly5eHXll7WHXlb1QfnKx6X5bqz09Rfe2/VatLO/zcZ1ft0VH3/FUve+Qt9dTVqwYaVD37VfevV/3wNdV1z6iW/Fz1F6c7n/3Heapl73ctvj7A4guPxRcei6/7gNUa4poayVbOUqD1jQcKaVNNpKoeYAGAOHe23+U+PgPsUtVKd93zwFnAHyMYb2ipOXDGLTDjJti53Lm94L/+15nbZcznYMZXnQnljlZCTaVzU/KaCjZt30nKrp38NauWMQE/cQ9Vum0aQUptw2c6A8VGnt3Lf5wxxgQXyQSxChgtIqOAfTiNzFe23kBEcoBaddoobgBWqKpHRPYAZ4hIGk4V0xxgdQRj7Zy4uGN34Dr8Eax63Gkn2PxS0M2HayoDkweQlzeMuIzhkHEWpOc5j4x8SM93X+dZ3acxps+JWIJQ1YCI3Aa8itML6QlV3SgiN7vrHwXGAk+JSCOwCfiKu+4dEVkKvAcEgLXAwkjF2i25I+H8H8Ks7zijmhtqISMfTc/jibU1/PzNw3y6aAS/vHwScfH9e854Y0z/FNGO9Kq6DFjWZtmjrV6vBIKOTFPV+4D7Ihlfj0hKcyabw2nP+cnLW1j4ZjmXTT2Zn15aFPx+DcYY8zFgI616SFOTct9LG1n09m6uPXME379ofOfmVTLGmD7KEkQPaGxS7n5uPUvXlHLTeSdzz9wxyMd5biRjjMESRI/4zvNOcrjjU6fxjTmnWnIwxvQLliDC1NSkPP/ePi6fNpzbPxX5if6MMaa3WPeaMB2qrSfQpIwbmhXtUIwxpkdZgghTuccHQEFWcpQjMcaYnmUJIkwVHj8A+VmRmY3cGGOixRJEmI6VICxBGGP6F0sQYSp3SxB5GVbFZIzpXyxBhKnC62NAehJJCXYqjTH9i13VwlTu8ZOfaaUHY0z/YwkiTBVen7U/GGP6JUsQYSr3+KyLqzGmX7IEEYbGJqXS67cShDGmX7IEEYaqGj9NirVBGGP6JUsQYajw2iA5Y0z/ZQkiDDZIzhjTn1mCCEPzIDlrpDbG9EeWIMJQ7vEhAoNsFLUxph+yBBGGCq+PgelJJMbbaTTG9D92ZQtDhcdPfqa1Pxhj+idLEGEo99ogOWNM/2UJIgzlHhskZ4zpvyxBdFOgsYmDNX4bA2GM6bcsQXTTwZp6VK2LqzGm/7IE0U3Ng+SskdoY019FNEGIyFwR2Soi20XkniDrc0XkBRFZLyLvisiEVutyRGSpiGwRkc0icmYkY+2q5mk2rARhjOmvIpYgRCQeeBi4ABgHXCEi49ps9l1gnaoWAdcCD7Za9yDwiqqOAYqBzZGKtTtsmg1jTH8XyRLEDGC7qu5U1XpgMXBxm23GAa8DqOoWYKSIFIhIFnAu8Li7rl5VqyMYa5dVeHzECQxMT4p2KMYYExGiqpH5YJF5wFxVvcF9fw0wU1Vva7XN/UCKqt4pIjOAt4CZQCOwENiEU3pYA9yuqkeDHOdG4EaAgoKCqYsXL+5WvDU1NWRkZHR6+yc2+Flf2civZqd163hd1dX4epvFFx6LLzwWX/fNnj17japOC7pSVSPyAC4DHmv1/hrg1222yQJ+D6wDFgGrcBLCNCCAk1DAqW76YUfHnDp1qnbX8uXLu7T9dU+8o597aEW3j9dVXY2vt1l84bH4wmPxdR+wWkNcUxMil5coBYa3el8IlLXeQFU9wAIAERFgl/tIA0pV9R1306XACY3c0VTh8TMk29ofjDH9VyTbIFYBo0VklIgkAfOBl1pv4PZUaq7EvwFYoaoeVT0A7BWR0911c3Cqm/qMCq/PBskZY/q1iJUgVDUgIrcBrwLxwBOqulFEbnbXPwqMBZ4SkUacBPCVVh/xdeBpN4HsxC1p9AUNjU0crKm3Lq7GmH4tklVMqOoyYFmbZY+2er0SGB1i33U4bRF9TmXLGAgrQRhj+i8bSd0Nx8ZAWAnCGNN/WYLohuZbjdo0G8aY/swSRDdUet15mKwEYYzpxyxBdEO5x098nDAw3RKEMab/sgTRDeUeH3kZycTHSbRDMcaYiLEE0Q3lXr81UBtj+j1LEN1Q4fGRZw3Uxph+zhJEN1RYCcIYEwMsQXSRP9DIoaP1NkjOGNPvWYLookq7k5wxJkZYguiilkFyVoIwxvRzliC6qKJ5mg1rpDbG9HOWILqoeR4mG0VtjOnvLEF0UYXXT0KcMCDN7kVtjOnfLEF0UbnHT35mMnE2itoY089Zgugiu5OcMSZWWILoonKPz7q4GmNigiWILnKqmKwEYYzp/yxBdIGvoZEjdQ1WgjDGxARLEF3QPIra2iCMMbHAEkQXHLsXtSUIY0z/FzJBiMjPReTmIMvvEJGfRTasvql5mg2rYjLGxIL2ShAXAguDLH8Q+Fxkwunbym2aDWNMDGkvQaiqNgVZ2ATE5CixCq+fpPg4ctISox2KMcZEXHsJolZERrdd6C6ri1xIfZdzJ7lkRGIyPxpjYkxCO+v+C3hZRH4ErHGXTQO+A3wzwnH1SeVeGyRnjIkdIROEqr4sIpcAdwFfdxdvAC5V1Q96IbY+p9zjZ3R+RrTDMMaYXtFeL6YUoFxVr1PVqe7jOqDcXdchEZkrIltFZLuI3BNkfa6IvCAi60XkXRGZ0GZ9vIisFZG/dvUPiwRnmg1roDbGxIb22iAeAj4RZPmngV929MEiEg88DFwAjAOuEJFxbTb7LrBOVYuAa3F6SLV2O7C5o2P1hrr6Rry+AHmZVsVkjIkN7SWIc1T1+bYLVfVp4NxOfPYMYLuq7lTVemAxcHGbbcYBr7ufuwUYKSIFACJSiNOd9rFOHCviKrw2SM4YE1vaa6Rur6tOZ0ZgDwP2tnpfCsxss837wBeBf4vIDGAEUAiUA78C/gPIbO8gInIjcCNAQUEBJSUlnQjtRDU1Ne3uu/VQIwAHdm2lxLu9W8cIR0fxRZvFFx6LLzwWX2S0lyAqRGSGqr7beqF7Ia/sxGcHSzDa5v1PgQdFZB3wAbAWCIjIhUCFqq4RkVntHURVF+IO6Js2bZrOmtXu5iGVlJTQ3r7e98vg3bWc/4kZnFbQbs6KiI7iizaLLzwWX3gsvshoL0HcBSwRkSc5vpvrtcD8Tnx2KTC81ftCoKz1BqrqARYAiDO4YJf7mA98XkQ+C6QAWSLyR1W9uhPHjQgbRW2MiTUhq4rcksNMnJLA9cB17qrrcJJER1YBo0VklIgk4Vz0X2q9gYjkuOsAbgBWqKpHVb+jqoWqOtLd741oJgdwRlEnJ8SRldpeTjXGmP6j3audqpYD94nIZOAKnORwLvBcRx+sqgERuQ14FYgHnlDVjc0TAKrqo8BY4CkRaQQ2AV8J54+JpAqPj/wsG0VtjIkdIROEiJyG8+v9CqAKeBYQVZ3d2Q9X1WXAsjbLHm31eiVwwnQebbYvAUo6e8xIKff4rXrJGBNT2uuNtAWYA1ykqueo6q+Bxt4Jq+9xptmwBGGMiR3tJYhLgQPAchH5nYjMIUZncQWo8PjJt3mYjDExpL1G6hdU9XJgDE4Vzx1AgYg8IiLn91J8fUKNP0CNP2AlCGNMTOlwwJuqHlXVp1X1QpyuquuAE+ZV6s8q3C6u+TbNhjEmhnTpntSqekhVf6uqn4xUQH1Rhbf5VqNWgjDGxI4uJYhY1TJIztogjDExxBJEJ1R4nBJEvpUgjDExxBJEJ5R7fKQmxpOZbKOojTGxwxJEJ5R7/TaK2hgTcyxBdEKFx2ejqI0xMccSRCdUeG2QnDEm9liC6ICq2r2ojTExyRJEB2r8AWrrG62LqzEm5liC6EC5xwbJGWNikyWIDjRPs5Fn02wYY2KMJYgO2DQbxphYZQmiA8em2bAEYYyJLZYgOlDu8ZOeFE+GjaI2xsQYSxAdsDvJGWNilSWIDlR4fNZAbYyJSZYgOlDh9VsJwhgTkyxBtOPYKGorQRhjYo8liHZ4fAF8DU1WgjDGxCRLEO1ouRe1JQhjTAyyBNGOlmk2rJHaGBODLEG0o9xKEMaYGBbRBCEic0Vkq4hsF5F7gqzPFZEXRGS9iLwrIhPc5cNFZLmIbBaRjSJyeyTjDKV5mo18K0EYY2JQxBKEiMQDDwMXAOOAK0RkXJvNvgusU9Ui4FrgQXd5APiWqo4FzgBuDbJvxJV7fGQmJ5Buo6iNMTEokiWIGcB2Vd2pqvXAYuDiNtuMA14HUNUtwEgRKVDV/ar6nrvcC2wGhkUw1qAqvD67k5wxJmaJqkbmg0XmAXNV9Qb3/TXATFW9rdU29wMpqnqniMwA3nK3WdNqm5HACmCCqnqCHOdG4EaAgoKCqYsXL+5WvDU1NWRkZBy37Edv15EYB3fPSO3WZ/akYPH1JRZfeCy+8Fh83Td79uw1qjot6EpVjcgDuAx4rNX7a4Bft9kmC/g9sA5YBKwCilutzwDWAF/szDGnTp2q3bV8+fITlp3909f19mfe6/Zn9qRg8fUlFl94LL7wWHzdB6zWENfUSFaulwLDW70vBMpab6BOiWABgIgIsMt9ICKJwHPA06r6fATjDEpVbZoNY0xMi2QbxCpgtIiMEpEkYD7wUusNRCTHXQdwA7BCVT1usngc2KyqD0QwxpCO1DVQH2iyLq7GmJgVsRKEqgZE5DbgVSAeeEJVN4rIze76R4GxwFMi0ghsAr7i7n42TpXUByKyzl32XVVdFql42zp2L2prpDbGxKaI9t90L+jL2ix7tNXrlcDoIPv9G5BIxtYRu5OcMSbW2UjqEFoSRKYlCGNMbLIEEULLKGqrYjLGxChLECFUeHxkpSSQkhgf7VCMMSYqLEGEUO6xLq7GmNhmCSKEcq/PEoQxJqZZggihwuO39gdjTEyzBBGEM4raR771YDLGxDBLEEEcrm2goVFtkJwxJqZZggjCBskZY4wliKCOJQgrQRhjYpcliCAqPM23GrUShDEmdlmCCKK5BGG9mIwxscwSRBAVXj85aYkkJ9goamNM7LIEEUS5x2eT9BljYp4liCDKvTZIzhhjLEEEUeGxaTaMMcYSRBtNTc33orYShDEmtlmCaKPqaD2NTWpdXI0xMc8SRBsVXhskZ4wxYAniBC2D5KwNwhgT4yxBtGHzMBljjMMSRBvlbgkiL8OqmIwxsc0SRBvlXh8D0pNISrBTY4yJbXYVbKPC4yc/00oPxhhjCaKNCrsXtTHGAJYgTlDu8VkXV2OMARKiHUBf0tikVHr9VoIwpo9paGigtLQUn88XdH12djabN2/u5ag6ry/El5KSQmFhIYmJiZ3eJ6IJQkTmAg8C8cBjqvrTNutzgSeAUwAf8GVV3dCZfSOhqsZPk9oYCGP6mtLSUjIzMxk5ciQicsJ6r9dLZmZmFCLrnGjHp6pUVVVRWlrKqFGjOr1fxKqYRCQeeBi4ABgHXCEi49ps9l1gnaoWAdfiJITO7tvjylvuJGdVTMb0JT6fj4EDBwZNDqZjIsLAgQNDlsBCiWQbxAxgu6ruVNV6YDFwcZttxgGvA6jqFmCkiBR0ct8ed2yaDStBGNPXWHIIT3fOXySrmIYBe1u9LwVmttnmfeCLwL9FZAYwAijs5L4AiMiNwI0ABQUFlJSUdCvYmpoaVu9dD8DOje9RvaNvtd/X1NR0+2/rDRZfeCy+9mVnZ+P1ekOub2xsbHd9tPWV+Hw+X5f+HSOZIIKlK23z/qfAgyKyDvgAWAsEOrmvs1B1IbAQYNq0aTpr1qxuBVtSUkJ28lBk04dc9OlZJMb3rQRRUlJCd/+23mDxhcfia9/mzZvbrcOPdh1/R/pKfCkpKUyePLnT20fyKlgKDG/1vhAoa72BqnpUdYGqTsJpg8gDdnVm30io8PoYmJ7c55KDMSa6qqur+c1vftPl/T772c9SXV3d8wH1kkiWIFYBo0VkFLAPmA9c2XoDEckBat12hhuAFarqEZEO942EchtFbUyf94P/t5FNZZ7jljU2NhIfH9/tzxw3NIv7Lhofcn1zgvja177WpeMuW7YMoE9UL3VHxH4qq2oAuA14FdgMLFHVjSJys4jc7G42FtgoIltweizd3t6+kYq1mTOK2hKEMeZ499xzDzt27GDSpElMnz6d2bNnc+WVVzJx4kQALrnkEqZOncr48eNZuHBhy34jR47k4MGD7N69m7Fjx/LVr36V8ePHc/7551NXVxfyeL/73e+YPn06xcXFXHrppdTW1gJQXl7OF77wBYqLiykuLuatt94C4KmnnqKoqIji4mKuueaanvvDVbXfPKZOnardtXz5cp32o3/o3Uvf7/ZnRNLy5cujHUK7LL7wWHzt27RpU7vrPR5PRI+/a9cuHT9+vKo65yItLU137tzZsr6qqkpVVWtra3X8+PF68OBBVVUdMWKEVlZW6gcffKDx8fG6du1aVVW97LLLdNGiRSGP17y/quq9996rDz30kKqqfulLX9Jf/vKXqqoaCAS0urpaN2zYoKeddppWVlYeF0swwc4jsFpDXFNtJLWrsUk5WOO3QXLGmA7NmDHjuAFnDz30EC+88AIAe/fu5cMPP2TgwIHH7TNq1CgmTZoEwNSpU/noo49Cfv6GDRv43ve+R3V1NTU1NXzmM58B4I033uCpp54CID4+nuzsbJ566inmzZvHoEGDABgwYEBP/ZmWIJp56hVVu9WoMaZj6enpLa9LSkp47bXXWLlyJWlpacyaNSvogLTk5GPXlvj4+HarmK6//npefPFFiouLefLJJ9vtmqqqERsjYt11XIf9Ti/agkwrQRhjjpeZmRmyofnIkSPk5uaSlpbGli1bePvtt8M+ntfrZciQITQ0NPD000+3LJ8zZw6PPPII4DSQezwe5syZw5IlS6iqqgLg0KFDYR+/mSUI1xE3QeRbCcIY08bAgQM5++yzmTBhAnfddddx6+bOnUsgEKCoqIj//M//5Iwzzgj7eD/84Q+ZOXMmn/70pxkzZkzL8gcffJDly5czceJEpk6dysaNGxk/fjz33nsv5513HsXFxdx5551hH7+ZVTG5DvvcEoS1QRhjgvjTn/4UdHlycjIvv/xy0HXN7QzJycls2LChZfm3v/3tdo91yy23cMstt5ywvKCggL/85S8nLL/uuuu47rrr2v3M7rAShKvar8QJDExPinYoxhjTJ1gJwlXtVwZlJJNgo6iNMb3k1ltv5c033zxu2e23386CBQuiFNHxLEG4qn1q1UvGmF718MMPRzuEdtnPZddhv9o0G8YY04olCNcRf5MNkjPGmFYsQQANjU146m2QnDHGtGYJAqj0OrcatTYIY4w5xhIEUO5pvtWolSCMMeHLyMgAoKysjHnz5gXdZtasWaxevbo3w+oy68WEcx8IgHybZsOYvu/le+DAB8ctSm0MQHwYl7PBE+GCn4YZ2ImGDh3K0qVL7X4QH2eVXqcEYdNsGGOCufvuu4+7o9z3v/99fvCDHzBnzhymTJnCxIkTg45w/uijj5gwYQIAdXV1zJ8/n6KiIi6//PJ2J+sDZzT1tGnTGD9+PPfdd1/L8lWrVnHWWWdRXFzMjBkz8Hq9NDY28u1vf5uJEydSVFTEr3/96x75u60EgVOCcEZRW4Iwps8L8ku/LsL3fJ4/fz7f/OY3W+4ot2TJEl555RXuuOMOsrKyOHjwIGeccQaf//znQ86s+sgjj5CWlsb69etZv349U6ZMafeYP/7xjxkwYACNjY3MmTOH9evXM2bMGC6//HKeffZZpk+fjsfjITU1lYULF7Jr1y7Wrl1LQkJCj03YZwkCpw0iO0mIj4vMlLnGmI+3yZMnU1FRQVlZGZWVleTm5jJkyBDuuOMOVqxYQVxcHPv27aO8vJzBgwcH/YwVK1bwjW98A4CioiKKioraPeaSJUtYuHAhgUCA/fv3s2nTJkSEIUOGMH36dACysrIAeO2117j55ptJSHAu6T11TwhLEEC5109OiiUHY0xo8+bNY+nSpRw4cID58+fz9NNPU1lZyZo1a0hMTGTkyJFB7wPRWmfv27Br1y5+8YtfsGrVKnJzc7n++uvx+Xwh7/0QqXtCWBsEUOHxkZNsCcIYE9r8+fNZvHgxS5cuZd68eRw5coT8/HwSExNZvnw5u3fvbnf/c889t+XeDhs2bGD9+vUht/V4PKSnp5OdnU15eXnLbLFjxoyhrKyMVatWAc59IwKBAOeffz6PPvoogUAA6Ll7QlgJAqeKadJASxDGmNDGjx+P1+tl2LBhDBkyhKuuuoqLLrqIadOmMWnSpOPu2xDMLbfcwoIFCygqKmLSpEnMmDEj5LbFxcVMnjyZ8ePHc/LJJ3P22WcDkJSUxLPPPsvXv/516urqSE1N5bXXXuOGG25g27ZtFBUVkZiYyFe/+lVuu+22sP/mmE8Qqsrs0/MZ1Hgw2qEYY/q4Dz441r120KBBrFy5Muh2NTU1AIwcOZINGzbg9XpJTU1l8eLFnT7Wk08+GXT59OnTg9617oEHHuCBBx7o9Od3RsxXMYkID1w+ibOGxnyuNMaY49hV0RhjomjmzJn4/f7jli1atIiJEydGKaJjLEEYYz4WItVTJ9reeeedXjmOqnZ5n5ivYjLG9H0pKSlUVVV16yJnnORQVVVFSkrXphOyEoQxps8rLCyktLSUysrKoOt9Pl+XL369qS/El5KSQmFhYZf2iWiCEJG5wINAPPCYqv60zfps4I/ASW4sv1DV37vr7gBuABT4AFigqu2PQjHG9EuJiYmMGjUq5PqSkhImT57cixF1TV+PL5SIVTGJSDzwMHABMA64QkTGtdnsVmCTqhYDs4D/FZEkERkGfAOYpqoTcBLM/EjFaowx5kSRbIOYAWxX1Z2qWg8sBi5us40CmeK0PGUAh4CAuy4BSBWRBCANKItgrMYYY9qQSDX6iMg8YK6q3uC+vwaYqaq3tdomE3gJGANkAper6t/cdbcDPwbqgL+r6lUhjnMjcCNAQUHB1K4MRGmtpqam5SYffZHFFx6LLzwWX3j6cnyzZ89eo6rTgq2LZBtEsP5obbPRZ4B1wCeBU4B/iMi/cKqULgZGAdXAn0XkalX94wkfqLoQWAggIpWzZ89uf0KU0AYBfXk4tcUXHosvPBZfePpyfCNCrYhkgigFhrd6X8iJ1UQLgJ+qU4zZLiK7cEoTI4BdqloJICLPA2fhNGiHpKp53Q1WRFaHyqJ9gcUXHosvPBZfePp6fKFEsg1iFTBaREaJSBJOI/NLbbbZA8wBEJEC4HRgp7v8DBFJc9sn5gCbIxirMcaYNiJWglDVgIjcBryKU2X0hKpuFJGb3fWPAj8EnhSRD3CqpO5W1YPAQRFZCryH02i9FrcayRhjTO+I6DgIVV0GLGuz7NFWr8uA80Psex9wX7B1EdLXE5DFFx6LLzwWX3j6enxBRawXkzHGmI83m4vJGGNMUJYgjDHGBBVTCUJE5orIVhHZLiL3BFkvIvKQu369iEzp5fiGi8hyEdksIhvdwYJtt5klIkdEZJ37+K9ejvEjEfnAPfbqIOujdg5F5PRW52WdiHhE5JtttunV8yciT4hIhYhsaLVsgIj8Q0Q+dJ9zQ+zb7vc1gvH9j4hscf/9XhCRnBD7tvtdiGB83xeRfa3+DT8bYt9onb9nW8X2kYisC7FvxM9f2FQ1Jh44Pal2ACcDScD7wLg223wWeBmnR9UZwDu9HOMQYIr7OhPYFiTGWcBfo3gePwIGtbM+quewzb/3AWBENM8fcC4wBdjQatnPgXvc1/cAPwsRf7vf1wjGdz6Q4L7+WbD4OvNdiGB83we+3Yl//6icvzbr/xf4r2idv3AfsVSC6MzcUBcDT6njbSBHRIb0VoCqul9V33Nfe3HGfgzrreP3kKiew1bmADtUtbsj63uEqq7AmWOstYuBP7iv/wBcEmTXznxfIxKfqv5dVZvnRHsbZ5BrVIQ4f50RtfPXzB3D9SXgmZ4+bm+JpQQxDNjb6n0pJ158O7NNrxCRkcBkINjtps4UkfdF5GURGd+7kaHA30VkjTjzYLXVV87hfEL/x4zm+QMoUNX94PwoAPKDbNNXzuOXcUqEwXT0XYik29wqsCdCVNH1hfP3CaBcVT8MsT6a569TYilBdGZuqM5sE3EikgE8B3xTVT1tVr+HU21SDPwaeLGXwztbVafgTON+q4ic22Z91M+hO3L/88Cfg6yO9vnrrL5wHu/FGaj6dIhNOvouRMojOHO3TQL241TjtBX18wdcQfulh2idv06LpQTRmbmhOrNNRIlIIk5yeFpVn2+7XlU9qlrjvl4GJIrIoN6KT53BjahqBfACTlG+taifQ5z/cO+pannbFdE+f67y5mo397kiyDZRPY8ich1wIXCVuhXmbXXiuxARqlquqo2q2gT8LsRxo33+EoAvAs+G2iZa568rYilBdGZuqJeAa92eOGcAR5qrAnqDW2f5OLBZVR8Isc1gdztEZAbOv2FVL8WXLs4U7YhIOk5j5oY2m0X1HLpC/nKL5vlr5SXgOvf1dcBfgmzTme9rRIhzJ8i7gc+ram2IbTrzXYhUfK3btL4Q4rhRO3+uTwFbVLU02Mponr8uiXYreW8+cHrYbMPp3XCvu+xm4Gb3teDcBW8Hzm1Op/VyfOfgFIPX40yDvs6NuXWMtwEbcXplvA2c1Yvxnewe9303hr54DtNwLvjZrZZF7fzhJKr9QAPOr9qvAAOB14EP3ecB7rZDgWXtfV97Kb7tOPX3zd/BR9vGF+q70EvxLXK/W+txLvpD+tL5c5c/2fyda7Vtr5+/cB821YYxxpigYqmKyRhjTBdYgjDGGBOUJQhjjDFBWYIwxhgTlCUIY4wxQVmCMKYLRKRRjp8xtsdmCRWRka1nBTUm2iJ6y1Fj+qE6VZ0U7SCM6Q1WgjCmB7hz+/9MRN51H6e6y0eIyOvuxHKvi8hJ7vIC914L77uPs9yPiheR34lzP5C/i0hq1P4oE/MsQRjTNaltqpgub7XOo6ozgP8DfuUu+z+c6c+LcCa9e8hd/hDwT3UmDZyCM5oWYDTwsKqOB6qBSyP61xjTDhtJbUwXiEiNqmYEWf4R8ElV3elOuHhAVQeKyEGcqSAa3OX7VXWQiFQCharqb/UZI4F/qOpo9/3dQKKq/qgX/jRjTmAlCGN6joZ4HWqbYPytXjdi7YQmiixBGNNzLm/1vNJ9/RbOTKIAVwH/dl+/DtwCICLxIpLVW0Ea01n268SYrkltcxP6V1S1uatrsoi8g/PD6wp32TeAJ0TkLqASWOAuvx1YKCJfwSkp3IIzK6gxfYa1QRjTA9w2iGmqejDasRjTU6yKyRhjTFBWgjDGGBOUlSCMMcYEZQnCGGNMUJYgjDHGBGUJwhhjTFCWIIwxxgT1/wEx0AU1MB28wgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "metrics = pd.read_csv(f\"{trainer.logger.log_dir}/metrics.csv\")\n",
    "\n",
    "aggreg_metrics = []\n",
    "agg_col = \"epoch\"\n",
    "for i, dfg in metrics.groupby(agg_col):\n",
    "    agg = dict(dfg.mean())\n",
    "    agg[agg_col] = i\n",
    "    aggreg_metrics.append(agg)\n",
    "\n",
    "df_metrics = pd.DataFrame(aggreg_metrics)\n",
    "df_metrics[[\"train_loss\", \"valid_loss\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='Loss')\n",
    "df_metrics[[\"train_acc\", \"valid_acc\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='ACC')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "128705f2",
   "metadata": {},
   "source": [
    "- The `trainer` automatically saves the model with the best validation accuracy automatically for us, we which we can load from the checkpoint via the `ckpt_path='best'` argument; below we use the `trainer` instance to evaluate the best model on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4ebf782c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at logs/my-model/version_13/checkpoints/epoch=19-step=4279.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from checkpoint at logs/my-model/version_13/checkpoints/epoch=19-step=4279.ckpt\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b6d15bbfa3ab4c86a91488ebfdcb5a94",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'test_acc': 0.9911999702453613}\n",
      "--------------------------------------------------------------------------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'test_acc': 0.9911999702453613}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.test(model=lightning_model, datamodule=data_module, ckpt_path='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "354b53ad",
   "metadata": {},
   "source": [
    "## Predicting labels of new data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77512166",
   "metadata": {},
   "source": [
    "- You can use the `trainer.predict` method on a new `DataLoader` or `DataModule` to apply the model to new data.\n",
    "- Alternatively, you can also manually load the best model from a checkpoint as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "24c0a90c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logs/my-model/version_13/checkpoints/epoch=19-step=4279.ckpt\n"
     ]
    }
   ],
   "source": [
    "path = trainer.checkpoint_callback.best_model_path\n",
    "print(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "07d994cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "lightning_model = LightningModel.load_from_checkpoint(\n",
    "    path, model=pytorch_model)\n",
    "lightning_model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b822fe2c",
   "metadata": {},
   "source": [
    "- Note that our PyTorch model, which is passed to the Lightning model requires input arguments. However, this is automatically being taken care of since we used `self.save_hyperparameters()` in our PyTorch model's `__init__` method.\n",
    "- Now, below is an example applying the model manually. Here, pretend that the `test_dataloader` is a new data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ec894236",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([7, 2, 1, 0, 4])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataloader = data_module.test_dataloader()\n",
    "\n",
    "all_true_labels = []\n",
    "all_predicted_labels = []\n",
    "for batch in test_dataloader:\n",
    "    features, labels = batch\n",
    "    \n",
    "    with torch.no_grad():  # since we don't need to backprop\n",
    "         logits = lightning_model(features)\n",
    "    \n",
    "    predicted_labels = torch.argmax(logits, dim=1)\n",
    "    all_predicted_labels.append(predicted_labels)\n",
    "    all_true_labels.append(labels)\n",
    "    \n",
    "all_predicted_labels = torch.cat(all_predicted_labels)\n",
    "all_true_labels = torch.cat(all_true_labels)\n",
    "all_predicted_labels[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7301365",
   "metadata": {},
   "source": [
    "Just as an internal check, if the model was loaded correctly, the test accuracy below should be identical to the test accuracy we saw earlier in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e08e8836",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9912 (99.12%)\n"
     ]
    }
   ],
   "source": [
    "test_acc = torch.mean((all_predicted_labels == all_true_labels).float())\n",
    "print(f'Test accuracy: {test_acc:.4f} ({test_acc*100:.2f}%)')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}