pytorch_ipynb/cnn/cnn-densenet121-mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UEBilEjLj5wY"
   },
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 536,
     "status": "ok",
     "timestamp": 1524974472601,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "GOzuY8Yvj5wb",
    "outputId": "c19362ce-f87a-4cc2-84cc-8d7b4b9e6007"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "\n",
      "CPython 3.7.3\n",
      "IPython 7.6.1\n",
      "\n",
      "torch 1.2.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rH4XmErYj5wm"
   },
   "source": [
    "# DenseNet-121 MNIST Digits Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network in this notebook is an implementation of the DenseNet-121 [1] architecture on the MNIST digits dataset (http://yann.lecun.com/exdb/mnist/) to train a handwritten digit classifier.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following figure illustrates the main concept of DenseNet: within each \"dense\" block, each layer is connected with each previous layer -- the feature maps are concatenated.\n",
    "\n",
    "\n",
    "![](../images/densenet/densenet-fig-2.jpg)\n",
    "\n",
    "Note that this is somewhat related yet very different to ResNets. ResNets have skip connections approx. between every other layer (but don't connect all layers with each other). Also, ResNets skip connections work via addition\n",
    "\n",
    "$$\\mathbf{x}_{\\ell}=H_{\\ell}\\left(\\mathbf{X}_{\\ell-1}\\right)+\\mathbf{X}_{\\ell-1}$$,\n",
    "\n",
    "whereas $H_{\\ell}(\\cdot)$ can be a composite function  of operations such as Batch Normalization (BN), rectified linear units (ReLU), Pooling, or Convolution (Conv).\n",
    "\n",
    "In DenseNets, all the previous feature maps $\\mathbf{X}_{0}, \\dots, \\mathbf{X}_{\\ell}-1$ of a feature map $\\mathbf{X}_{\\ell}$ are concatenated:\n",
    "\n",
    "$$\\mathbf{x}_{\\ell}=H_{\\ell}\\left(\\left[\\mathbf{x}_{0}, \\mathbf{x}_{1}, \\ldots, \\mathbf{x}_{\\ell-1}\\right]\\right).$$\n",
    "\n",
    "Furthermore, in this particular notebook, we are considering the DenseNet-121, which is depicted below:\n",
    "\n",
    "\n",
    "\n",
    "![](../images/densenet/densenet-tab-1-dnet121.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References**\n",
    "    \n",
    "- [1] Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708), http://openaccess.thecvf.com/content_cvpr_2017/html/Huang_Densely_Connected_Convolutional_CVPR_2017_paper.html\n",
    "\n",
    "- [2] http://yann.lecun.com/exdb/mnist/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MkoGLH_Tj5wn"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "ORj09gnrj5wp"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.dataset import Subset\n",
    "\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I6hghKPxj5w0"
   },
   "source": [
    "## Model Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23936,
     "status": "ok",
     "timestamp": 1524974497505,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "NnT0sZIwj5wu",
    "outputId": "55aed925-d17e-4c6a-8c71-0d9b3bde5637"
   },
   "outputs": [],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Hyperparameters\n",
    "RANDOM_SEED = 1\n",
    "LEARNING_RATE = 0.0001\n",
    "BATCH_SIZE = 128\n",
    "NUM_EPOCHS = 10\n",
    "\n",
    "# Architecture\n",
    "NUM_CLASSES = 10\n",
    "\n",
    "# Other\n",
    "DEVICE = \"cuda:0\"\n",
    "GRAYSCALE = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MNIST Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_indices = torch.arange(0, 59000)\n",
    "valid_indices = torch.arange(59000, 60000)\n",
    "\n",
    "resize_transform = transforms.Compose([transforms.Resize((32, 32)),\n",
    "                                       transforms.ToTensor()])\n",
    "\n",
    "\n",
    "train_and_valid = datasets.MNIST(root='data', \n",
    "                                 train=True, \n",
    "                                 transform=resize_transform,\n",
    "                                 download=True)\n",
    "\n",
    "test_dataset = datasets.MNIST(root='data', \n",
    "                              train=False, \n",
    "                              transform=resize_transform,\n",
    "                              download=True)\n",
    "\n",
    "train_dataset = Subset(train_and_valid, train_indices)\n",
    "valid_dataset = Subset(train_and_valid, valid_indices)\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=True)\n",
    "\n",
    "valid_loader = DataLoader(dataset=valid_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=False)\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         num_workers=4,\n",
    "                         shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1 | Batch index: 0 | Batch size: 128\n",
      "Epoch: 2 | Batch index: 0 | Batch size: 128\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(DEVICE)\n",
    "torch.manual_seed(0)\n",
    "\n",
    "for epoch in range(2):\n",
    "\n",
    "    for batch_idx, (x, y) in enumerate(train_loader):\n",
    "        \n",
    "        print('Epoch:', epoch+1, end='')\n",
    "        print(' | Batch index:', batch_idx, end='')\n",
    "        print(' | Batch size:', y.size()[0])\n",
    "        \n",
    "        x = x.to(device)\n",
    "        y = y.to(device)\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1, 9, 7, 6, 7, 9, 2, 0, 4, 3])\n",
      "tensor([5, 4, 4, 7, 4, 6, 8, 7, 1, 3])\n"
     ]
    }
   ],
   "source": [
    "# Check that shuffling works properly\n",
    "# i.e., label indices should be in random order.\n",
    "# Also, the label order should be different in the second\n",
    "# epoch.\n",
    "\n",
    "for images, labels in train_loader:  \n",
    "    pass\n",
    "print(labels[:10])\n",
    "\n",
    "for images, labels in train_loader:  \n",
    "    pass\n",
    "print(labels[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2, 2, 0, 9, 4, 0, 1, 2, 3, 4])\n",
      "tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 0])\n"
     ]
    }
   ],
   "source": [
    "# Check that validation set and test sets are diverse\n",
    "# i.e., that they contain all classes\n",
    "\n",
    "for images, labels in valid_loader:  \n",
    "    pass\n",
    "print(labels[:10])\n",
    "\n",
    "for images, labels in test_loader:  \n",
    "    pass\n",
    "print(labels[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "# The following code cell that implements the DenseNet-121 architecture \n",
    "# is a derivative of the code provided at \n",
    "# https://github.com/pytorch/vision/blob/master/torchvision/models/densenet.py\n",
    "\n",
    "import re\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.utils.checkpoint as cp\n",
    "from collections import OrderedDict\n",
    "\n",
    "\n",
    "\n",
    "def _bn_function_factory(norm, relu, conv):\n",
    "    def bn_function(*inputs):\n",
    "        concated_features = torch.cat(inputs, 1)\n",
    "        bottleneck_output = conv(relu(norm(concated_features)))\n",
    "        return bottleneck_output\n",
    "\n",
    "    return bn_function\n",
    "\n",
    "\n",
    "class _DenseLayer(nn.Sequential):\n",
    "    def __init__(self, num_input_features, growth_rate, bn_size, drop_rate, memory_efficient=False):\n",
    "        super(_DenseLayer, self).__init__()\n",
    "        self.add_module('norm1', nn.BatchNorm2d(num_input_features)),\n",
    "        self.add_module('relu1', nn.ReLU(inplace=True)),\n",
    "        self.add_module('conv1', nn.Conv2d(num_input_features, bn_size *\n",
    "                                           growth_rate, kernel_size=1, stride=1,\n",
    "                                           bias=False)),\n",
    "        self.add_module('norm2', nn.BatchNorm2d(bn_size * growth_rate)),\n",
    "        self.add_module('relu2', nn.ReLU(inplace=True)),\n",
    "        self.add_module('conv2', nn.Conv2d(bn_size * growth_rate, growth_rate,\n",
    "                                           kernel_size=3, stride=1, padding=1,\n",
    "                                           bias=False)),\n",
    "        self.drop_rate = drop_rate\n",
    "        self.memory_efficient = memory_efficient\n",
    "\n",
    "    def forward(self, *prev_features):\n",
    "        bn_function = _bn_function_factory(self.norm1, self.relu1, self.conv1)\n",
    "        if self.memory_efficient and any(prev_feature.requires_grad for prev_feature in prev_features):\n",
    "            bottleneck_output = cp.checkpoint(bn_function, *prev_features)\n",
    "        else:\n",
    "            bottleneck_output = bn_function(*prev_features)\n",
    "        new_features = self.conv2(self.relu2(self.norm2(bottleneck_output)))\n",
    "        if self.drop_rate > 0:\n",
    "            new_features = F.dropout(new_features, p=self.drop_rate,\n",
    "                                     training=self.training)\n",
    "        return new_features\n",
    "\n",
    "\n",
    "class _DenseBlock(nn.Module):\n",
    "    def __init__(self, num_layers, num_input_features, bn_size, growth_rate, drop_rate, memory_efficient=False):\n",
    "        super(_DenseBlock, self).__init__()\n",
    "        for i in range(num_layers):\n",
    "            layer = _DenseLayer(\n",
    "                num_input_features + i * growth_rate,\n",
    "                growth_rate=growth_rate,\n",
    "                bn_size=bn_size,\n",
    "                drop_rate=drop_rate,\n",
    "                memory_efficient=memory_efficient,\n",
    "            )\n",
    "            self.add_module('denselayer%d' % (i + 1), layer)\n",
    "\n",
    "    def forward(self, init_features):\n",
    "        features = [init_features]\n",
    "        for name, layer in self.named_children():\n",
    "            new_features = layer(*features)\n",
    "            features.append(new_features)\n",
    "        return torch.cat(features, 1)\n",
    "\n",
    "\n",
    "class _Transition(nn.Sequential):\n",
    "    def __init__(self, num_input_features, num_output_features):\n",
    "        super(_Transition, self).__init__()\n",
    "        self.add_module('norm', nn.BatchNorm2d(num_input_features))\n",
    "        self.add_module('relu', nn.ReLU(inplace=True))\n",
    "        self.add_module('conv', nn.Conv2d(num_input_features, num_output_features,\n",
    "                                          kernel_size=1, stride=1, bias=False))\n",
    "        self.add_module('pool', nn.AvgPool2d(kernel_size=2, stride=2))\n",
    "\n",
    "\n",
    "class DenseNet121(nn.Module):\n",
    "    r\"\"\"Densenet-BC model class, based on\n",
    "    `\"Densely Connected Convolutional Networks\" <https://arxiv.org/pdf/1608.06993.pdf>`_\n",
    "\n",
    "    Args:\n",
    "        growth_rate (int) - how many filters to add each layer (`k` in paper)\n",
    "        block_config (list of 4 ints) - how many layers in each pooling block\n",
    "        num_init_featuremaps (int) - the number of filters to learn in the first convolution layer\n",
    "        bn_size (int) - multiplicative factor for number of bottle neck layers\n",
    "          (i.e. bn_size * k features in the bottleneck layer)\n",
    "        drop_rate (float) - dropout rate after each dense layer\n",
    "        num_classes (int) - number of classification classes\n",
    "        memory_efficient (bool) - If True, uses checkpointing. Much more memory efficient,\n",
    "          but slower. Default: *False*. See `\"paper\" <https://arxiv.org/pdf/1707.06990.pdf>`_\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, growth_rate=32, block_config=(6, 12, 24, 16),\n",
    "                 num_init_featuremaps=64, bn_size=4, drop_rate=0, num_classes=1000, memory_efficient=False,\n",
    "                 grayscale=False):\n",
    "\n",
    "        super(DenseNet121, self).__init__()\n",
    "\n",
    "        # First convolution\n",
    "        if grayscale:\n",
    "            in_channels=1\n",
    "        else:\n",
    "            in_channels=3\n",
    "        \n",
    "        self.features = nn.Sequential(OrderedDict([\n",
    "            ('conv0', nn.Conv2d(in_channels=in_channels, out_channels=num_init_featuremaps,\n",
    "                                kernel_size=7, stride=2,\n",
    "                                padding=3, bias=False)), # bias is redundant when using batchnorm\n",
    "            ('norm0', nn.BatchNorm2d(num_features=num_init_featuremaps)),\n",
    "            ('relu0', nn.ReLU(inplace=True)),\n",
    "            ('pool0', nn.MaxPool2d(kernel_size=3, stride=2, padding=1)),\n",
    "        ]))\n",
    "\n",
    "        # Each denseblock\n",
    "        num_features = num_init_featuremaps\n",
    "        for i, num_layers in enumerate(block_config):\n",
    "            block = _DenseBlock(\n",
    "                num_layers=num_layers,\n",
    "                num_input_features=num_features,\n",
    "                bn_size=bn_size,\n",
    "                growth_rate=growth_rate,\n",
    "                drop_rate=drop_rate,\n",
    "                memory_efficient=memory_efficient\n",
    "            )\n",
    "            self.features.add_module('denseblock%d' % (i + 1), block)\n",
    "            num_features = num_features + num_layers * growth_rate\n",
    "            if i != len(block_config) - 1:\n",
    "                trans = _Transition(num_input_features=num_features,\n",
    "                                    num_output_features=num_features // 2)\n",
    "                self.features.add_module('transition%d' % (i + 1), trans)\n",
    "                num_features = num_features // 2\n",
    "\n",
    "        # Final batch norm\n",
    "        self.features.add_module('norm5', nn.BatchNorm2d(num_features))\n",
    "\n",
    "        # Linear layer\n",
    "        self.classifier = nn.Linear(num_features, num_classes)\n",
    "\n",
    "        # Official init from torch repo.\n",
    "        for m in self.modules():\n",
    "            if isinstance(m, nn.Conv2d):\n",
    "                nn.init.kaiming_normal_(m.weight)\n",
    "            elif isinstance(m, nn.BatchNorm2d):\n",
    "                nn.init.constant_(m.weight, 1)\n",
    "                nn.init.constant_(m.bias, 0)\n",
    "            elif isinstance(m, nn.Linear):\n",
    "                nn.init.constant_(m.bias, 0)\n",
    "\n",
    "    def forward(self, x):\n",
    "        features = self.features(x)\n",
    "        out = F.relu(features, inplace=True)\n",
    "        out = F.adaptive_avg_pool2d(out, (1, 1))\n",
    "        out = torch.flatten(out, 1)\n",
    "        logits = self.classifier(out)\n",
    "        probas = F.softmax(logits, dim=1)\n",
    "        return logits, probas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "_lza9t_uj5w1"
   },
   "outputs": [],
   "source": [
    "torch.manual_seed(RANDOM_SEED)\n",
    "\n",
    "model = DenseNet121(num_classes=NUM_CLASSES, grayscale=GRAYSCALE)\n",
    "model.to(DEVICE)\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RAodboScj5w6"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_acc(model, data_loader, device):\n",
    "    correct_pred, num_examples = 0, 0\n",
    "    model.eval()\n",
    "    for i, (features, targets) in enumerate(data_loader):\n",
    "            \n",
    "        features = features.to(device)\n",
    "        targets = targets.to(device)\n",
    "\n",
    "        logits, probas = model(features)\n",
    "        _, predicted_labels = torch.max(probas, 1)\n",
    "        num_examples += targets.size(0)\n",
    "        assert predicted_labels.size() == targets.size()\n",
    "        correct_pred += (predicted_labels == targets).sum()\n",
    "    return correct_pred.float()/num_examples * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 1547
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2384585,
     "status": "ok",
     "timestamp": 1524976888520,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "Dzh3ROmRj5w7",
    "outputId": "5f8fd8c9-b076-403a-b0b7-fd2d498b48d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/010 | Batch 000/461 | Cost: 2.3666\n",
      "Epoch: 001/010 | Batch 150/461 | Cost: 0.1477\n",
      "Epoch: 001/010 | Batch 300/461 | Cost: 0.0662\n",
      "Epoch: 001/010 | Batch 450/461 | Cost: 0.1357\n",
      "Epoch: 001/010\n",
      "Train ACC: 98.87 | Validation ACC: 98.10\n",
      "Time elapsed: 0.93 min\n",
      "Epoch: 002/010 | Batch 000/461 | Cost: 0.0476\n",
      "Epoch: 002/010 | Batch 150/461 | Cost: 0.0443\n",
      "Epoch: 002/010 | Batch 300/461 | Cost: 0.0520\n",
      "Epoch: 002/010 | Batch 450/461 | Cost: 0.0377\n",
      "Epoch: 002/010\n",
      "Train ACC: 99.58 | Validation ACC: 98.50\n",
      "Time elapsed: 2.01 min\n",
      "Epoch: 003/010 | Batch 000/461 | Cost: 0.0080\n",
      "Epoch: 003/010 | Batch 150/461 | Cost: 0.0085\n",
      "Epoch: 003/010 | Batch 300/461 | Cost: 0.0125\n",
      "Epoch: 003/010 | Batch 450/461 | Cost: 0.0178\n",
      "Epoch: 003/010\n",
      "Train ACC: 99.57 | Validation ACC: 98.90\n",
      "Time elapsed: 3.10 min\n",
      "Epoch: 004/010 | Batch 000/461 | Cost: 0.0057\n",
      "Epoch: 004/010 | Batch 150/461 | Cost: 0.0368\n",
      "Epoch: 004/010 | Batch 300/461 | Cost: 0.0026\n",
      "Epoch: 004/010 | Batch 450/461 | Cost: 0.1089\n",
      "Epoch: 004/010\n",
      "Train ACC: 99.79 | Validation ACC: 98.90\n",
      "Time elapsed: 4.17 min\n",
      "Epoch: 005/010 | Batch 000/461 | Cost: 0.0026\n",
      "Epoch: 005/010 | Batch 150/461 | Cost: 0.0098\n",
      "Epoch: 005/010 | Batch 300/461 | Cost: 0.0161\n",
      "Epoch: 005/010 | Batch 450/461 | Cost: 0.0116\n",
      "Epoch: 005/010\n",
      "Train ACC: 99.77 | Validation ACC: 99.10\n",
      "Time elapsed: 5.23 min\n",
      "Epoch: 006/010 | Batch 000/461 | Cost: 0.0033\n",
      "Epoch: 006/010 | Batch 150/461 | Cost: 0.0009\n",
      "Epoch: 006/010 | Batch 300/461 | Cost: 0.0114\n",
      "Epoch: 006/010 | Batch 450/461 | Cost: 0.0582\n",
      "Epoch: 006/010\n",
      "Train ACC: 99.59 | Validation ACC: 98.50\n",
      "Time elapsed: 6.33 min\n",
      "Epoch: 007/010 | Batch 000/461 | Cost: 0.0034\n",
      "Epoch: 007/010 | Batch 150/461 | Cost: 0.0059\n",
      "Epoch: 007/010 | Batch 300/461 | Cost: 0.0064\n",
      "Epoch: 007/010 | Batch 450/461 | Cost: 0.0239\n",
      "Epoch: 007/010\n",
      "Train ACC: 99.69 | Validation ACC: 97.90\n",
      "Time elapsed: 7.42 min\n",
      "Epoch: 008/010 | Batch 000/461 | Cost: 0.0042\n",
      "Epoch: 008/010 | Batch 150/461 | Cost: 0.0282\n",
      "Epoch: 008/010 | Batch 300/461 | Cost: 0.0022\n",
      "Epoch: 008/010 | Batch 450/461 | Cost: 0.0025\n",
      "Epoch: 008/010\n",
      "Train ACC: 99.85 | Validation ACC: 99.10\n",
      "Time elapsed: 8.51 min\n",
      "Epoch: 009/010 | Batch 000/461 | Cost: 0.0090\n",
      "Epoch: 009/010 | Batch 150/461 | Cost: 0.0034\n",
      "Epoch: 009/010 | Batch 300/461 | Cost: 0.0026\n",
      "Epoch: 009/010 | Batch 450/461 | Cost: 0.0042\n",
      "Epoch: 009/010\n",
      "Train ACC: 99.81 | Validation ACC: 98.30\n",
      "Time elapsed: 9.53 min\n",
      "Epoch: 010/010 | Batch 000/461 | Cost: 0.0413\n",
      "Epoch: 010/010 | Batch 150/461 | Cost: 0.0042\n",
      "Epoch: 010/010 | Batch 300/461 | Cost: 0.0021\n",
      "Epoch: 010/010 | Batch 450/461 | Cost: 0.0040\n",
      "Epoch: 010/010\n",
      "Train ACC: 99.88 | Validation ACC: 98.80\n",
      "Time elapsed: 10.63 min\n",
      "Total Training Time: 10.63 min\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "cost_list = []\n",
    "train_acc_list, valid_acc_list = [], []\n",
    "\n",
    "\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    \n",
    "    model.train()\n",
    "    for batch_idx, (features, targets) in enumerate(train_loader):\n",
    "        \n",
    "        features = features.to(DEVICE)\n",
    "        targets = targets.to(DEVICE)\n",
    "            \n",
    "        ### FORWARD AND BACK PROP\n",
    "        logits, probas = model(features)\n",
    "        cost = F.cross_entropy(logits, targets)\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        cost.backward()\n",
    "        \n",
    "        ### UPDATE MODEL PARAMETERS\n",
    "        optimizer.step()\n",
    "        \n",
    "        #################################################\n",
    "        ### CODE ONLY FOR LOGGING BEYOND THIS POINT\n",
    "        ################################################\n",
    "        cost_list.append(cost.item())\n",
    "        if not batch_idx % 150:\n",
    "            print (f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d} | '\n",
    "                   f'Batch {batch_idx:03d}/{len(train_loader):03d} |' \n",
    "                   f' Cost: {cost:.4f}')\n",
    "\n",
    "        \n",
    "\n",
    "    model.eval()\n",
    "    with torch.set_grad_enabled(False): # save memory during inference\n",
    "        \n",
    "        train_acc = compute_acc(model, train_loader, device=DEVICE)\n",
    "        valid_acc = compute_acc(model, valid_loader, device=DEVICE)\n",
    "        \n",
    "        print(f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d}\\n'\n",
    "              f'Train ACC: {train_acc:.2f} | Validation ACC: {valid_acc:.2f}')\n",
    "        \n",
    "        train_acc_list.append(train_acc)\n",
    "        valid_acc_list.append(valid_acc)\n",
    "        \n",
    "    elapsed = (time.time() - start_time)/60\n",
    "    print(f'Time elapsed: {elapsed:.2f} min')\n",
    "  \n",
    "elapsed = (time.time() - start_time)/60\n",
    "print(f'Total Training Time: {elapsed:.2f} min')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "paaeEQHQj5xC"
   },
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6514,
     "status": "ok",
     "timestamp": 1524976895054,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "gzQMWKq5j5xE",
    "outputId": "de7dc005-5eeb-4177-9f9f-d9b5d1358db9"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXhU5dn48e89k8ke1rCjAi2ICAmyiYCIouCCuGEFfW0ptvLaUqyt9ke1daHLa1ttK651V2px16qgCCiiKGpYZZNNlrCGBLJvM/P8/jhnJpNkkgyQySSc+3Ndc82Zs97nJHPueZ7znOeIMQallFLO5Yp1AEoppWJLE4FSSjmcJgKllHI4TQRKKeVwmgiUUsrh4mIdwLFKT083PXr0iHUYSinVoqxcufKwMaZDuGktLhH06NGDrKysWIehlFItiojsqmuaVg0ppZTDaSJQSimH00SglFIO1+KuESiljl9lZSXZ2dmUlZXFOhQVJYmJiXTv3h2PxxPxMpoIlHKQ7Oxs0tLS6NGjByIS63BUIzPGkJubS3Z2Nj179ox4Oa0aUspBysrKaN++vSaBk5SI0L59+2Mu8WkiUMphNAmc3I7n7+uYRPDtgUIe/PBbcovKYx2KUko1K45JBNsOFfHwR9s4XFQR61CUcjQR4cYbbwx+9nq9dOjQgQkTJgDwzjvvcP/999e7jn379jFp0iQAnn/+eWbMmHFMMfz5z39ucJ6pU6fy+uuvH9N6j8eaNWtYsGBB1LdTH8ckAo/bKi5V+vwxjkQpZ0tJSWH9+vWUlpYCsGjRIrp16xacPnHiRGbNmlXvOrp27XpCJ+lIEkFT0UTQhDxua1c1ESgVe5dccgnz588HYN68eUyZMiU4LfQX/tSpU5k5cyYjRoygV69ewZP/zp076d+/f3CZPXv2cPHFF3P66adz3333BcdfeeWVDB48mDPPPJMnn3wSgFmzZlFaWsrAgQO54YYbAHjxxRfJyMggMzOzWmll2bJltbZdU7hld+3axdixY8nIyGDs2LHs3r0bgNdee43+/fuTmZnJ6NGjqaio4O677+aVV15h4MCBvPLKKyd2YI+TY5qPxtklAq9fH82pFMB9725g476CRl1nv66tuOfyMxucb/LkycyePZsJEyawbt06pk2bxqeffhp23v379/PZZ5+xefNmJk6cGKwSCvXVV1+xfv16kpOTGTp0KJdddhlDhgzh2WefpV27dpSWljJ06FCuueYa7r//fh555BHWrFkDwIYNG/jTn/7E8uXLSU9PJy8vL+Jt17XsjBkz+OEPf8iPfvQjnn32WWbOnMnbb7/N7NmzWbhwId26dePo0aPEx8cze/ZssrKyeOSRRyI+zo3NMSWCOJeWCJRqLjIyMti5cyfz5s3j0ksvrXfeK6+8EpfLRb9+/Th48GDYeS666CLat29PUlISV199NZ999hkAc+bMITMzk+HDh7Nnzx62bt1aa9mPPvqISZMmkZ6eDkC7du0i3nZdy37xxRdcf/31ANx4443BeEaOHMnUqVN56qmn8Pl89e53U3JMiSA+zi4R+LREoBQQ0S/3aJo4cSK33347S5cuJTc3t875EhISgsPGhP/+1mwyKSIsXbqUxYsX88UXX5CcnMyYMWPCtq83xtTZ5LKhbde3bLj4nnjiCb788kvmz5/PwIEDg6WSWNMSgVIqJqZNm8bdd9/NgAEDTnhdixYtIi8vj9LSUt5++21GjhxJfn4+bdu2JTk5mc2bN7NixYrg/B6Ph8rKSgDGjh3Lq6++GkxGoVVDDalr2REjRvDyyy8D8NJLLzFq1CgAtm/fztlnn83s2bNJT09nz549pKWlUVhYeMLH4EQ4JxEEWw1piUCp5qB79+7ceuutjbKuUaNGceONNzJw4ECuueYahgwZwsUXX4zX6yUjI4Pf//73DB8+PDj/zTffTEZGBjfccANnnnkmd911F+eddx6ZmZn86le/ini7dS07Z84cnnvuOTIyMpg7dy4PPfQQAHfccQcDBgygf//+jB49mszMTM4//3w2btwY04vFUldRq7kaMmSIOZ4H02w5WMi4fyzjkevPYkJG1yhEplTzt2nTJs4444xYh6GiLNzfWURWGmOGhJvfOSUCl14jUEqpcByTCAL3EVToNQKllKrGcYlASwRKKVWdYxJB1Q1lWiJQSqlQjkkEnmDzUS0RKKVUKOckgjjtdE4ppcJxTCII3FDm1USgVEy53W4GDhxI//79ufzyyzl69GhUtjNixIiorPdk5JhE4NEbypRqFpKSklizZg3r16+nXbt2PProo1HZzueffx6V9TaG5tTPEDgoEYgIIuBvYTfQKXUyO+ecc9i7dy8AS5cuDT6cBqwePJ9//nkAevTowT333MOgQYMYMGAAmzdvBuDee+9l2rRpjBkzhl69ejFnzpzg8qmpqcH1jhkzhkmTJtG3b19uuOGGYL9BCxYsoG/fvowaNYqZM2dW237Azp07Offccxk0aBCDBg0KJpjrrruu2nMEpk6dyhtvvIHP5+OOO+5g6NChZGRk8K9//SsYx/nnn8/1118f7FYjXDfZAM888wx9+vRhzJgx/PSnPw12y52Tk8M111zD0KFDGTp0KMuXLz+Bo1/FMZ3OAbhFNBEoFfD+LDjwTeOus/MAuKT+p4sF+Hw+lixZwk033RTR/Onp6axatYrHHnuMBx54gKeffhqAzZs38/HHH1NYWMjpp5/OLbfcgsfjqbbs6tWr2bBhA127dmXkyJEsX76cIUOGMH36dJYtW0bPnj2rPRMhVMeOHVm0aBGJiYls3bqVKVOmkJWVxeTJk3nllVe49NJLqaioYMmSJTz++OM888wztG7dmq+//pry8nJGjhzJuHHjgKrusnv27AkQtpvs8vJy/vCHP7Bq1SrS0tK44IILyMzMBODWW2/ltttuY9SoUezevZvx48ezadOmiI5ffRyVCFwi6CUCpWIr8FCYnTt3MnjwYC666KKIlrv66qsBGDx4MG+++WZw/GWXXUZCQgIJCQl07NiRgwcP0r1792rLDhs2LDgusO3U1FR69eoVPClPmTKl2q/ygMrKSmbMmMGaNWtwu91s2bIFsB6uM3PmTMrLy/nggw8YPXo0SUlJfPjhh6xbty74IJv8/Hy2bt1KfHw8w4YNC24PrD6J3nrrLYBgN9kHDhzgvPPOC3Zpfe211wa3uXjxYjZu3BhcvqCggMLCQtLS0iI6hnVxViJwadWQUkER/nJvbIFrBPn5+UyYMIFHH32UmTNnEhcXhz/kPp+aXUYHuoR2u914vd5a48NNq2+eSPtZ+8c//kGnTp1Yu3Ytfr+fxMREABITExkzZgwLFy7klVdeCZYojDE8/PDDjB8/vtp6li5dSkpKSrXP4brJri8uv9/PF198QVJSUkSxR8ox1wjArhrSJ5Qp1Sy0bt2aOXPm8MADD1BZWclpp53Gxo0bKS8vJz8/nyVLlkR1+3379mXHjh3s3LkToM6eP/Pz8+nSpQsul4u5c+dWu9A7efJknnvuOT799NPgiX/8+PE8/vjjwW6ut2zZQnFxcdj1husme9iwYXzyySccOXIEr9fLG2+8EVxm3Lhx1Z5k1ljPM3BUInCJ4NMSgVLNxllnnUVmZiYvv/wyp5xyCj/4wQ+C3UOfddZZUd12UlISjz32GBdffDGjRo2iU6dOtG7dutZ8P/vZz3jhhRcYPnw4W7Zsqfarfty4cSxbtowLL7yQ+Ph4AH7yk5/Qr18/Bg0aRP/+/Zk+fXrYUkpd3WR369aNO++8k7PPPpsLL7yQfv36BeOaM2cOWVlZZGRk0K9fP5544olGORaO6YYaIPO+D7lyYFfuu6J/wzMrdRLSbqirKyoqIjU1FWMMP//5z+nduze33XZbrMMKxuX1ernqqquYNm0aV111VcTLazfU9XC7BK0ZUkoFPPXUUwwcOJAzzzyT/Px8pk+fHuuQAKtZbOCmu549e3LllVdGdXvOulgsaNWQUirotttuaxYlgJoeeOCBJt2eo0oELr1YrFTErWVUy3Q8f19HJQKraki/BMq5EhMTyc3N1WRwkjLGkJubG2ziGqmoVQ2JyCnAi0BnwA88aYx5qMY8AjwEXAqUAFONMauiFZPeUKacrnv37mRnZ5OTkxPrUFSUJCYm1rqhriHRvEbgBX5tjFklImnAShFZZIzZGDLPJUBv+3U28Lj9HhV6Q5lyOo/HU+3OVqUgilVDxpj9gV/3xphCYBPQrcZsVwAvGssKoI2IdIlWTG4RfHqNQCmlqmmSawQi0gM4C/iyxqRuwJ6Qz9nUThaIyM0ikiUiWSdSpHXpNQKllKol6olARFKBN4BfGmMKak4Os0itM7Ux5kljzBBjzJAOHTocdywu7X1UKaVqiWoiEBEPVhJ4yRjzZphZsoFTQj53B/ZFKx6tGlJKqdqilgjsFkHPAJuMMX+vY7Z3gB+KZTiQb4zZH62Yvj1YyMINB6O1eqWUapGi2WpoJHAj8I2IBLrIuxM4FcAY8wSwAKvp6Das5qM/jmI8SimlwohaIjDGfEb4awCh8xjg59GKoaZze6dTVF67F0CllHIyR91ZrF1MKKVUbY5KBNr7qFJK1eaoROAStNWQUkrV4LBEoPcRKKVUTZoIlFLK4RyVCESg3KvdjyqlVChHPaHs/fUHYh2CUko1O44qESillKpNE4FSSjmcJgKllHI4RyYCfV6rUkpVcWgiiHUESinVfDgqEfz6oj4A+DQTKKVUkKMSgctldYaqN5UppVQVZyUCsROB3lOmlFJBjkoEbntvtUSglFJVHJUIAvQagVJKVXFUInhs6XYA1uw+GuNIlFKq+XBUIjhaUgnAgfyyGEeilFLNh6MSQVC9T1JWSilncVQiuHVsbwB6pafEOBKllGo+HJUI4uOs3X34o20xjkQppZoPRyWCnMJyAFbvPhLjSJRSqvlwVCJw23cW6wPslVKqijMTgd5HoJRSQY5MBNrFhFJKVXFWIrD7GvJqJlBKqSBHJYJJg7sD8NPRvWIciVJKNR+OSgRtU+IB6JCaEONIlFKq+WgwEYiIuykCaQpufR6BUkrVEkmJYJuI/E1E+kU9migLXCPw6SUCpZQKiiQRZABbgKdFZIWI3CwiraIcV1S49HkESilVS4OJwBhTaIx5yhgzAvgNcA+wX0ReEJHv17WciDwrIodEZH0d08eISL6IrLFfdx/3XkSoqkSgiUAppQLiGprBvkZwGfBjoAfwIPAScC6wAOhTx6LPA48AL9az+k+NMRMiD/fE6J3FSilVW4OJANgKfAz8zRjzecj410VkdF0LGWOWiUiPEwuvcYldIjhcVB7jSJRSqvmI6BqBMeamGkkAAGPMzBPc/jkislZE3heRM09wXRF76cvdTbUppZRq9iJJBB1F5F0ROWzX+f9XRBrjjqxVwGnGmEzgYeDtuma0L1BniUhWTk5OI2xaKaVUQCSJ4D/Aq0BnoCvwGjDvRDdsjCkwxhTZwwsAj4ik1zHvk8aYIcaYIR06dDjRTSullAoRSSIQY8xcY4zXfv0bOOGrrSLSWexKexEZZseSe6LrVUopdWwiuVj8sYjMAl7GSgDXAfNFpB2AMSYv3EIiMg8YA6SLSDZWs1OPvcwTwCTgFhHxAqXAZGO0gb9SSjW1SBLBdfb79Brjp2ElhrDXC4wxU+pbqTHmEazmpUoppWKowURgjOnZFIEopZSKjUhuKPMAtwCBewaWAv8yxlRGMS6llFJNJJKqocex6vYfsz/faI/7SbSCUkop1XQiSQRD7bb+AR+JyNpoBaSUUqppRdJ81Cci3wt8sG8m80UvJKWUUk0pkkRwB1YT0qUi8gnwEfDr6IYVPdNH9yLR46gHsymlVL3qrRoSERdWG//ewOmAAJuNMS221zYRQZ9dr5RSVepNBMYYv4g8aIw5B1jXRDFFldsFPr1vTSmlgiKpI/lQRK4JdAfR0rlF9AllSikVIpJWQ78CUgCviJRhVQ8ZY0yLfFyliGAMGGM4SXKbUkqdkEjuLE5rikCaSuhTyuLcmgiUUqrBqiERWRLJuJbCzgPo0yqVUspSZ4lARBKBZKzeQ9tiVQkBtMJ6LkGL5LIzgV4nUEopS31VQ9OBX2Kd9FdSlQgKgEejHFfUuEUTgVJKhaozERhjHgIeEpFfGGMebsKYosolVdcIlFJKRXax+GERGQH0CJ3fGPNiFOOKmjdWZQPwyZYcJmS02BoupZRqNJF0Qz0X+B6whqo+hgzQIhPB5gOFAGw7VBTjSJRSqnmI5D6CIUC/k+0xkifX3iil1PGL5M7i9UDnaAfSVCTYfFQzgVJKQWQlgnRgo4h8BQQ7mzPGTIxaVFHkEsFnjCYCpZSyRZII7o12EE3JJdaFDs0DSillqe+Gsr7GmM3GmE9EJCG062kRGd404TU+q38ho3cWK6WUrb5rBP8JGf6ixrTHaKECd8WdZNe+lVLquNWXCKSO4XCfW4xze6cD0L1tUowjUUqp5qG+RGDqGA73ucW4aVQvAL7XITXGkSilVPNQ38Xi7iIyB+vXf2AY+3O3qEcWJYHeR1tsJlNKqUZWXyK4I2Q4q8a0mp9bjF15JQC8vjKbkd9Pj3E0SikVe/V1OvdCUwbSVLbYXUws3HAgxpEopVTzEMmdxSeVQJVQi73arZRSjcxxiSA+ztrlBI87xpEopVTz4LhEMGXoqQD8/PzvxzgSpZRqHiJ5ZvFfRaSViHhEZImIHBaR/2mK4KIh0WPt8gfr98c4EqWUah4iKRGMM8YUABOAbKAP1VsUtShuu/3o1zuPxDgSpZRqHiJJBB77/VJgnjEmL5IVi8izInJIRNbXMV1EZI6IbBORdSIyKMKYT0ggESillLJEkgjeFZHNWA+oWSIiHYCyCJZ7Hri4numXAL3t183A4xGs84S5NBEopVQ1DSYCY8ws4BxgiDGmEigGrohguWVAfaWHK4AXjWUF0EZEukQW9vFziyYCpZQKFcnF4msBrzHGJyK/A/4NNMZT37sBe0I+Z1NH1xUicrOIZIlIVk5OzgltVPOAUkpVF0nV0O+NMYUiMgoYD7xA41TjhDslh+0CyBjzpDFmiDFmSIcOHU5oox6341rMKqVUvSI5K/rs98uAx40x/wXiG2Hb2cApIZ+7A/saYb318rhd9OmUyvBe7aK9KaWUahEiSQR7ReRfwA+ABSKSEOFyDXkH+KHdemg4kG+MaZLG/W2SGiOPKaXUySGSZxb/AKv1zwPGmKP2Bd0G7yMQkXnAGCBdRLKBe7CbohpjngAWYDVJ3QaUAD8+nh04HjlF5Xx3uLipNqeUUs1ag4nAGFMiItuB8SIyHvjUGPNhBMtNaWC6AX4ecaSNSJOAUkpViaTV0K3AS0BH+/VvEflFtANTSinVNCKpGroJONsYUwwgIn/Bepj9w9EMTCmlVNOI5KKvUNVyCHtYW+MrpdRJIpJE8BzwpYjcKyL3AiuAZ6IaVZTdPLoXSfo8AqWUAiK7WPx3EVkKjMIqCfzYGLM62oFFkwB+o4+vV0opaCARiIgLWGeM6Q+sapqQmoDUcQuzUko5UL1VQ8YYP7BWRE5toniaxOpdR6nw+qn0+WMdilJKxVwkrYa6ABtE5CusnkcBMMZMjFpUUfbVTqtT1D15JfTqkBrjaJRSKrYiSQT3RT2KGNHqIaWUqicRiMj3gU7GmE9qjB8N7I12YE3B6AVjpZSq9xrBP4HCMONL7GktnuYBpZSqPxH0MMasqznSGJMF9IhaRE1I84BSStWfCBLrmZbU2IHEgt5LoJRS9SeCr0XkpzVHishNwMrohdR0jhRXxjoEpZSKufpaDf0SeEtEbqDqxD8E6+lkV0U7sKZw37sb+OCXo2MdhlJKxVSdicAYcxAYISLnA/3t0fONMR81SWRNoLTS1/BMSil1koukr6GPgY+bIJYmM2XYqcz7ajeHC8tjHYpSSsVcYzx7uMW5aVRPAIortESglFKOTAQd0hJiHYJSSjUbjkwE8W5H7rZSSoXlyDOix60PWFNKqQBHJgK3SxOBUkoFODIRiGgiUEqpgEi6oT4picDwnu1jHYZSSsWcI0sEAP26tCIlQR9gr5RSjk0EpRU+9h0ti3UYSikVc46tGtpxuLjhmZRSygEcWyJQSill0USglFIOp4lAKaUczjmJ4Nv34YHTIW9HrCNRSqlmxTmJwBUHRQeg+HC10Q8t3hqjgJRSqnlwTiJItm8eq5EIdueVxCAYpZRqPqKaCETkYhH5VkS2icisMNOnikiOiKyxXz+JWjAp6dZ7iZUI/nXjYEC7pFZKqaglAhFxA48ClwD9gCki0i/MrK8YYwbar6ejFQ/JdiKwSwTZR0oBeOKT7VHbpFJKtQTRLBEMA7YZY3YYYyqAl4Erori9+sUngycZSnIB0A5IlVLKEs1E0A3YE/I52x5X0zUisk5EXheRU8KtSERuFpEsEcnKyck5/oiS04MlAs0DSilliWYiCHeuNTU+vwv0MMZkAIuBF8KtyBjzpDFmiDFmSIcOHY4/opT2wWsE2hW1UkpZopkIsoHQX/jdgX2hMxhjco0x5fbHp4DBUYyneokgJA/4/TXzk1JKOUc0E8HXQG8R6Ski8cBk4J3QGUSkS8jHicCmKMZjtRyyrxGEuv21teQWlYdZQCmlTn5R633UGOMVkRnAQsANPGuM2SAis4EsY8w7wEwRmQh4gTxgarTiAax7CewSgQkpBLy5ei9xbuGvkzKjunmllGqOotoNtTFmAbCgxri7Q4Z/C/w2mjFUk9oRvKVQXogxWh2klFLgpDuLAVrZjZby9zKsxmMqvz1QGIOAlFIq9pyVCFp3t97zs+nXtVW1SaWVvhgEpJRSsefQRGDd3pCeGh+c5NLmpEoph3JWIkjtDOKG/Gx7RNXJP6dQWw0ppZzJWYnAHQetukLBXgBGfr/qOkFucQXLtx2ua0mllDppOSsRgFU9ZJcIfnpur2qT1u/Nj0VESikVU85LBK26Ba8RJMRV3329TqCUciLnJYLW3SF/L/j9xLlrJALtklQp5UDOTAT+Sig6SFyNE3/gY7nXx/acohgEp5RSTc95iaCdfV0gdxsJnuq7f9+7GwG46631jH3wE44UVzR1dEop1eSclwg62g9JO7SJjmmJYWf5YrvVMV1RubepolJKqZhxXiJI6wxJbeHQhjpn0WvGSikncV4iELFKBYfq7vE68Dxj7ZdOKeUEzksEAJ0HwP514K19DeCCB5cGhw2GNXuO0mPWfDbtL2hwtcYYvUNZKdXiODMRnDbC6o56/9pak3bkFAeHv9mbz/vr9wPw8beHGlztE5/sYOifFrM7t6TxYlVKqShzaCIYCQhs/ZCXbx5e52wz/rM6eJPZ0m9zGlxtIFnsyy9tlDCVUqopODMRpKRDz9Gw/nWG92xX76yPL90OwFff5TW42sDDbvQOZaVUS+LMRAAw4FrI2wH7VjF1RI+IFnn/m/11Tvt48yEKy6zmpnqDslKqJXFuIjjjcnDHw+qXuHfimREtcstLq5jy5Ipa4/fklfDj579ms/2UM9ESgVKqBXFuIkhqA5lTYPVcOLon4sW+2JHL+9/s5/qnqhJCcUX1G8+0RKCUakmcmwgARt8B4oL3bkPwR7zYLS+t4nP77mOofb+BS4S3V+/l1a8jTzBKKRUrcbEOIKbanALj/ggLbmeauyPP+C47psUPFZYx/M9L8NdIBG+uyuaFL3YB4HYJX36Xy52XnkGb5Pgwa1FKqdhydokAYOhPoO8E7op/hfNcte8rqM+wP9VOAkAwCQD8+rW1vJqVzT8Xbz3RSBv090Vb6PXb+VHfjlLq5KKJQAQmPoyr4xm8EP8X/hD3LIk0/t3BFb7qVU/rso/S+64F9bZECvhs62HWZR9l9e4j9c43Z8nWsIlJKaXqo4kAILkd3LSQksH/y41xi1ma8Csmuz/CdQzXDRpSXll9XQ9/tI1Kn+GWl1ZR7vXVu+z/PPMlEx9ZzlWPfV7nPD1maUlAKXV8NBEExKeQfPlfWHLOC+w16dzveZrPE37BrLh5dJeG7ypuyBursvl6Zx5F5V6mPvcV+0PuPvb6DPkllazLPnrC2znZrdyVF7xxTynVODQR1DB2/JW4bvqQ6RW/5Bt/T37ins+y+F8y1/NnJro+x039v97rM+35r+l/z0KWfpvD+r1VndhV+vwM+dMiJj6yvDF2IaikwsvOw8UNz9hCfLT5INc8/gVzV+wKO33LwUL63f0B+446q4uPqx9bzl8+2AzA7twSVjVQhahUTZoIwkhJ9LDQP4yfVt7OueUPMcd3FafJQebEP8KGhGk87vkHl7lW0ImGu50IFbjzuKaBsxdR6bN+5b6atYeCssrgtG/tm9QCVuzIJfO+D9m4r4DconLySysZ9IdF1eYpq/Qxf91++t29kDEPLA27zYcWb2XNnvAlkOJyL5c//Bnzvtod6a41iUD34FsPhn+M6L9X7KKkwseijQebMqyYmPGfVQz5o/V3X7X7aLArlNF/+5ir66lCbCzGGP40fyPr9+ZHfVvRsGl/AQfyy6qNW7vnKK9mObPJt7Obj9ahT6c0AM7tnc6nW+Gf3kk8xNWMda1mnCuLMe61XOL+GoDN/lNY4j+LRb4hrDW9MCeYW3/z+jp+8/o6JmR0YcmmQ5RWVi+BTLbvbL50zqd1rqPv7z9ocDv/WLyFfyzews77azeZ/WD9Ab7Zm89v3/yGKcNOrTX90Y+38eCH37Lj/2ovu2TTQRI9bkZ+P73BGI5V4I5tXx1VQ357/BursrnnnQ2s+O1YOrcO/xS6aPH7DZV+Pwlx7qhto9Ln57119TcyWL83n/7dWkcthpIKH099+h3/XrGbTX+4OGrbCcfvN7hO8K7NSx6yvj+h//9XPGqVyH8w5JQTWndLpCWCOuy8/zLm3nQ2H/36PAAMLhb7B/Mb73TOKX+Yq8vv5Y+VN3CUVKa73+PthLtZlfC/POh5nKtcnzJItpxQNdJ76/bXSgLHK7+kstrnJ5dtDw5vz7F+XecVV7A/v5SDBWXkhXlWc3G5l6JyLwfyy/jbwm/xG6vq6RfzVvNNdj5PfLIdYww3vZDFDU9/2Shx5xaV47VbW3l9fj+YhzMAABQCSURBVH7/9nrA+jVaVumr9Ws00GJqXbY1fsfh8CWHSG3Yl8/0uVlU+iJvNPCXDzZz+u8+aLABQDhHSyooLKusNT77SAlZO/Moq/Txh/c28pvX14VdfuO+qurGCQ9/BsCu3GL8x9iUzO833PvOBt5anY0vzLJv2te7wHpmRzi7chu3SnL+uv0cLChj8caD9LpzQa2SclNavzefuV/sjNn2o0FLBA3o1SGVLq0T2Z9fxmntk9mVW4IPN6tMH1b5+vC07zJaUcQFrjWc617HOFcW17itXxtHTCrL/f1Z6s/kE18mObSJyT5kzv6QjbPHkxxv/bn/vGBzcNrYBz/hs/93PqP+8nHYZSu8fl76chf3vbux1rRrHv+CTfsLeHftPgBW7qqqm+4xaz4/HtmDey6v6sdp68FC3l6zl9vHnQ7APxdv5ZpB3enWNgl3jV94ZZU+Bv9xMVOGncL/XZ3BrryqZzwcLCgPlnpW/f4iKrx+lm87XOsi8l/e38yL086mdbIHgG+y8/lexxSKyrxsPlDI6D4d6jxmizce5CcvZgFW9Vz3tkm0SY7nueXf8ejH28n63YXBeZ9f/h33vruRRbeN5qUvd9vx110q2J1bwocbDxDnEqaO7BkcP3D2IhI9Ljb/4RI27itgw758zju9Q/Bv069LKzbWeEDS8m2Hg8M1S4nbDhVy4d+Xcfu4Psy4oDfbDhWRU1hOudfHeX06kF9aSX5pJae1T6m2XPaRUp7/fCdgPZ/j1/bfK+BXr1bdb1NW6efdtfu4PLNrcNynW3O48ZmveGjyQK4Y2C3sMajPvqOljPnbUi4d0Jl/Tj6Lcq+Pn/9nFQAD7FLOqt1HOL1z2jGv+/Pth7n+qfp/qPSYNZ81d19U5w2ggSR7w9mn1Vsy+b8Fm+iZnsLkYaeyLvsoXdsk0TrJQ0mFjySPm/i4yH6He31+XCInXAqqjyaCCPx3xkh2Hi7hjtetL8Ad40/nbwu/DU4vIJW3/aN42z8KF356yn76SDZj3as517WOCe4V4IE8k0qBScGPUIEHg1BMIrmmFYdMG74zXSgikWTK6SRHaEMR8VKJwUWJSeCwac1u05GtpjubzSn4iLz6od/dC/n6rgvZnVf7oTnvf3OgzuX6/O79OqfVfGpbzbr555bv5LeXnEF8nIvconIu+scyANokxVNc4eWhJVt5aMlWTmmXRJfWSRSWeXn/1nMBKxEAzPtqD3vyShl/Zqfgej/aXPWQoEOFZdw6bw3fHqz9C3Ftdj6Zsz/kyzvHsml/AVOfs6rzWiXGUVDm5f9d3Jfpo3uRW1zB7rxizujSiqMllcz4zypW7a66fhL44i+6bXQwId711jf86aoBPP3pDv4433rs6Vur9wZLccu3HebSAV0AyCks5+7/rufKs7oxfe7KajG+uXov67Lz+fquC+399uP3m+BJfVRIFVvNJADUW/q68O/W8f7yuzxmABf+/ZPgtAeuzeT+9zdzuKg8WD1y5t0fcNtFfbjEjhusZs69OqSQEh/H+r35vL4yu9Z2fjFvdTARlFX6eG+tVW21evfRsIngcFE50+eu5OEpZ9G5VWKtE9yI+z8C4O01++jVIZUdOVUlu2/sUmC4kkqAMaZax49HiitYvv0wEzK68kqNbl925BTRq0NqrXUs35ZLz/QU+nVtFRxXVunjsaVVpenLH/mMbYeK+OukDO54fR0rf3chaYme4PR/LdsBwORhpzLxkeUkedzB/49hPdoxaXB3Siq8TB3Zk+05RYx98BPe+8WoWlV637/rfXp3TGVU73TuuvQM4tyNX5EjLa0p3pAhQ0xWVlZMtj373Y08u/w7Vv3+Iv6xaAtzV+ziwWszuSyjS9h6+XYp8eQVl3OG7OZc1zpOk0OkSiku/MThw42fFMpoLwV0kVxaS9VJusK4OUIaFcaDiCGZMtpQjEusv1elcbPftGM/7dlr0tnu78oGcxrr/N8jj1a1YomVTq0S8Lhd9O2cxuJNDT/lLeCtn42o976JlmLdvePIuPfDBuebfl4v/vWJdeK4+MzOfLCh7uR8PBbMPLdaieHn53+PRz+2Tmr3Xt6Pe0NKfHOmnMXMeauPaf0PTzmLc77XniF/XBwcl3lKGx67YRDd2iRxtKSC+9/fTHyci6ydR4JJrUf7ZH5xQW8uz+zKvK92c6iwLBhXfWZfcSY/PKcHfr+hzOsj+0gpv397PX+dlMF5f1vK/109gEv6d+b+9zfzsn3y/3zWBTyw8FveXL232rqmjujBj0f24Ly/La21nTsv7cvoPh0oqfA1eBH+gWszuf21tTx4bSa/fq2q1NS9bVKwoUM4n9wxpta2/3ndQM7v25HM+6r/77w4bVi9Jdn6iMhKY8yQsNOimQhE5GLgIcANPG2Mub/G9ATgRWAwkAtcZ4zZWd86Y5kIvD4/ucUVdGqVSEmFl9eysvnhOachIpRV+qjw+WmV6OGxpdvYdrCIX48/nZH2r5tIdOAoHryU4yGXVkD1X0qJlNNNDnOm7KSvaw/d5DCdJY/ukkM3qeoE77BpxV6Tzn7Tnv2mHaUkEE8l8XhJoJJEqcCNH8GPG4MLP4LBjR+X/dllT0uw5y02iRSQQqFJopBkCkmmwCTZ45LJJ4V8Y72OkkoxibXir84Qj5ckykmmnGQps94pJ0nKSaWU9lJAHD4qiMMglBFPoUkmz6RRQiJlePDhQgAvLopNEsUkUk7gV1nL6gbWhZ9USkmllGQpI5EK4vHafw+DFzfFJFJMEkXGeq9sAYX6bm2S2BuFJr1//0EmSzYdYn4Ed+cDXDu4O6+FKdG0JKd3SmPhbaOPa9mYJAIRcQNbgIuAbOBrYIoxZmPIPD8DMowx/ysik4GrjDHX1bfeWCaC42GMYcT9H7G/RlO1xpZCKf1lJwNcO+gl++gmuXSVXDpLHglUUIGH8sDLeOzyiNinfJc9LCHDLny4KDce/LhIkVLSKCVNSkijhFbS8Be7wriptMs+gZOZGz/WpXeDRxrnYngk/EYwgEHslzUM4ccHht348eHChxuvnSYN4McV3KcK4qgwcVTgsYbx4EfsOQmuKzDsx0UcPhKosBIzFbSTQlLl2P9Hyk2clZhNEkUkUUwSxSaRUuIpx0M8XpIpJ01K7LTpx22/AiVTF36S7eTrw00FcXhx48VNpXHjEWsePy6Msaozi+ztFJAcPF7+asfQhTFUGy8YkqQiGI8xgi/kf81rR+Wzj6wXF17iKDDWD49A8is3npD0KLjxEYcfl1SVtK13a7w7ME781cYF9j34WarGA/bf3YXfWLH5cFFJXLDHgUCsofsY+CHltse47MvpVf9TYv94MbznO4c15vvH9Pfu3jaJz/7fBcf8fwL1J4Jo/pwYBmwzxuywg3gZuAIIvep4BXCvPfw68IiIiGlp9VX1EBGW3jEGv9+qG63w+Smv9LNxfwGb9hew83AxV57Vjcszu7L5QAG7cku4//3NfHe4mO91SGHsGZ04XFRO385p3Dz6e+zIKeKfi7fyjn2BFqBv5zT6d+vO6yuT2OgZQFpCHPmllVw1qBv/XhGdewECv17TKCFNSmkjRbSmmFZSTFsKSRHr97rH/nr77S9A1YlU7BNWAiUkUmISKLGHy0w8xSRy2LTGi8s+gVmlk9YU00aKSKGMBCqDX0qP+EimjBTKSRCr1VNVeSDwFSTk60jwPdz0wPIGgl/uwIkDwI3BZy/hwUu8eIOlrngqcWGCX/jQ9QkGlxhKTTxHSQkmjzx/KwpIptAkU0gSJSaRMuLtRGolZw8+UiglRcpIpZQUykiVMtIoIVWskkSqlNJJjpBIBYlSQbnxUEoC+SaFUhKqTm413kv98RSRFDyJevDiES8efFQYDz5jJT63+K0TOSW0kSK6k1PtmFqRgoipNd4glJl4SkmggGREDJ5gYvIFk5M75BUnPtpQdFxJMpxK466WdLzBJG+Pt6dbf2M7Ybr8weMSj9f+bySYQLD/RwRTLTkE5qv+f2f9GDIIm/ynHXMiqK+K6UREs0QwCbjYGPMT+/ONwNnGmBkh86y358m2P2+35zlcY103AzcDnHrqqYN37Qp/Z6nT+P0Gr99E1Pqg3Ovju8PF5BZVkBzvprDMS7uUeFonediZW0xhmZfhvdpzuKiconIv32TnE+cW2qfE07VNEit3HaFfl1Z0bZPEN3vzySuu4PLMrrRO8lDh9ZO1M48e6Sm0TvKQEOfi/fUHmLtiV/BZz9b1kgpuOPtU5n+znz4d0/hqZx4TM7tWS2pgJbbiCi85heW0S45nX0hp6s5L+3JK22Ryisr5ckceBwrKOJBfxv8MP42DBWW0T4nnwUVbwh6DtMQ4Csu8nN4pDb8xbD1URNtkDx3SEtgScpPa9zumsu1Q5E1Pk+Pd3Dy6F1sPFbFpXwE7TqK7uZsLF36SsaoPE6TCKgHYv7oDJ3GfCTmhU/OE7w6kqFjvygl545YRDD6t7XEtG6uqoWuB8TUSwTBjzC9C5tlgzxOaCIYZY3LDrRNaXtWQUko1B/UlgmjeUJYNhN6i1x3YV9c8IhIHtIZj7LdBKaXUCYlmIvga6C0iPUUkHpgMvFNjnneAH9nDk4CPTqbrA0op1RJE7WKxMcYrIjOAhVjNR581xmwQkdlAljHmHeAZYK6IbMMqCUyOVjxKKaXCi2ojZGPMAmBBjXF3hwyXAddGMwallFL1007nlFLK4TQRKKWUw2kiUEoph9NEoJRSDtfieh8VkRzgeG8tTgcONzjXyU+Pg0WPQxU9FpaT+TicZowJ23Vpi0sEJ0JEsuq6s85J9DhY9DhU0WNhcepx0KohpZRyOE0ESinlcE5LBE/GOoBmQo+DRY9DFT0WFkceB0ddI1BKKVWb00oESimlatBEoJRSDueYRCAiF4vItyKyTURmxTqexiYiz4rIIfupb4Fx7URkkYhstd/b2uNFRObYx2KdiAwKWeZH9vxbReRH4bbVnInIKSLysYhsEpENInKrPd5Rx0JEEkXkKxFZax+H++zxPUXkS3ufXrG7iEdEEuzP2+zpPULW9Vt7/LciMj42e3RiRMQtIqtF5D37syOPQ52MMSf9C6sb7O1ALyAeWAv0i3VcjbyPo4FBwPqQcX8FZtnDs4C/2MOXAu9jPbdvOPClPb4dsMN+b2sPt431vh3jcegCDLKH04AtQD+nHQt7f1LtYQ/wpb1/rwKT7fFPALfYwz8DnrCHJwOv2MP97O9LAtDT/h65Y71/x3E8fgX8B3jP/uzI41DXyyklgmHANmPMDmNMBfAycEWMY2pUxphl1H662xXAC/bwC8CVIeNfNJYVQBsR6QKMBxYZY/KMMUeARcDF0Y++8Rhj9htjVtnDhcAmoBsOOxb2/gQevOyxXwa4AHjdHl/zOASOz+vAWBERe/zLxphyY8x3wDas71OLISLdgcuAp+3PggOPQ32ckgi6AXtCPmfb4052nYwx+8E6QQId7fF1HY+T6jjZxfqzsH4NO+5Y2NUha4BDWIlsO3DUGOO1Zwndp+D+2tPzgfacBMcB+CfwG8Bvf26PM49DnZySCCTMOCe3m63reJw0x0lEUoE3gF8aYwrqmzXMuJPiWBhjfMaYgVjPCx8GnBFuNvv9pDwOIjIBOGSMWRk6OsysJ/VxaIhTEkE2cErI5+7AvhjF0pQO2tUc2O+H7PF1HY+T4jiJiAcrCbxkjHnTHu3IYwFgjDkKLMW6RtBGRAJPJgzdp+D+2tNbY1U1tvTjMBKYKCI7saqEL8AqITjtONTLKYnga6C33VIgHusi0DsxjqkpvAMEWrv8CPhvyPgf2i1mhgP5dnXJQmCciLS1W9WMs8e1GHZ97jPAJmPM30MmOepYiEgHEWljDycBF2JdL/kYmGTPVvM4BI7PJOAjY10lfQeYbLem6Qn0Br5qmr04ccaY3xpjuhtjemB97z8yxtyAw45Dg2J9tbqpXlitQ7Zg1ZPeFet4orB/84D9QCXWr5ebsOo2lwBb7fd29rwCPGofi2+AISHrmYZ1IWwb8ONY79dxHIdRWEX2dcAa+3Wp044FkAGsto/DeuBue3wvrBPYNuA1IMEen2h/3mZP7xWyrrvs4/MtcEms9+0EjskYqloNOfY4hHtpFxNKKeVwTqkaUkopVQdNBEop5XCaCJRSyuE0ESillMNpIlBKKYfTRKAcS0SK7PceInJ9I6/7zhqfP2/M9SvVmDQRKAU9gGNKBCLibmCWaonAGDPiGGNSqsloIlAK7gfOFZE1InKb3Vnb30Tka/sZBdMBRGSM/ayD/2DdfIaIvC0iK+0+/2+2x90PJNnre8keFyh9iL3u9SLyjYhcF7LupSLyuohsFpGX7LuklYq6uIZnUeqkNwu43RgzAcA+oecbY4aKSAKwXEQ+tOcdBvQ3VlfEANOMMXl2Nw5fi8gbxphZIjLDWB2+1XQ1MBDIBNLtZZbZ084CzsTqw2Y5Vj85nzX+7ipVnZYIlKptHFb/Q2uwurBuj9W3DMBXIUkAYKaIrAVWYHVK1pv6jQLmGatn0IPAJ8DQkHVnG2P8WF1j9GiUvVGqAVoiUKo2AX5hjKnWyZyIjAGKa3y+EDjHGFMiIkux+qppaN11KQ8Z9qHfT9VEtESgFBRiPdYyYCFwi92dNSLSR0RSwizXGjhiJ4G+WN08B1QGlq9hGXCdfR2iA9YjRk+eXixVi6S/OJSyeuj02lU8zwMPYVXLrLIv2OZQ9SjDUB8A/ysi67B6pFwRMu1JYJ2IrDJWt8cBbwHnYD3/1gC/McYcsBOJUjGhvY8qpZTDadWQUko5nCYCpZRyOE0ESinlcJoIlFLK4TQRKKWUw2kiUEoph9NEoJRSDvf/AQGDrpinnd3lAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_list, label='Minibatch cost')\n",
    "plt.plot(np.convolve(cost_list, \n",
    "                     np.ones(200,)/200, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3yV9fXA8c9JAoQ9QsIIe8jekaHIVFxUASfWukEp1tHWX9VaR+uoVm2tWhVFtJXiRosDBIwMkRGUvRI2hBFWIISEjPP743vBAAnckHvvc5Oc9+uV102e+4xzr3jPfb7riKpijDHG+CvC6wCMMcaULpY4jDHGFIslDmOMMcViicMYY0yxWOIwxhhTLFFeBxAKdevW1WbNmnkdhjHGlCqLFy/eo6qxJ28vF4mjWbNmJCUleR2GMcaUKiKyubDt1lRljDGmWCxxGGOMKRZLHMYYY4rFEocxxphiscRhjDGmWCxxGGOMKRZLHMYYY4rFEocxxpQxBzKP8u2aXTw3dQ070o8E/PzlYgKgMcaUVarKpr2ZJG3ax+LN+1m8eT/JuzMAiIoQEprVpkHNygG9piUOY4wpRbJz81ixPZ2kTS5J/LhlP3syjgJQIzqKHk1rM6xbPD2a1qZLo1pUrhgZ8BgscRhjTBjbm5HNj1sOkLR5H4s37WfZ9nSO5uYD0CymCv3PiaNH09okNKtNq9hqRERI0GOyxGGMKfWycvL49MftvDNvIweP5NKodmXfT5UTHhvUiqZSVOC/gQeKqrI+7TCLN+87fkexYc9hACpECh3ja3Jzn6b0aFqHHk1rE1u9kidxWuIwxpRa6Zk5vLdgMxO+38SejGw6xdfk/Fa12H4gk6TN+5mybAd5+Xp8fxGoVz06bBJLVk4ey7alH7+bWLxlPwcycwCoXaUCPZrW5pqExiQ0q02n+JpEVwiPpGeJwxhT6mw/cITxczby/qItZB7No985sdzVrwV9WsYg8nNTTW5ePjsPZrFt/xHfT+bxx6ISS1z1SgUSyonJpWEJE0vaoWwWb3ad2Emb97Niezo5ee76LWKrMqR9PXo0rU2PpnVoGVv1hNcSTixxGGNKjVWpBxk3ez1Tlu1AgF90acioC1rQvmGNQvePiozwffBXKfT5kxPL9gLJ5cct+/miBIklP19JScsgadN+d0exeT+b92YCUDEqgs7xNbmtb3MSfM1OdapWDOybFUSWOIwxYU1V+WH9Xl6fvYHZ69KoUjGSW85rxm19mxNfq2TDTP1JLLsOZbNtX+Ypdy2FJRaAejUqUb9GNBv3HOZgVi4AMVUr0qNpbX7Zqwk9mtahY3yNsO5rORNLHKbUyc3LJ0IkJKNHjHdy8/L5esVO3pi9nhXbD1K3WiUeuLgNN/ZqSs0qFUISQ1RkBPG1KhNfqzK9ioixsMSyIz2Lyzo18I12qkOzmCph2+x0NixxmFLjaG4+42av55XEFLJy8qlWKcr9RLvH6tHux22vQLXoKKoXeP7kv6tXqkDVSpFERdoCCuEk82guHyVt4625G9i67wgt6lblmRGdGN4tPmw6h485U2Ipq4KaOETkXmAUIMCbqvoPEekCvA5UAzYBv1TVgycd1wb4oMCmFsCjvuMf950zzffcw6r6VTBfh/He4s37eOjT5azblcHFHerRpn4NMrJyycjOISM7l0NZuWRk57IzPev47xnZuX6du3KFyFOTzEmJpnp0hePJ6djzdatXomVstSC/8vJjb0Y2//5hM//+YRP7M3Po3qQWj1zenova1bO7yzATtMQhIh1xH/A9gaPAVBH5EngL+L2qzhKR24AHgD8VPFZV1wJdfeeJBLYDkwvs8ndVfT5YsZvwkX4kh+emrmHigi00rBnNWzclcGH7en4dm5+vHD7qSyJZuRzyPZ7698/J51jS2ZuR6dvmnjupGfu481vF8PshbejWpHYAX3X5snnvYd6as5EPk7aSnZvPhe3qcVf/FiQ0q+N1aKYIwbzjaAfMV9VMABGZBQwH2gCzfftMB6ZxUuI4yWBgvaoWWjTdlE2qylfLd/L4lJXszcjmtvOb87sh51C1kv//ZCMihOrRFageXQFqliyWIzl5pySflanpvDFrA8P/NY8L28XxuyFtaNeg8NE95lRLtx5g3OwNfL1iB1EREQzvFs+ofs1pFVfd69DMGQQzcawAnhKRGOAIcBmQ5Nt+BfA5cA3Q+AznuR6YdNK2u0XkJt/5fqeq+wMZuPHWtv2ZPPr5Sr5ds5sODWvw9s3n0qlRCT75S0hEqFIxiioVo4grsP38VnX5Za+mTPh+I2/M3sClL81haOcG3H/ROdaEVQRV5bt1aYybtYEfNuylenQUd/Zvya3nNSOuRrTX4Rk/iWoR9+CBOLnI7cBYIANYhUsgbwD/BGKA/wH3qGpMEcdXBFKBDqq6y7etHrAHUOAvQANVva2QY0cDowGaNGnSY/Nmu2EJd7l5+bwzbxMvTl+HKvxuyDnccl6zUtF5nZ6Zw7g565nw/SaycvK4qnsj7r2wdZHDPMubo7n5TFmayptzNrBm5yEa1IzmtvObc33Pxu6O0IQlEVmsqgmnbA9m4jgpgKeBbar6rwLbzgHeU9WeRRxzJTBWVYcU8Xwz4AtV7Xi6ayckJGhSUtLZhm5CYPm2dB6avIwV2w8ysE0sf76yI43rlL4P3T0Z2fwrcT3vLdiMqnJDzyaMHdiq3H6bzsjO5f2FWxg/dyM70rNoU686o/u14BddGlIxKvy/EJR3RSWOYI+qilPV3SLSBBgB9CmwLQJ4BDfCqigjOamZSkQaqOoO35/DcU1fppQ6nJ3Li9PXMeH7jcRUq8QrN3Tj8k4NSu2Y97rVKvHoL9pzxwXNefnbFCYu2MIHSVu5uU8z7urfktqlaHZwSew+mMWEeZt4b/5mDmXl0rtFHZ4e0YkB58SW2v+25mfBbqqag2uSygF+q6ozfUN0x/p2+RR4SFVVRBoCb6nqZb5jqwBbgRaqml7gnP/BjbhS3HDeOwskkkLZHUd4mrl6F49+vpLtB45wQ68m/OGSttSsXLaaLTbvPcw/ZiTz2ZLtVK0Yxe19m3PHBc3LbPNMyu4M3py9gck/bSc3P59LOzZgdL8WdGlcy+vQzFnwvKnKS5Y4wsvug1k8PmUlXy3fSeu4ajwzolOZH3q5btchXvxmHVNX7qRWlQqM6d+Sm/o0C0qRnVDLysnjhw17+e+CLUxftYtKURFck9CIO/q2oFndql6HZ0rAEoclDs/l5ysTF27hua/XkJ2Xzz2DWjG6X8ty1da9fFs6z3+zllnr0oitXonfDGrFdec2LnXrFm3dl8l3a3eTuDaNeev3kJWTT60qFbipTzNu7tOUmGre1IkwgWWJwxKHp9buPMTDk5ezePN+zmsZw1PDO9G8HH8bXbhxH89/s5aFG/cRX6sy9w5uzYju8WE7guxobj5Jm/fx3do0EtfsPl7TukmdKgxqG8eANrH0bhETdkuCmJKxxGGJwxNZOXm8/G0yb8zaQPXoKB65vD0jusdbByluTsOc5D08/81alm1Lp0Xdqtx30TkM7dQgLJbY2H0wi+/WpvHtmt3MTdlDRnYuFSKFXs1jGNAmlkFt42heN3xrRpiSs8RhiSPkvk/Zwx8nL2fT3kxGdI/nkcvbl6qaA6GiqnyzahcvfrOOtbsO0bZ+dX4/pA2D28WF9EM5L19ZsvUAiWt2k7h2NytT3RJyDWpGM6BNHAPbxHJ+q7rFmr1vSjdLHJY4QmZvRjZPfbmaT3/aTrOYKjw1vBPnt6rrdVhhLy9f+WJZKn+fvo5NezPp2rgWD1zcJqjv3b7DR5m9Lo3EtbuZtS6NA5k5REYIPZrUZkDbWAa2iaNt/ep2V1FOWeKwxBF0qsonP27nqS9XcSgrl7v6t+TuQa2s3buYcvLy+WTxNv45M5nU9Cz6tIjh9xefQ4+mJR95lp+vrNpx8PhdxU9bD6DqCg319zU/XdAqNmT1Lkx4s8RhiSOoNu45zB8nL2fe+r30aFqbp4d3ok19W6yuJLJy8pi0cAuvJqawJ+Mog9rG8bsh59ChYfHW7TqYlcPc5D0krtnNd+vSSDuUjQh0blSLgW3cXUWn+Jph0a9iwoslDkscQXE0N583Zq3n5cQUKkVF8IdL2nJDzyb2IRRAmUdzeWfeJt6YtYH0Izlc3sktpNgqrvCFFFWV5N0Zx+8qkjbtJzdfqREdRb9zXKLo3yaWujZk1pyBJQ5LHAGXtMkVV0rencHlnRrw2C/al9s1mUIh/UgO4+dsYPzcjRzJyWN4t0bcd2FrGtepQubRXH5Yv5fEtbtJXJPG9gNHAGhbvzoD28YxqG0c3RrXCtvhviY8WeKwxBEw6Udy+OvXa5i0cAvxtSrzl2EdGNTWv+JKpuT2ZmTz+qz1/PuHzeTlK10b12LZ9nSO5uZTpWIkfVvVZaBvbkWDmpW9DteUYpY4SnHi2H0wi+TdGURGCFER4nuMINL3+wnbI33bxLdP5M/PRYqUqAlJVfli2Q6emLKKfYezufX85vz2ouIVVzKBszM9i1cSk/lpywF6t4hhYJs4zm1eu9TNQjfhy5PVcU3J5eUr1785nw1phwNyPhFOST5RBRLQ6ZJTVk4ea3YeomN8DSbc4m1xJQP1a0bz5LBOXodhyiFLHGHu6xU72JB2mIcva0vHhjXJUyU3X8nL8z3mK7n5+eSrkpt37O+Cj/nk5UNefv5J23/+yc3Pd4++40+9hju2WqUo/jS0MTf3aWpt5caUY5Y4wpiq8mrielrEVuX2vi2ItJFKxpgwYF8bw1ji2t2s3nGQMf1bWtIwxoQNSxxhSlV55dsU4mtVZli3eK/DMcaY44KaOETkXhFZISIrReQ+37YuIvKDiCwXkSkiUqOIYzf59lkiIkkFttcRkekikux7rB3M1+CV+Rv28eOWA9zZvwUVrD/BGBNGgvaJJCIdgVFAT6ALMFREWgNvAQ+qaidgMvDAaU4zUFW7njQc7EFgpqq2Bmb6/i5zXk1MoW61Slyb0NjrUIwx5gTB/CrbDpivqpmqmgvMAoYDbYDZvn2mA1cV87xXAu/6fn8XGBaAWMPKkq0HmJuyhzsuaG4LBBpjwk4wE8cKoJ+IxIhIFeAyoLFv+xW+fa7xbSuMAt+IyGIRGV1gez1V3QHge4wr7GARGS0iSSKSlJaWFoCXEzqvJqZQs3IFbuzd1OtQjDHmFEFLHKq6GngWd1cxFVgK5AK3AWNFZDFQHThaxCnOV9XuwKW+/fsV8/rjVDVBVRNiY2PP9mWE3JqdB5m+ahe3nNeMajYj2xgThoLa66qq41W1u6r2A/YByaq6RlWHqGoPYBKwvohjU32Pu3F9IT19T+0SkQYAvsfdwXwNofbad+upUjGSW85r5nUoxhhTqGCPqorzPTYBRgCTCmyLAB4BXi/kuKoiUv3Y78AQXBMXwP+Am32/3wx8HszXEEqb9hxmytJUbuzdlNpWYtUYE6aCPc7zExFZBUwBxqrqfmCkiKwD1gCpwAQAEWkoIl/5jqsHzBWRpcBC4EtVnep77q/ARSKSDFzk+7tMeGP2eqIiI7ijb3OvQzHGmCIFtRFdVS8oZNtLwEuFbE/FdaCjqhtwQ3gLO+deYHBgI/XejvQjfLx4G9ed29hqWhhjwprNLAsT42ZvIF/hzn4tvQ7FGGNOyxJHGNibkc2khVu4smtDGtep4nU4xhhzWpY4wsDb328kOzefXw+wuw1jTPizxOGxg1k5/HveZi7pUJ9WcdW9DscYY87IEofH/vPDZg5l5zJ2YCuvQzHGGL9Y4vDQkaN5jJ+7kQFtYukYb2VYjTGlgyUOD01auIV9h4/a3YYxplSxxOGR7Nw8xs3eQM/mdTi3WR2vwzHGGL9Z4vDI5B+3s/Nglt1tGGNKHUscHsjNy+e1WevpFF+Tfq3reh2OMcYUiyUOD3y5fAeb92YydmArRMTrcIwxplgscYRYfr7yr8T1tI6rxpD29bwOxxhjis0SR4jNWL2LtbsO8euBLYmIsLsNY0zpY4kjhFSVV79bT+M6lflF54Zeh2OMMWfFEkcIfZ+yl6VbD3BX/5ZERdpbb4wpnezTK4ReTUwhrnolrureyOtQjDHmrAW7dOy9IrJCRFaKyH2+bV1E5AcRWS4iU0SkRiHHNRaRRBFZ7Tv23gLPPS4i20Vkie/nsmC+hkBZvHk/P2zYy+h+LYiuEOl1OMYYc9aCljhEpCMwCuiJq+Y3VERaA28BD6pqJ2Ay8EAhh+cCv1PVdkBvYKyItC/w/N9Vtavv56tCjg87/0pMoXaVCozs2cTrUIwxpkSCecfRDpivqpmqmgvMAoYDbYDZvn2mA1edfKCq7lDVH32/HwJWA/FBjDWoVqUeZOaa3dx6fnOqVgpqtV5jjAm6YCaOFUA/EYkRkSq4euKNfduv8O1zjW9bkUSkGdANWFBg890iskxE3haR2kUcN1pEkkQkKS0trWSvpIRe/S6FapWiuLlPM0/jMMaYQAha4lDV1cCzuLuKqcBSXBPUbbimp8VAdeBoUecQkWrAJ8B9qnrQt/k1oCXQFdgBvFDE9cepaoKqJsTGxgbmRZ2FDWkZfLV8Bzf2bkrNKhU8i8MYYwIlqJ3jqjpeVburaj9gH5CsqmtUdYiq9gAmAesLO1ZEKuCSxkRV/bTAOXepap6q5gNv4vpQwtZr362nYmQEt/dt7nUoxhgTEMEeVRXne2wCjAAmFdgWATwCvF7IcQKMB1ar6osnPdegwJ/DcU1fYWn7gSNM/mk7I3s2IbZ6Ja/DMcaYgAj2PI5PRGQVMAUYq6r7gZEisg5YA6QCEwBEpKGIHBshdT7wK2BQIcNun/MN5V0GDATuD/JrOGvjZrmbqVH9WngciTHGBE5Qh/io6gWFbHsJeKmQ7am4DnRUdS5Q6EJOqvqrAIcZFGmHsnl/0VZGdI8nvlZlr8MxxpiAsZnjQTJ+7kZy8vK5q39Lr0MxxpiAssQRBOmZObw3fzOXdWpAi9hqXodjjDEBZYkjCN79YRMZ2blWFtYYUyZZ4giww9m5vP39Rga3jaNdg1OW4TLGmFLPEkeATVq4hQOZOYwdZHcbxpiyyRJHAGXl5DFu9gb6tIihe5NCV0IxxphSzxJHAH28eBu7D2Vzt91tGGPKMEscAZKbl8/rs9bTtXEtzmsZ43U4xhgTNJY4AuR/S1PZtv8IYwe2wq2YYsq0jDRY/jHk53kdiSloy3zYk+J1FGWeFYcIgPx85V/fradt/eoMbhvndTgmmPJyYNFbkPgMZKfDkf3Qc5TXURmAvFz473VQuxmM/g7sC1zQnPGOQ0TuLqrmhXG+WbWTlN0ZjBnQkogI+8daZm2cDa9fAFMfhEY9oFFPSHwajhzwOjIDsH0xZB2AHUtgW5LX0ZRp/jRV1QcWiciHInKJWDvMCVSVVxPX0yymCkM7N/Q6HBMMB7bChzfDu7+AnMNw3US48VO4/AV3xzH7b15HaABSpoNEQMXqsPANr6Mp086YOFT1EaA1bpnzW4BkEXlaRGwRJmB28h6Wb09nzICWRNrdRtmSkwWz/gavnAvrpsKAh2HsQmg31DWDNOgM3X4JC96AvYWWlTGhlDIDGp0L3W6ElZ/BoV1eR1Rm+dU5rqoK7PT95AK1gY9F5LkgxlYqvPptCg1qRjO8WyOvQzGBogprvoRXe0Lik9D6Irh7EQz4A1Q4aaXjQX+CyIow4zFvYjVORhqk/gStLoRz74D8HFj8jtdRlVn+9HHc4yvz+hzwPdBJVccAPYCrghxfWFu4cR8LN+1jdL8WVIyyAWplwp5keO8qeP8GiIqGmz6H6/4DtZoUvn/1+tD3flg9BTbNDW2s5mfrv3WPrQZD3VYugSS97QYzmIDz59OuLjBCVS9W1Y9UNQfAV7p1aFCjC3OvJqYQU7Ui159bxIeKKT2yD8E3f4J/9YFti+DiZ2DM99BiwJmPPe9uqNEIpj0M+fnBjtQUJmUGVImBBt3c3z3vhIydsPp/3sZVRvmTOL7C1QsHQESqi0gvAFVdfboDReReEVkhIitF5D7fti4i8oOvit8UESl0JUBfR/xaEUkRkQcLbG8uIgtEJFlEPhCRiv680EBbsT2dWevSuK1vcypXjPQiBBMIqrD0A3g5Aeb9EzpfB79ZDH1+DZEV/DtHhcpw4eOwYyksnRTMaE1h8vNh/UxoORgifB9prS6E2s1hwThvYyuj/EkcrwEZBf4+7Nt2WiLSERgF9AS6AENFpDXwFvCgqnYCJgMPFHJsJPAqcCnQHldutr3v6WeBv6tqa2A/cLsfryHgXk1MoXp0FL/q09SLy5tA2LEU3r4EJo+GGg3hjpkw7FWodhZzcTpdDfEJMPPPcPRw4GM1RdvxE2TudX1Rx0REuPk1W+e7/84moPxJHOLrHAeON1H5M3GwHTBfVTNVNReYBQwH2gCzfftMp/B+kp5AiqpuUNWjwPvAlb6hwIOAj337vQsM8yOWgErZfYipK3dyc59m1Ij281upCR+Z++CL++GN/rA3Ba542SWNRglnf04RuPhp1zzy/SmVkU0wpcwEBFoOOnF7119ChSqw0O46As2fxLHB10FewfdzL7DBj+NWAP1EJEZEquDqiTf2bb/Ct881vm0niwe2Fvh7m29bDHDAl4gKbj+FiIwWkSQRSUpLS/MjXP/967v1REdFclvf5gE9rwmy/Dw36/vl7rD4Xeh1p2uW6n7Tz00cJdGkF3QYAd//E9K3l/x8xj/J06FhV6ha98TtlWu5psflH7svCyZg/Pm/5S7gPGA77oO6FzD6TAf5+j+exd1VTAWW4oby3gaM9Y3Uqg4cLeTwwiZE6Gm2F3b9caqaoKoJsbGxZwrXb1v3ZfL5klRG9mxCnaqedK+Ys7H5BxjXH778HdTrCHfNgUufdR8ugXTRE6D5MPOJwJ7XFC5zH2xPglYXFf58z1GQmwU//ju0cZVx/kwA3K2q16tqnKrWU9UbVHW3PydX1fGq2l1V++E62JNVdY2qDlHVHsAkoLCZU9s48U6kEZAK7AFqiUjUSdtD5o3Z64kUYXS/FqG8rDlbB1Phk1Ew4RL3IXP1BLh5CtTrEJzr1WoCfcbCsg/cEhgmuDYkukTd6sLCn6/XAZpdAIvG24KUAeTPPI5oERkrIv8SkbeP/fhzchGJ8z02AUYAkwpsiwAeAV4v5NBFQGvfCKqKwPXA/3x9LYnA1b79bgY+9yeWQNh9MIsPk7ZxVY9G1K8ZHarLmrORmw1z/+5GS636DC74vZvE13FE8Be/63s/VI2FqQ+7UVsmeFJmQnRNiO9R9D49R0H6Fjf73wSEP01V/8GtV3UxroO7EXDIz/N/IiKrgCnAWFXdjxshtQ5Yg7tbmAAgIg1F5CsAXx/G3cA0YDXwoaqu9J3zD8BvRSQF1+cx3s9YSuytuRvJzcvnrv52txHWkqe7+RgzHofm/WDsAhj8J6hYNTTXj64Bgx5xI3pWfRaaa5ZHqm7+RstBEHma8TptLnfzbBbY+lWB4s/oqFaqeo2IXKmq74rIf3Ef6GekqhcUsu0l4JRhJ6qaiutAP/b3V7g5JCfvtwE36iqk9h8+ynvzN3NFl4Y0jQnRB5Apnn0b3Lf8dV9DnZbwy49PHKIZSt1+BQvfhOmPwjmXQgW7Qw24ncshY1fRzVTHREbBube5odJpayG2TWjiK8P8ueM4Nmf/gG9uRk2gWdAiClMT5m0i82geYwZYWdiwc/QwzPwLvNobNs2BC5+AX8/3LmkARETCkCfhwBZYUFhrrCmxlBnu8UyJA6D7zRBZyYbmBog/iWOcrx7HI8D/gFW40VLlRkZ2Lu98v5Eh7evRpn51r8Mxx6jCik/c6rVznof2V8LdSdD3PogKgxFvLQfCOZfA7OfdInwmsFJmQr1Obr2wM6laFzpeBUsmQVZ68GMr406bOHwd2AdVdb+qzlbVFr7RVeWqsfC9+Zs5mJXL2IF2txE2dq109TE+vg2q1IFbp8JVb0KNBl5HdqIhT0LuEUh8yutIypasg64PqdVg/4/pOcrVU1liy8KU1Gn7OFQ1X0TuBj4MUTxhJysnj7fmbOSC1nXp0jjAY/5Lm+Ufw+7TLk8WGod2ujWhomu4Yko9bnVNQ+Gobmu3zPfCce6DK1jDgMubjbMgP7d4zZHx3V29jkVvQs/RgZn0WU750zk+XUR+D3yAW6cKAFUtF1MxP0zayp6MbMYO7OZ1KN5aNB6+/K2rsFboPMwQioiCHje7WhhV6ngbiz/6/wGWvg/T/gi/mmy1sAMhebqr9Ne4V/GO6zkaPh0FG771r2/EFMqfxHGb73FsgW0KlPkxqTl5+bwxawM9mtamV/NS8AEVLCkz4asHoPXFMHJS+H67D1dV6rjkMe0h94F3zhCvIyrdVN2/yRb9/V/B+Jj2w1wCX/imJY4S8GfmePNCfsp80gD47KftbD9whLsHtqLcllrfvQY+ugXi2sHV4y1pnK1z73BDhL/5oxUXKqm0tXBw29l98EdVhIRbYd002Lcx8LGVE/7MHL+psJ9QBOe1bfuP0KVRTQa0CdxaV6XK4T3w32tdJbyR70MlG1F21qIquo7yPesgaYLX0ZRuKdPd49neMRzrE1v0VuBiKmf86R06t8DPBcDj/Ly6bZl2/0Xn8MmY88rn3UZOliufmrHLJY1ahS1ibIqlzaVuJvt3z8CR/V5HU3qlzIDYtmf/b7JGA2h3Bfz0H6udcpb8aar6TYGfUUA3IAwGyYdGVGQ5HHmhCp+Pha0LYPgb0Og06wAZ/4nAkKdc0pj9vNfRlE5HD8PmeSXvn+g52s3nWFZuB4yWyNl8KmYCrQMdiAkjs56FFR/D4EehQ8jrZJVtDTpDtxvdukl7C1sY2pzWxjmQd7R48zcK06Q31O/kOsltIcpi86ePY4qI/M/38wWwlhCuSGtCbPnHrimlyw3Q97deR1M2DfoTRFVy61iZ4kmZ4ar6NTmvZOcRcXcdu1fC5u8DE1s54s9w3IL31LnAZlXdFqR4jJe2LIDPfg1Nz4dfvGTzDYKlej239Pq3f3HfoJufshaoKYyq6xhvdkFgFo3sdI1L3gvHQbO+JT9fOeJPU9UWYIGqzlLV74G9ItIsqOX5tZIAACAASURBVFGZ0Nu/yXWG14yH694Lj7WeyrI+Y6FmY5j2sBUY8te+De7faaAWr6xQ2a1ivPoLSC+D34WPHobP7w7Ka/MncXwE5Bf4O8+3zZQVWenw3+sgPwdu+LB0zMYu7SpUhgsfh53L3PIp5syOr4Zbwv6Ngs69A9CyOUR6+mPw03su2QaYP4kjSlWP1wX3/W5fR8uKvFw3wW9virvTqGvjHkKm41Vu7aSZf4bsDK+jCX/J06FOC/cTKLWbunopi99xQ9DLivWJbk2u3mOC0gznT+JIE5Hj8zZE5Epc7e8zEpF7RWSFiKwUkft827qKyHwRWSIiSSJySlEmERnoe/7YT5aIDPM9946IbCzwXFf/Xqo5hSp8/X+w/lsY+nc3x8CEjghc/IybK/P9KbXNTEE5R2DTXGgVhBorPUdB5h5YOTnw5/ZCVrproopp7UZGBoE/ieMu4GER2SIiW3ClW+8800G+ok+jcNX6ugBDRaQ18BzwhKp2BR71/X0CVU1U1a6+fQbhhgB/U2CXB449r6pL/HgNpjALXoek8XDePdC9XCwGEH4an+vuPOa9XDbb2QNl8zy3PH0w1pdqMQDqnlN2ijxNexgOpcLw112TaBD4MwFwvar2BtoDHVT1PFVN8ePc7YD5qprpqyE+CxiOWyCxhm+fmri646dzNfC1qmb6cU3jr3XT3D+wtkNdxTzjnQsfBxRm2H+HIqXMcBX8gjH66djQ3NQfYVtS4M8fSmunun6N8++DRglBu4w/8zieFpFaqpqhqodEpLaIPOnHuVcA/UQkRkSq4OqJNwbuA/4mIltxQ30fOsN5rgdO7j18SkSWicjfRaRSEXGP9jWFJaWlWfW1E+xc4Qog1e8EI8ZZXQKv1WriRlkt/xC2LfY6mvCUMgOanQ8VqwTn/F2ud8u0l+a7jsx9MOUeiOsAAx4M6qX8+cS4VFUPHPtDVffjksBpqepqXInZ6cBUYCluHsgY4H5VbQzcD4wv6hwi0gDoBEwrsPkhoC1u7aw6uKazwq4/TlUTVDUhNracLlJYmEM73QiqSjVg5AdQsarXERlw8zqqxrml120m84n2b3aLQwZzGfRK1aHrDbDiU8jYHbzrBNNXD0DmXtdEFVXo9+mA8SdxRBb8Vi8ilQG/olLV8araXVX7AfuAZOBm4FPfLh/h+kCKci0wWVWPr0OtqjvUyQYmnOF4U9DRTJg0Eo7sgxveD78yq+VZpeow6BG3PlhZ6aQNlOPDcIPQMV5Qz1FuSPrid4N7nWBYOdktE9T/D25ZmyDzJ3G8B8wUkdtF5HbcHYRf76yIxPkemwAjcE1OqUB/3y6DcMmkKCM5qZnKdxeCuCVrh+GaxMyZ5OfDZ3dB6k9w1Xho0MXriMzJut0I9TrBjMfK1tDQkkqZCTWbBH+oeN3W0HKQGzBSmmqmZOyGL34LDbu5O9cQ8Kdz/DngSVxnd3tcs1NTP8//iYisAqYAY33NXKOAF0RkKfA0MBpARBJE5PgC+b7Z6Y1xneoFTRSR5cByoK4vNnMm3/4FVn0OQ/4Cbc/Y0mi8EBEJFz8FB7bAgte8jiY85B519cVbDQ7NEjg974RDO2D1lOBfKxBUYcp9bpb4sNeLXxHxLPmzVhXATtzs8WuBjcAn/hykqqcswqOqc4FT1ulW1STgjgJ/bwLiC9lvkJ8xm2N+mghzX4TuN0Ofu72OxpxOi/7Q5jKY/QJ0/SVUi/M6Im9tnQ9HMwK3zMiZtL4Iajdzq+Z2HBGaa5bEsg9g7ZeuSFhc25Bdtsg7DhE5R0QeFZHVwCvAVkBUdaCqvhKyCE3JbJoLU+6F5v3h8hds4cLS4KK/uDkLiU95HYn3UmZARFToJqdGRLplSLbMg53LQ3PNs5W+Hb76P2jcG3r/OqSXPl1T1RpgMPALVe2rqi/j1qkypcXe9fDBjVCnOVz7bshuY00J1W0F546CH/8Nu1Z6HY23kmdAkz6hLVvc7UaIqhzeQ3NV4X93u878Yf9yCS+ETpc4rsI1USWKyJsiMhiwr6ulReY+Vy8cgRs+gMq1vY7IFEf//3NDpqc9XH6H5x5MdfUygjkMtzCVa0Pna2HZR+7/o3C0+B23VNBFf4aYliG/fJGJQ1Unq+p1uDkT3+HmXNQTkddEZEiI4jNnI/cofHiT62S9/r+BXRTOhEaVOjDgIdjwHSR/c8bdy6Tjw3BDnDjAzSTPPeJmYYebfRth2h9d83PC7Z6E4M+oqsOqOlFVhwKNgCVAcKclmrOnCl/eD5vmwBWvQNM+Xkdkzta5t0NMK/chUZqGhwZKygyo3gDqdQj9tet3dAXNFr0ZXvVS8vPdAoYSAVe+6tmqD8W6qqruU9U3bGRTGPv+Jfctqd8D0OU6r6MxJRFZwY2W2ZsMSW97HU1o5eXC+u9CNwy3MD1Hu7v2cLrjW/gGbJ4Ll/4VajX2LAxbpKgsWT0FZjwOHYbDgIe9jsYEwjmXuCaJ756BI/u9jiZ0ti2C7HRvmqmOaTsUasTDgje8i6GgPcnu/+9zLnFDtT1kiaOsSP0JPhkF8T1g2Gu2cGFZIQIXPw1HDsCsv3kdTeikzACJhBYDvYshMgoSboUNiZC2zrs4wN2BTb4LoqLhFy95PqzePl3KgvTt8N/roWpdGDkpaGvwG4/U7wjdf+WGh+5d73U0oZEyw1VHrFzL2zi63wKRFV1fh5fm/RO2J7m5WNXrexsLljhKv+wMmHSdW3Lghg9spnFZNfARt+LpN3/yOpLgy9gNO5Z420x1TLVY6DAClvwXsg56E8OulZD4NLS/0hX9CgOWOEqz/Dz4dJT7h3XNO96MPjGhUb0eXPBbt7zExtleRxNc6791j63DIHEA9Brtlj1Z+n7or517FCbf6e68Ln/R8yaqYyxxlGbTH4W1X8Elz4bP/2QmeHqPdavETns4vIaIBlrydKgaC/XDZAXn+B7uZ+E4Nxw2lOY875Y+GfoP1xQdJixxlFZJE+CHV9yQwV6jvY7GhEKFaLjwMfdBsuS/XkcTHPl57o6j5eDwGuDR8043LHrjd6G75vYfYfbz0GUktBsauuv6IYz+yxi/rU+EL3/nCttc/IzX0ZhQ6ngVNOrplsnPPuR1NIGXusQVGguH/o2COgxzd0ELQrR+VU6WG0VVrR5c8tfQXLMYLHGUNmlr4cObIbYNXP22GzJoyg8RuOQZyNjlJnuWNSnTAXEFlcJJVCXocQusm+qW/Ai2xCdhz1q48mXvR5YVwhJHaXJ4r1u4MKqiG0EVXcPriIwXGiVAp2tg3stwYKvX0QRWygyI7w5VY7yO5FQJt7mlPpLGB/c6W+bDvFegx63hd+flE9TEISL3isgKEVkpIvf5tnUVkfkiskREkkSk0JrhIpLn22eJiPyvwPbmIrJARJJF5AMRqRjM1xA2crPhg1/CwR1w/SSo1cTriIyXBj/mHmc+4W0cgZS5D7YvDtsPS2o0hHa/gB//A0czg3ONo4ddE1Wtxq5aZ5gKWuIQkY64MrE9gS7AUBFpDTwHPKGqXYFHfX8X5oiqdvX9XFFg+7PA31W1NbAf8GZ5yFBShf/9Brb8AMNfg8bneh2R8Vqtxq6a4/KPYFuS19EExvpvQfPDN3EA9LoTsg649z0Ypj8G+ze61R9CWYOkmIJ5x9EOmK+qmaqai6sdPhxQ4FgbS00g1d8TiogAg4CPfZveBYYFLOJwNftvrkTkwEfCZgKQCQN973edp1MfKhs1O1JmQnQtN/Q1XDXpA/U6uqG5gX7PN3znZqj3/jU06xvYcwdYMBPHCqCfiMSISBXgMqAxcB/wNxHZCjwPPFTE8dG+pqz5InIsOcQAB3yJCGAbhdQlBxCR0b7jk9LS0gL1mkJvxSeuhGjn66Df772OxoSTStVg0J9g20JY+anX0ZRMfr7r32g5KOTV7IpFxA2B37XCtQAESla6Wy49phUMfjRw5w2SoCUOVV2Na1aaDkwFlgK5wBjgflVtjCsOVVRPUxNVTQBuAP4hIi0pvAJhoWlfVcepaoKqJsTGxpbsxXhl6yKYPMZ9y7ni5bCZNWrCSNcboH4nmP64G8JZWu1aDod3Q+uLvI7kzDpd4+6MArlq7rSH4eB2GPZ6qVhrLqid46o6XlW7q2o/YB+QDNwMHPt69BGuD6SwY1N9jxtwFQi7AXuAWiJybAxqI4rR1FWq7N8M74+EGg3guoluOKAxJ4uIdKvnpm+B+a96Hc3ZO1btL9yG4RamYhW36OTqKa68bUmtm+Zq6Jx/X6npvwz2qKo432MTYAQwCfdB39+3yyBcMjn5uNoiUsn3e13gfGCVqiqQCFzt2/Vm4PNgvgZPZB2ESde7dWpu+Cg8hyaa8NG8H7S5HOa86BYILI2SZ7g7pzBY+dUvCbe7jvySFtjK3OcGvsR1gAGlp7BqsOdxfCIiq4ApwFhV3Y8bafWCiCwFngZGA4hIgoi85TuuHZDk2ycR+KuqrvI99wfgtyKSguvzCPKg6hDLy4WPb3UT/a59F2LP8ToiUxoM+QvkZsG3T3odSfFlpcPWBW4lhNKiTnNXUGnxO26o/Nn66gHI3AvDXy9VrQpBnXasqhcUsm0ucMqwCVVNAu7w/T4P6FTEOTdQRPNWmTDtIXfbPvQf0NLDIjamdIlp6TptF7zuHut39Doi/22YBZoX3sNwC9NrNPzna1j52dmVaV75Gaz4GAb+ERp0Dnx8QWQzx8PJgjfcML8+d7vKY8YUR///g+iarqO1NA3PTZkOlWpA41L2fbD5AIhp7eqAF1fGbvjyt9CwmxtWXcpY4ggX676BqQ9Cm8vgoj97HY0pjSrXhgEPwcZZrsO1NFB18zda9IfICl5HUzwREe7ubvti2LbY/+NU4Yv7XRG2Ya+XvteNJY7wsGslfHybK8Q04s3wHsduwlvCbe5b8DePQF6O19Gc2e7VbhhqaWumOqbL9VCxmmsp8NeyD2HNFzDoEYhrG7zYgsgSh9cO7YL/Xucmc438wD0ac7YiK8CQJ13tiEWlYNzIsWG4pTVxRNdwc2lWfgoZfkw0Tt/uOsQb94Y+Y4MfX5BY4vBSzhF4/wY3qmLkJKhZ6CR4Y4rnnIuhxQD47hk33DOcpcyA2HZQs5HXkZy9c0dB3lH48Z3T73dszbn8HBj2r1LdsmCJwyv5+fDZGNc+OmKc6yQzJhBE3KTA7INunbNwlZ3hlu1oNdjrSEom9hxoMRAWvX365sHF78D6ma4PM6ZlyMILBkscXvnuaVg5GS56wi3VbEwg1esA3W9ybe97UryOpnCb5rhv6qVhmZEz6XUnHEqFNV8W/vz+Ta7fqXl/N3mwlLPE4YUlk9w3wW6/gvPu8ToaU1YN/CNEVYbpf/I6ksKlzIAKVdxabKVd6yGuRk5hneT5+fDZWEDgylfDq5b6WSr9r6C02TzPtXM2uwAuf9EWLjTBUy0OLvgtrP3KTbILJ6qQPN0tl1KKZkwXKSLS9XVs/h52rjjxuYVvwOa5ruRvrcbexBdgljhCae96eP+XULspXPcfVwLWmGDq/Wv3TXjaw5Cf53U0P9u7Hg5sLr2jqQrT7UZ3h1fwrmNPMsx4HFpf7J4vIyxxhMqR/W7YLQo3fOgmaxkTbBWi4cInXP2IJRO9juZnKdPdY1lKHFXqQOdr3DyNI/vdunOfjYGoaLjin2WqdcESRyjk5cCHN7kOsusmlvoRFaaU6TAcGveCmX+B7ENeR+OkzHBFi+o09zqSwOo5GnKPuGXS5/0Tti2Cy18oPav++skSR7Cpwpe/g42z3beOZud7HZEpb0Tg4mdcoaS5f/c6Gjd/adPcsnW3cUz9TtDkPJj3sptH0/7KMlnu2RJHsP3wCvz4LlzwOzfD1BgvNOoBna6Fea/AgS3exrLpe7cEfFlMHAA9R0HGLrfgZBkdAGOJI5jWfAnf/Ml96xj4iNfRmPLuwsdAImDGE97GkTLDtfs36+ttHMHS7hduqP1V46FqXa+jCQpLHMGSugQ+ucPNCB/2epkYu21KuZqN4LzfuBoQWxd5F0fKDGh6fqmorX1WIivAla+4FX/LqGCXjr1XRFaIyEoRuc+3rauIzBeRJSKSJCKnLMLv2+cH33HLROS6As+9IyIbfccvEZGuwXwNZ+Vgqiv9WrkOjHzf1Sg2Jhycfy9Uq+8KhnlRs2P/JrcAY1ltpiongpY4RKQjrkxsT6ALMFREWgPPAU+oalfgUd/fJ8sEblLVDsAlwD9EpFaB5x9Q1a6+nyXBeg1n5ehhN+w2+xDc8AFUr+d1RMb8rFI1GPwnN9pnxSehv/6x1XDLwjIj5Vgw7zjaAfNVNVNVc4FZwHBAgRq+fWoCqScfqKrrVDXZ93sqsBuIDWKsgZGfD5+OdmPmr367dJXvNOVHlxugfmc3MS3nSGivnTzDTUiMaRXa65qACmbiWAH0E5EYEakCXAY0Bu4D/iYiW4HngYdOdxJfU1ZFYH2BzU/5mrD+LiKFrlcgIqN9TWFJaWl+rJMfCDMecwVaLn7GLW1tTDiKiHCr56ZvhR9eDd11c7PdsPRWF5XJkUblSdASh6quBp4FpgNTgaVALjAGuF9VGwP3A0VWmxGRBsB/gFtVNd+3+SGgLXAuUAf4QxHXH6eqCaqaEBsbgpuVxe+6CT/n3uFWyjQmnDW/ANoOdfM6Du0KzTW3zIecw9a/UQYEtXNcVcerandV7QfsA5KBm4FPfbt8hOsDOYWI1AC+BB5R1fkFzrlDnWxgQlHHh9SGWa7wfMvBcMmz9m3KlA4X/dndBXz7l9BcL2U6RFRwScuUasEeVRXne2wCjAAm4fo0jo1TG4RLJicfVxGYDPxbVT866bkGvkcBhuGaxLyzJxk+/JVrs71mAkRGeRqOMX6Laenujn96D3YuD/71UmZC0z5QqXrwr2WCKtiTCz4RkVXAFGCsqu7HjbR6QUSWAk8DowFEJEFE3vIddy3QD7ilkGG3E0VkObAcqAs8GeTXULTDe2HiNe5b1A0fuJmixpQm/R5wC25Oezi4w3PTt8PuVdZMVUYE9euxqp5yT6qqc4EehWxPAu7w/f4e8F4R5xwU4DDPTm42fHCjm7NxyxdQu5nXERlTfJVrwYCH4OsHYO3X0Pay4Fzn2DBcSxxlgk1nPhuqMOVe2DLPFZ1v7H03izFnLeFWqHuOK22aezQ410iZAdUbQlz74JzfhJQ1yJ+NOS/A0kkw4GHodLXX0RhTMpEVYMhT8N9rIGk89B4T2PPn5cCG79yabSUcOJKTk8O2bdvIysoKTGwGgOjoaBo1akSFChX82t8SR3GtnOxGoXS6Fvr/n9fRGBMYrS+CloPgu79C5+tcUaJA2bYIsg8GZLb4tm3bqF69Os2aNUNs9GJAqCp79+5l27ZtNG/uX30Ua6oqjm2LYfJdrijOFS/bsFtTdoi4u47sgzDr2cCeO2UGSCQ0L/mif1lZWcTExFjSCCARISYmplh3cZY4/HVgq1u4sFo9uP6/riSnMWVJvfbQ/WZY9JYbZh4oydNdP2DlWmfe1w+WNAKvuO+pJQ5/ZB10CxfmZrl64WV0jX1jGPhHqFDF1ZEJhEO7YOcyG01VxljiOJO8XPjkdkhbA9e+C3FtvY7ImOCpFuuqVa772nVol9T6b91jGUkce/fupWvXrnTt2pX69esTHx9//O+jR/0bkXbrrbeydu3a0+7z6quvMnHixECEHBTWOX4m3/wRkr+BoX93nYfGlHW97nKjq6b9Ee6cDRGRZ3+ulOlQNdatxlsGxMTEsGSJq+Tw+OOPU61aNX7/+9+fsI+qoqpEFFG8bcKECWe8ztixY0sebBBZ4jidhW/Cgteh91hIuM3raIwJjQrRbh2rj26Bn/4DPW45u/Pk57k7jnMuCUoFzCemrGRV6sGAnrN9wxo89osOxT4uJSWFYcOG0bdvXxYsWMAXX3zBE088wY8//siRI0e47rrrePTRRwHo27cvr7zyCh07dqRu3brcddddfP3111SpUoXPP/+cuLg4HnnkEerWrct9991H37596du3L99++y3p6elMmDCB8847j8OHD3PTTTeRkpJC+/btSU5O5q233qJr1+DXtrOmqtOJrODqBw8J0SJwxoSL9sOgSR/49klXlOxsbP8RjuwvM81UZ7Jq1Spuv/12fvrpJ+Lj4/nrX/9KUlISS5cuZfr06axateqUY9LT0+nfvz9Lly6lT58+vP3224WeW1VZuHAhf/vb3/jzn/8MwMsvv0z9+vVZunQpDz74ID/99FNQX19BdsdxOj1ucaNMbBSHKW9E4OKn4M1BMOdFuPCx4p8jZQYgQWviPZs7g2Bq2bIl55577vG/J02axPjx48nNzSU1NZVVq1bRvv2JM+crV67MpZdeCkCPHj2YM2dOoeceMWLE8X02bdoEwNy5c/nDH1xViS5dutChQ+jeD7vjOBNLGqa8iu8Bna93xZ72by7+8Skz3DkCOZkwjFWtWvX478nJybz00kt8++23LFu2jEsuuaTQeRIVK1Y8/ntkZCS5ubmFnrtSpUqn7KNe1Iz3scRhjCna4EdBIlyZ2eI4vBe2Ly43zVQnO3jwINWrV6dGjRrs2LGDadOmBfwaffv25cMPPwRg+fLlhTaFBYslDmNM0WrGw/n3wMpPYetC/4/bkAhoQJYZKY26d+9O+/bt6dixI6NGjeL8888P+DV+85vfsH37djp37swLL7xAx44dqVkzNKUdxMvbnVBJSEjQpKQkr8MwpnQ6ehhe7gE14uH26f6NkJp8F6ybCg+sL9lw3pOsXr2adu3aBex8pVlubi65ublER0eTnJzMkCFDSE5OJirq7LquC3tvRWSxqiacvK91jhtjTq9iVddk9dkYWPEJdL7m9Pvn57v+jZaDApo0zIkyMjIYPHgwubm5qCpvvPHGWSeN4gp26dh7RWSFiKwUkft827qKyHxfVb8kESmq5vjNIpLs+7m5wPYeIrJcRFJE5J9iC9cYE3ydr4cGXVxfx9HM0++7cxkcToNW5bOZKlRq1arF4sWLWbp0KcuWLWPIkCEhu3bQEoeIdMSVie0JdAGGikhr4DngCVXtCjzq+/vkY+sAjwG9fMc/JiK1fU+/his329r3c0mwXoMxxiciAi5+Bg5uc6OsTidluntsNTj4cRlPBPOOox0wX1UzVTUXmAUMBxSo4dunJpBayLEXA9NVdZ+vTvl04BIRaQDUUNUf1HXO/BsYFsTXYIw5ptn5bkLs3L/DoZ1F75cy092dVIsLXWwmpIKZOFYA/UQkRkSqAJcBjYH7gL+JyFbgeeChQo6NB7YW+Hubb1u87/eTt59CREb7msKS0tLSSvxijDG4pUjyjrpiZoU5csCNviqnw3DLi6AlDlVdDTyLu1uYCiwFcoExwP2q2hi4HxhfyOGF9VvoabYXdv1xqpqgqgmxsbFn8QqMMaeo0wJ63Qk/TYQdS099fsN3oHmWOMq4oHaOq+p4Ve2uqv2AfUAycDPwqW+Xj3B9GCfbhrs7OaYRrklrm+/3k7cbY0Kl3wNuNvi0P8LJw/lTZkClmtCo0DEvpd6AAQNOmcz3j3/8g1//+tdFHlOtWjUAUlNTufrqq4s875mmDPzjH/8gM/PngQmXXXYZBw4c8Df0gAr2qKo432MTYAQwCfdBf6yG5CBcMjnZNGCIiNT2dYoPAaap6g7gkIj09o2mugn4PJivwRhzksq1YMBDsGkOrP3q5+2qrn+jRX+ILJsj/UeOHMn7779/wrb333+fkSNHnvHYhg0b8vHHH5/1tU9OHF999RW1agWmqmJxBfu/7iciEgPkAGNVdb+IjAJeEpEoIAs3QgoRSQDuUtU7VHWfiPwFWOQ7z59VdZ/v9zHAO0Bl4GvfjzEmlHrc6soOfPOIG3YbVRF2r4JDqaFrpvr6Qdi5PLDnrN8JLv1rkU9fffXVPPLII2RnZ1OpUiU2bdpEamoqXbt2ZfDgwezfv5+cnByefPJJrrzyyhOO3bRpE0OHDmXFihUcOXKEW2+9lVWrVtGuXTuOHDlyfL8xY8awaNEijhw5wtVXX80TTzzBP//5T1JTUxk4cCB169YlMTGRZs2akZSURN26dXnxxRePr6x7xx13cN9997Fp0yYuvfRS+vbty7x584iPj+fzzz+ncuXKJX6bgpo4VPWCQrbNBXoUsj0JuKPA328Dp6wx7NuvY2AjNcYUS2SUWz134tWw6E3oM9a3Gi5lun8jJiaGnj17MnXqVK688kref/99rrvuOipXrszkyZOpUaMGe/bsoXfv3lxxxRVF1vJ+7bXXqFKlCsuWLWPZsmV07979+HNPPfUUderUIS8vj8GDB7Ns2TLuueceXnzxRRITE6lb98TS1YsXL2bChAksWLAAVaVXr17079+f2rVrk5yczKRJk3jzzTe59tpr+eSTT7jxxhtL/D6UzftJY0zwtb4IWg6GWc9Cl5GQPB3i2rv1rULhNHcGwXSsuepY4nj77bdRVR5++GFmz55NREQE27dvZ9euXdSvX7/Qc8yePZt77rkHgM6dO9O5888VEj/88EPGjRtHbm4uO3bsYNWqVSc8f7K5c+cyfPjw46vzjhgxgjlz5nDFFVfQvHnz44WdCi7JXlK2yKEx5uxd/BRkZ7gmqy3zy/TdxjHDhg1j5syZx6v7de/enYkTJ5KWlsbixYtZsmQJ9erVK3QZ9YIKuxvZuHEjzz//PDNnzmTZsmVcfvnlZzzP6dYbPLYcO5x+2fbissRhjDl7ce1cwbMlEyE/p1wkjmrVqjFgwABuu+22453i6enpxMXFUaFCBRITE9m8+fT1S/r168fEiRMBWLFiBcuWLQPccuxVq1alZs2a7Nq1i6+//rkLt3r16hw6dGo1xn79+vHZZ5+RmZnJ4cOHmTx5MhdccEovQUBZ4jDGlMzAh6FSDahQFZr09jqakBg5ciRLly7l+uuvB+CXv/wlSUlJJCQkMHHiRNq2bXva48eMKENr2QAABZZJREFUGUNGRgadO3fmueeeo2dPN3y5S5cudOvWjQ4dOnDbbbedsBz76NGjufTSSxk4cOAJ5+revTu33HILPXv2pFevXtxxxx1069YtwK/4RLasujGm5FZ/AZl73N1HMC9jy6oHjS2rbowJrXZDvY7AhJA1VRljjCkWSxzGmFKlPDSvh1px31NLHMaYUiM6Opq9e/da8gggVWXv3r1ER0f7fYz1cRhjSo1GjRqxbds2rFRCYEVHR9OoUaMz7+hjicMYU2pUqFCB5s2bex1GuWdNVcYYY4rFEocxxphiscRhjDGmWMrFzHERSQNOv3hM+KsL7PE6iDBi78fP7L04kb0fJyrJ+9FUVU+pvV0uEkdZICJJhU39L6/s/fiZvRcnsvfjRMF4P6ypyhhjTLFY4jDGGFMsljhKj3FeBxBm7P34mb0XJ7L340QBfz+sj8MYY0yx2B2HMcaYYrHEYYwxplgscYQ5EWksIokislpEVorIvV7H5DURiRSRn0TkC69j8ZqI1BKRj0Vkje/fSB+vY/KKiNzv+39khYhMEhH/l3stA0TkbRHZLSIrCmyrIyLTRSTZ91g7ENeyxBH+coHfqWo7oDcwVkTaexyT1+4FVnsdRJh4CZiqqm2BLpTT90VE4oF7gARV7QhEAtd7G1XIvQNcctK2B4GZqtoamOn7u8QscYQ5Vd2hqj/6fj+E+2CI9zYq74hII+By4C2vY/GaiNQA+gHjAVT1qKoe8DYqT0UBlUUkCqgCpHocT0ip6mxg30mbrwTe9f3+Lv/f3h2EWFXFcRz//iiNSWmjKIroFEWLoFQiQjehLcWCEIkKiVYtzDYVtWnjIiEiRAkqCqShCDVs0UIxCKKYoLKi2umgE2OOixIjRKafi3umHuZAF+7zvOn9PnC5h/Me9/3v4vG/59x7/wce7uK3kjjmEUmjwDpgvG4kVb0OPA/8VTuQAXAbMA28W6bu3pa0qHZQNdj+BXgVOA1MAb/bPlo3qoGw3PYUNBehwLIuDprEMU9IWgwcAp61faF2PDVI2gKcs/117VgGxI3AeuAN2+uAP+hoKmK+KXP3DwG3AiuBRZIerxvV/1cSxzwgaQFN0hizfbh2PBVtBLZKmgA+ADZJeq9uSFVNApO2Z0egB2kSyTB6EDhle9r2ZeAwsKFyTIPgV0krAMr+XBcHTeIYcJJEM4f9s+3XasdTk+0Xba+yPUpz4/NT20N7VWn7LHBG0p2lazPwU8WQajoN3C/p5vKf2cyQPihwlY+BHaW9AzjSxUGzdOzg2wg8Afwg6UTpe8n2JxVjisGxExiTtBA4CTxZOZ4qbI9LOgh8Q/Mk4rcMWekRSe8DDwBLJU0CLwOvAB9KeoomuW7r5LdSciQiItrIVFVERLSSxBEREa0kcURERCtJHBER0UoSR0REtJLEEdEBSTOSTvRsnb3BLWm0t+JpRG15jyOiG3/aXls7iIjrISOOiD6SNCFpj6SvynZ76V8j6bik78t+delfLukjSd+VbbZsxg2S3irrTRyVNFLtpGLoJXFEdGPkqqmq7T2fXbB9H7CPprovpX3A9t3AGLC39O8FPrN9D03dqR9L/x3Aftt3Ab8Bj/T5fCLmlDfHIzog6aLtxdfonwA22T5ZilWetb1E0nlghe3LpX/K9lJJ08Aq25d6jjEKHCuL8SDpBWCB7d39P7OIf8uII6L/PEd7ru9cy6We9gy5PxkVJXFE9N/2nv2Xpf0F/yxt+hjweWkfB56Gv9dWv+V6BRnxX+WqJaIbIz3Vi6FZB3z2kdybJI3TXKg9WvqeAd6R9BzNKn6zVW13AW+WaqYzNElkqu/RR7SQexwRfVTucdxr+3ztWCK6kqmqiIhoJSOOiIhoJSOOiIhoJYkjIiJaSeKIiIhWkjgiIqKVJI6IiGjlCrMabh2JbcS2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), train_acc_list, label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), valid_acc_list, label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation ACC: 98.80%\n",
      "Test ACC: 98.95%\n"
     ]
    }
   ],
   "source": [
    "with torch.set_grad_enabled(False):\n",
    "    test_acc = compute_acc(model=model,\n",
    "                           data_loader=test_loader,\n",
    "                           device=DEVICE)\n",
    "    \n",
    "    valid_acc = compute_acc(model=model,\n",
    "                            data_loader=valid_loader,\n",
    "                            device=DEVICE)\n",
    "    \n",
    "\n",
    "print(f'Validation ACC: {valid_acc:.2f}%')\n",
    "print(f'Test ACC: {test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pandas      0.24.2\n",
      "PIL.Image   6.0.0\n",
      "torch       1.2.0\n",
      "numpy       1.16.4\n",
      "torchvision 0.4.0a0+6b959ee\n",
      "matplotlib  3.1.0\n",
      "re          2.2.1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%watermark -iv"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "convnet-vgg16.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "371px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}