predict.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FN = 'predict'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if your GPU is busy you can use CPU for predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['THEANO_FLAGS'] = 'device=cpu,floatX=float32'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import theano\n",
    "DEBUG = False\n",
    "if DEBUG:\n",
    "    import warnings\n",
    "    warnings.filterwarnings('ignore','.*Warning, Cannot compute test value')\n",
    "    warnings.filterwarnings('ignore','.*has no test value')\n",
    "#     theano.config.optimizer='fast_compile'\n",
    "#     theano.config.exception_verbosity='high'\n",
    "    theano.config.optimizer='None'\n",
    "    theano.config.compute_test_value = 'warn'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.0.0'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate headlines using the \"simple\" model from http://arxiv.org/pdf/1512.01712v1.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use indexing of tokens from [vocabulary-embedding](./vocabulary-embedding.ipynb) this does not clip the indexes of the words to `vocab_size`.\n",
    "\n",
    "Use the index of outside words to replace them with several `oov` words (`oov` , `oov0`, `oov`...) that appear in the same description and headline. This will allow headline generator to replace the oov with the same word in the description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "FN0 = 'vocabulary-embedding'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we will generate predictions using the model generated in this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FN1 = 'train'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "input data (`X`) is made from `maxlend` description words followed by `eos`\n",
    "followed by headline words followed by `eos`\n",
    "if description is shorter than `maxlend` it will be left padded with `empty`\n",
    "if entire data is longer than `maxlen` it will be clipped and if it is shorter it will be padded.\n",
    "\n",
    "labels (`Y`) are the headline words followed by `eos` and clipped or padded to `maxlenh`\n",
    "\n",
    "In other words the input is made from a `maxlend` half in which the description is padded from the left\n",
    "and a `maxlenh` half in which `eos` is followed by a headline followed by another `eos` if there is enough space.\n",
    "\n",
    "The labels match only the second half and \n",
    "the first label matches the `eos` at the start of the second half (following the description in the first half)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the model parameters should be identical with what used in training but notice that `maxlend` is flexible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "maxlend=50 # 0 - if we dont want to use description at all\n",
    "maxlenh=25\n",
    "maxlen = maxlend + maxlenh\n",
    "rnn_size = 512\n",
    "rnn_layers = 3  # match FN1\n",
    "batch_norm=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the out of the first `activation_rnn_size` nodes from the top layer will be used for activation and the rest will be used to select predicted word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "activation_rnn_size = 40 if maxlend else 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# training parameters\n",
    "seed=42\n",
    "p_W, p_U, p_dense, weight_decay = 0, 0, 0, 0\n",
    "optimizer = 'adam'\n",
    "batch_size=64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nb_train_samples = 30000\n",
    "nb_val_samples = 3000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# read word embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import cPickle as pickle\n",
    "\n",
    "with open('data/%s.pkl'%FN0, 'rb') as fp:\n",
    "    embedding, idx2word, word2idx, glove_idx2idx = pickle.load(fp)\n",
    "vocab_size, embedding_size = embedding.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_unknown_words = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dimension of embedding space for words 100\n",
      "vocabulary size 40000 the last 10 words can be used as place holders for unknown/oov words\n",
      "total number of different words 523734 523734\n",
      "number of words outside vocabulary which we can substitue using glove similarity 122377\n",
      "number of words that will be regarded as unknonw(unk)/out-of-vocabulary(oov) 361357\n"
     ]
    }
   ],
   "source": [
    "print 'dimension of embedding space for words',embedding_size\n",
    "print 'vocabulary size', vocab_size, 'the last %d words can be used as place holders for unknown/oov words'%nb_unknown_words\n",
    "print 'total number of different words',len(idx2word), len(word2idx)\n",
    "print 'number of words outside vocabulary which we can substitue using glove similarity', len(glove_idx2idx)\n",
    "print 'number of words that will be regarded as unknonw(unk)/out-of-vocabulary(oov)',len(idx2word)-vocab_size-len(glove_idx2idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(nb_unknown_words):\n",
    "    idx2word[vocab_size-1-i] = '<%d>'%i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(vocab_size-nb_unknown_words, len(idx2word)):\n",
    "    idx2word[i] = idx2word[i]+'^'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "empty = 0\n",
    "eos = 1\n",
    "idx2word[empty] = '_'\n",
    "idx2word[eos] = '~'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.preprocessing import sequence\n",
    "from keras.utils import np_utils\n",
    "import random, sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prt(label, x):\n",
    "    print label+':',\n",
    "    for w in x:\n",
    "        print idx2word[w],\n",
    "    print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation, Dropout, RepeatVector\n",
    "from keras.layers.recurrent import LSTM\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.regularizers import l2\n",
    "from keras.layers.core import Lambda\n",
    "import keras.backend as K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# seed weight initialization\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "regularizer = l2(weight_decay) if weight_decay else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## rnn model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "start with a stacked LSTM, which is identical to the bottom of the model used in training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rnn_model = Sequential()\n",
    "if DEBUG:\n",
    "    from keras.layers import InputLayer\n",
    "    model_input = InputLayer(input_shape=(maxlen,))\n",
    "    rnn_model.add(model_input)\n",
    "    rnn_model.output.tag.test_value = np.random.randint(vocab_size,size=(batch_size,maxlen)).astype('float32')\n",
    "rnn_model.add(Embedding(vocab_size, embedding_size,\n",
    "                        input_length=maxlen,\n",
    "#                         batch_input_shape=(batch_size,maxlen),\n",
    "                        embeddings_regularizer=regularizer, weights=[embedding], mask_zero=True,\n",
    "                        name='embedding_1'))\n",
    "for i in range(rnn_layers):\n",
    "    lstm = LSTM(rnn_size, return_sequences=True, # batch_norm=batch_norm,\n",
    "                kernel_regularizer=regularizer, recurrent_regularizer=regularizer,\n",
    "                bias_regularizer=regularizer, dropout=p_W, recurrent_dropout=p_U,\n",
    "                name='lstm_%d'%(i+1)\n",
    "                  )\n",
    "    rnn_model.add(lstm)\n",
    "    rnn_model.add(Dropout(p_dense, name='dropout_%d'%(i+1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if DEBUG:\n",
    "    print rnn_model.output.tag.test_value.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### load"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "use the bottom weights from the trained model, and save the top weights for later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rnn_model.load_weights('data/%s.hdf5'%FN1,by_name=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import h5py\n",
    "with h5py.File('data/%s.hdf5'%FN1, mode='r') as f:\n",
    "    if 'layer_names' not in f.attrs and 'model_weights' in f:\n",
    "        f = f['model_weights']\n",
    "    weights = [np.copy(v) for v in f['timedistributed_1'].itervalues()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(944, 40000), (40000,)]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "map(lambda x: x.shape, weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## headline model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A special layer that reduces the input just to its headline part (second half).\n",
    "For each word in this part it concatenate the output of the previous layer (RNN)\n",
    "with a weighted average of the outputs of the description part.\n",
    "In this only the last `rnn_size - activation_rnn_size` are used from each output.\n",
    "The first `activation_rnn_size` output is used to computer the weights for the averaging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def simple_context(X, mask, n=activation_rnn_size, maxlend=maxlend, maxlenh=maxlenh):\n",
    "    desc, head = X[:,:maxlend,:], X[:,maxlend:,:]\n",
    "    head_activations, head_words = head[:,:,:n], head[:,:,n:]\n",
    "    desc_activations, desc_words = desc[:,:,:n], desc[:,:,n:]\n",
    "    \n",
    "    # RTFM http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.batched_tensordot\n",
    "    # activation for every head word and every desc word\n",
    "    activation_energies = K.batch_dot(head_activations, desc_activations, axes=([2],[2]))\n",
    "    # make sure we dont use description words that are masked out\n",
    "    activation_energies = activation_energies + -1e20*K.expand_dims(1.-K.cast(mask[:, :maxlend],'float32'),1)\n",
    "    \n",
    "    # for every head word compute weights for every desc word\n",
    "    activation_energies = K.reshape(activation_energies,(-1,maxlend))\n",
    "    activation_weights = K.softmax(activation_energies)\n",
    "    activation_weights = K.reshape(activation_weights,(-1,maxlenh,maxlend))\n",
    "\n",
    "    # for every head word compute weighted average of desc words\n",
    "    desc_avg_word = K.batch_dot(activation_weights, desc_words, axes=([2],[1]))\n",
    "    if DEBUG:\n",
    "        print desc_avg_word.tag.test_value.shape\n",
    "        print head_words.tag.test_value.shape\n",
    "    return K.concatenate((desc_avg_word, head_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(rnn_model)\n",
    "\n",
    "if activation_rnn_size:\n",
    "    model.add(Lambda(simple_context,\n",
    "                     mask = lambda inputs, mask: mask[:,maxlend:],\n",
    "                     output_shape = lambda input_shape: (input_shape[0], maxlenh, 2*(rnn_size - activation_rnn_size)),\n",
    "                     name='simplecontext_1'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# we are not going to fit so we dont care about loss and optimizer\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "944"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 2*(rnn_size - activation_rnn_size)\n",
    "n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "perform the top dense of the trained model in numpy so we can play around with exactly how it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# out very own softmax\n",
    "def output2probs(output):\n",
    "    output = np.dot(output, weights[0]) + weights[1]\n",
    "    output -= output.max()\n",
    "    output = np.exp(output)\n",
    "    output /= output.sum()\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def output2probs1(output):\n",
    "    output0 = np.dot(output[:n//2], weights[0][:n//2,:])\n",
    "    output1 = np.dot(output[n//2:], weights[0][n//2:,:])\n",
    "    output = output0 + output1 # + output0 * output1\n",
    "    output += weights[1]\n",
    "    output -= output.max()\n",
    "    output = np.exp(output)\n",
    "    output /= output.sum()\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def lpadd(x, maxlend=maxlend, eos=eos):\n",
    "    \"\"\"left (pre) pad a description to maxlend and then add eos.\n",
    "    The eos is the input to predicting the first word in the headline\n",
    "    \"\"\"\n",
    "    assert maxlend >= 0\n",
    "    if maxlend == 0:\n",
    "        return [eos]\n",
    "    n = len(x)\n",
    "    if n > maxlend:\n",
    "        x = x[-maxlend:]\n",
    "        n = maxlend\n",
    "    return [empty]*(maxlend-n) + x + [eos]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "samples = [lpadd([3]*26)]\n",
    "# pad from right (post) so the first maxlend will be description followed by headline\n",
    "data = sequence.pad_sequences(samples, maxlen=maxlen, value=empty, padding='post', truncating='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.all(data[:,maxlend] == eos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1, 75), [51])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape,map(len, samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 25, 944)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probs = model.predict(data, verbose=0, batch_size=1)\n",
    "probs.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# variation to https://github.com/ryankiros/skip-thoughts/blob/master/decoding/search.py\n",
    "def beamsearch(predict, start=[empty]*maxlend + [eos], avoid=None, avoid_score=1,\n",
    "               k=1, maxsample=maxlen, use_unk=True, oov=vocab_size-1, empty=empty, eos=eos, temperature=1.0):\n",
    "    \"\"\"return k samples (beams) and their NLL scores, each sample is a sequence of labels,\n",
    "    all samples starts with an `empty` label and end with `eos` or truncated to length of `maxsample`.\n",
    "    You need to supply `predict` which returns the label probability of each sample.\n",
    "    `use_unk` allow usage of `oov` (out-of-vocabulary) label in samples\n",
    "    \"\"\"\n",
    "    def sample(energy, n, temperature=temperature):\n",
    "        \"\"\"sample at most n different elements according to their energy\"\"\"\n",
    "        n = min(n,len(energy))\n",
    "        prb = np.exp(-np.array(energy) / temperature )\n",
    "        res = []\n",
    "        for i in xrange(n):\n",
    "            z = np.sum(prb)\n",
    "            r = np.argmax(np.random.multinomial(1, prb/z, 1))\n",
    "            res.append(r)\n",
    "            prb[r] = 0. # make sure we select each element only once\n",
    "        return res\n",
    "\n",
    "    dead_samples = []\n",
    "    dead_scores = []\n",
    "    live_samples = [list(start)]\n",
    "    live_scores = [0]\n",
    "\n",
    "    while live_samples:\n",
    "        # for every possible live sample calc prob for every possible label \n",
    "        probs = predict(live_samples, empty=empty)\n",
    "        assert vocab_size == probs.shape[1]\n",
    "\n",
    "        # total score for every sample is sum of -log of word prb\n",
    "        cand_scores = np.array(live_scores)[:,None] - np.log(probs)\n",
    "        cand_scores[:,empty] = 1e20\n",
    "        if not use_unk and oov is not None:\n",
    "            cand_scores[:,oov] = 1e20\n",
    "        if avoid:\n",
    "            for a in avoid:\n",
    "                for i, s in enumerate(live_samples):\n",
    "                    n = len(s) - len(start)\n",
    "                    if n < len(a):\n",
    "                        # at this point live_sample is before the new word,\n",
    "                        # which should be avoided, is added\n",
    "                        cand_scores[i,a[n]] += avoid_score\n",
    "        live_scores = list(cand_scores.flatten())\n",
    "        \n",
    "\n",
    "        # find the best (lowest) scores we have from all possible dead samples and\n",
    "        # all live samples and all possible new words added\n",
    "        scores = dead_scores + live_scores\n",
    "        ranks = sample(scores, k)\n",
    "        n = len(dead_scores)\n",
    "        dead_scores = [dead_scores[r] for r in ranks if r < n]\n",
    "        dead_samples = [dead_samples[r] for r in ranks if r < n]\n",
    "        \n",
    "        live_scores = [live_scores[r-n] for r in ranks if r >= n]\n",
    "        live_samples = [live_samples[(r-n)//vocab_size]+[(r-n)%vocab_size] for r in ranks if r >= n]\n",
    "\n",
    "        # live samples that should be dead are...\n",
    "        # even if len(live_samples) == maxsample we dont want it dead because we want one\n",
    "        # last prediction out of it to reach a headline of maxlenh\n",
    "        def is_zombie(s):\n",
    "            return s[-1] == eos or len(s) > maxsample\n",
    "        \n",
    "        # add zombies to the dead\n",
    "        dead_scores += [c for s, c in zip(live_samples, live_scores) if is_zombie(s)]\n",
    "        dead_samples += [s for s in live_samples if is_zombie(s)]\n",
    "        \n",
    "        # remove zombies from the living \n",
    "        live_scores = [c for s, c in zip(live_samples, live_scores) if not is_zombie(s)]\n",
    "        live_samples = [s for s in live_samples if not is_zombie(s)]\n",
    "\n",
    "    return dead_samples, dead_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# !pip install python-Levenshtein"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def keras_rnn_predict(samples, empty=empty, model=model, maxlen=maxlen):\n",
    "    \"\"\"for every sample, calculate probability for every possible label\n",
    "    you need to supply your RNN model and maxlen - the length of sequences it can handle\n",
    "    \"\"\"\n",
    "    sample_lengths = map(len, samples)\n",
    "    assert all(l > maxlend for l in sample_lengths)\n",
    "    assert all(l[maxlend] == eos for l in samples)\n",
    "    # pad from right (post) so the first maxlend will be description followed by headline\n",
    "    data = sequence.pad_sequences(samples, maxlen=maxlen, value=empty, padding='post', truncating='post')\n",
    "    probs = model.predict(data, verbose=0, batch_size=batch_size)\n",
    "    return np.array([output2probs(prob[sample_length-maxlend-1]) for prob, sample_length in zip(probs, sample_lengths)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def vocab_fold(xs):\n",
    "    \"\"\"convert list of word indexes that may contain words outside vocab_size to words inside.\n",
    "    If a word is outside, try first to use glove_idx2idx to find a similar word inside.\n",
    "    If none exist then replace all accurancies of the same unknown word with <0>, <1>, ...\n",
    "    \"\"\"\n",
    "    xs = [x if x < vocab_size-nb_unknown_words else glove_idx2idx.get(x,x) for x in xs]\n",
    "    # the more popular word is <0> and so on\n",
    "    outside = sorted([x for x in xs if x >= vocab_size-nb_unknown_words])\n",
    "    # if there are more than nb_unknown_words oov words then put them all in nb_unknown_words-1\n",
    "    outside = dict((x,vocab_size-1-min(i, nb_unknown_words-1)) for i, x in enumerate(outside))\n",
    "    xs = [outside.get(x,x) for x in xs]\n",
    "    return xs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def vocab_unfold(desc,xs):\n",
    "    # assume desc is the unfolded version of the start of xs\n",
    "    unfold = {}\n",
    "    for i, unfold_idx in enumerate(desc):\n",
    "        fold_idx = xs[i]\n",
    "        if fold_idx >= vocab_size-nb_unknown_words:\n",
    "            unfold[fold_idx] = unfold_idx\n",
    "    return [unfold.get(x,x) for x in xs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import Levenshtein\n",
    "\n",
    "def gensamples(X=None, X_test=None, Y_test=None, avoid=None, avoid_score=1, skips=2, k=10, batch_size=batch_size, short=True, temperature=1., use_unk=True):\n",
    "    if X is None or isinstance(X,int):\n",
    "        if X is None:\n",
    "            i = random.randint(0,len(X_test)-1)\n",
    "        else:\n",
    "            i = X\n",
    "        print 'HEAD %d:'%i,' '.join(idx2word[w] for w in Y_test[i])\n",
    "        print 'DESC:',' '.join(idx2word[w] for w in X_test[i])\n",
    "        sys.stdout.flush()\n",
    "        x = X_test[i]\n",
    "    else:\n",
    "        x = [word2idx[w.rstrip('^')] for w in X.split()]\n",
    "        \n",
    "    if avoid:\n",
    "        # avoid is a list of avoids. Each avoid is a string or list of word indeicies\n",
    "        if isinstance(avoid,str) or isinstance(avoid[0], int):\n",
    "            avoid = [avoid]\n",
    "        avoid = [a.split() if isinstance(a,str) else a for a in avoid]\n",
    "        avoid = [vocab_fold([w if isinstance(w,int) else word2idx[w] for w in a])\n",
    "                 for a in avoid]\n",
    "\n",
    "    print 'HEADS:'\n",
    "    samples = []\n",
    "    if maxlend == 0:\n",
    "        skips = [0]\n",
    "    else:\n",
    "        skips = range(min(maxlend,len(x)), max(maxlend,len(x)), abs(maxlend - len(x)) // skips + 1)\n",
    "    for s in skips:\n",
    "        start = lpadd(x[:s])\n",
    "        fold_start = vocab_fold(start)\n",
    "        sample, score = beamsearch(predict=keras_rnn_predict, start=fold_start, avoid=avoid, avoid_score=avoid_score,\n",
    "                                   k=k, temperature=temperature, use_unk=use_unk)\n",
    "        assert all(s[maxlend] == eos for s in sample)\n",
    "        samples += [(s,start,scr) for s,scr in zip(sample,score)]\n",
    "\n",
    "    samples.sort(key=lambda x: x[-1])\n",
    "    codes = []\n",
    "    for sample, start, score in samples:\n",
    "        code = ''\n",
    "        words = []\n",
    "        sample = vocab_unfold(start, sample)[len(start):]\n",
    "        for w in sample:\n",
    "            if w == eos:\n",
    "                break\n",
    "            words.append(idx2word[w])\n",
    "            code += chr(w//(256*256)) + chr((w//256)%256) + chr(w%256)\n",
    "        if short:\n",
    "            distance = min([100] + [-Levenshtein.jaro(code,c) for c in codes])\n",
    "            if distance > -0.6:\n",
    "                print score, ' '.join(words)\n",
    "        #         print '%s (%.2f) %f'%(' '.join(words), score, distance)\n",
    "        else:\n",
    "                print score, ' '.join(words)\n",
    "        codes.append(code)\n",
    "    return samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "seed = 8\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = \"* Billy Joel is looking for a buyer in Sagaponack^ . Now that he and wife Katie Lee Joel are splitting up , the singer is planning to sell the two oceanfront^ properties he bought for her in 2007 . The four-bedroom mansion ( No . 1 ) and smaller beach bungalow^ ( No . 2 ) will be listed with Corcoran 's Biana^ Stepanian^ for a combined $ 35 million . * Richard Bressler^ , the former CFO of Viacom and now a managing\"\n",
    "Y = \"Billy Joel Lists in Sagaponack^\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "18.397234031 Gossip Roundup : It 's Not a Enough\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X=X, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = \"18 Cake GIFs That 'll Make You Moist\"\n",
    "Y = \"Is it 350degF^ in here or is it just me ?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "8.397752285 The Cake\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = \"President Barack Obama 's re-election campaign is fundraising off of comments on Obama 's birth certificate by Mitt Romney 's son Matt .\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "15.4022150934 Obama : Obama 's \\ '' <0>^ ''\n",
      "20.8515859842 Obama , Obama : Obama , Obama\n",
      "20.8536094129 Obama , Obama : Obama 's Birth Certificate\n",
      "22.9639167339 Obama : Obama 's \\ '' <0>^ '' And \\ '' <0>^ ''\n",
      "27.8957381975 Obama , Obama : Obama 's Birth Certificate Is \\ '' <0>^ ''\n",
      "29.7963155061 Obama : Obama 's \\ '' <0>^ '' And \\ '' <0>^ '' Is A\n",
      "33.2783554345 Obama : Obama 's \\ '' <0>^ '' And \\ '' <0>^ '' Is A A\n",
      "33.4898159783 Obama , Obama : Obama 's Birth Certificate Is \\ '' <0>^ Of <0>^ ''\n",
      "35.9433095604 Obama : Obama 's \\ '' <0>^ '' And \\ '' <0>^ '' Is A Big Deal\n",
      "40.0852098018 Obama : Obama 's \\ '' <0>^ '' And \\ '' <0>^ '' Is A Big Deal '\n",
      "41.6016341131 Obama , Obama : Obama 's Birth Certificate Is \\ '' <0>^ Of <0>^ '' In Florida\n",
      "43.4185510259 Obama , Obama : Obama 's Birth Certificate Is \\ '' <0>^ Of <0>^ '' In New Hampshire\n",
      "45.0034724176 Obama : Obama 's \\ '' <0>^ '' And \\ '' <0>^ '' Is A Big Look At Voters\n",
      "45.7667950541 Obama : Obama 's \\ '' <0>^ '' And \\ '' <0>^ '' Is A Big Look At Women\n",
      "48.3252496552 Obama , Obama : Obama 's Birth Certificate Is \\ '' <0>^ Of <0>^ '' In Viral Speech\n",
      "52.2721019387 Obama , Obama : Obama 's Birth Certificate Is \\ '' <0>^ Of <0>^ '' In New Hampshire ( Updated )\n",
      "53.6473008674 Obama , Obama : Obama 's Birth Certificate Is \\ '' <0>^ Of <0>^ '' In New Hampshire 's Abortion\n",
      "53.8994252365 Obama , Obama : Obama 's Birth Certificate Is \\ '' <0>^ Of <0>^ '' In New Hampshire 's Cabinet\n",
      "54.5488198251 Obama : Obama 's \\ '' <0>^ '' And \\ '' <0>^ '' Is A Big Look At Women 's Image\n",
      "62.2592916042 Obama : Obama 's \\ '' <0>^ '' And \\ '' <0>^ '' Is A Big Look At Women 's Image Of Mitt\n"
     ]
    }
   ],
   "source": [
    "gensamples(X, skips=2, batch_size=batch_size, k=10, temperature=1, use_unk=True, short=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = \"What have you been listening to this year ? If you want to find out using cold , hard evidence , then Spotify 's new Year in Music tool will tell you .\"\n",
    "Y = \"Spotify Will Make You Smarter for Your App\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "24.2458644509 Spotify Wants to Get Your Mind Out of Your\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, skips=2, batch_size=batch_size, k=10, temperature=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "headline = samples[0][0][len(samples[0][1]):]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Spotify Wants to Get Your Mind Out of Your ~'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join(idx2word[w] for w in headline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "avoid = headline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "17.418826066 This Is What It 's Like To Be The Internet\n",
      "27.7085759401 Spotify 's New <0>^ App Might Be the Ultimate <0>^\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, avoid=avoid, avoid_score=.1, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "avoid = samples[0][0][len(samples[0][1]):]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "19.0077718496 How To Make Your Own , <0>^\n",
      "37.7090225875 Here 's What Exactly When You Do n't Want You to Know What It 's Like\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, avoid=avoid, avoid_score=.1, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def wsimple_context(X, mask, n=activation_rnn_size, maxlend=maxlend, maxlenh=maxlenh):\n",
    "    desc, head = X[:,:maxlend], X[:,maxlend:]\n",
    "    head_activations, head_words = head[:,:,:n], head[:,:,n:]\n",
    "    desc_activations, desc_words = desc[:,:,:n], desc[:,:,n:]\n",
    "    \n",
    "    # RTFM http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.batched_tensordot\n",
    "    # activation for every head word and every desc word\n",
    "    activation_energies = K.batch_dot(head_activations, desc_activations, axes=([2],[2]))\n",
    "    # make sure we dont use description words that are masked out\n",
    "    assert mask.ndim == 2\n",
    "    activation_energies = K.switch(mask[:, None, :maxlend], activation_energies, -1e20)\n",
    "    \n",
    "    # for every head word compute weights for every desc word\n",
    "    activation_energies = K.reshape(activation_energies,(-1,maxlend))\n",
    "    activation_weights = K.softmax(activation_energies)\n",
    "    activation_weights = K.reshape(activation_weights,(-1,maxlenh,maxlend))\n",
    "\n",
    "    return activation_weights\n",
    "\n",
    "\n",
    "class WSimpleContext(Lambda):\n",
    "    def __init__(self):\n",
    "        super(WSimpleContext, self).__init__(wsimple_context)\n",
    "        self.supports_masking = True\n",
    "\n",
    "    def compute_mask(self, input, input_mask=None):\n",
    "        return input_mask[:, maxlend:]\n",
    "    \n",
    "    def get_output_shape_for(self, input_shape):\n",
    "        nb_samples = input_shape[0]\n",
    "        n = 2*(rnn_size - activation_rnn_size)\n",
    "        return (nb_samples, maxlenh, n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wmodel = Sequential()\n",
    "wmodel.add(rnn_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "wmodel.add(Lambda(wsimple_context,\n",
    "                 mask = lambda inputs, mask: mask[:,maxlend:],\n",
    "                 output_shape = lambda input_shape: (input_shape[0], maxlenh, maxlend)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "wmodel.compile(loss='categorical_crossentropy', optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "seed = 8\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = \"Representatives of the groups depicted in The Revenant^ spoke with BuzzFeed News about the actor 's Golden Globes speech calling on listeners to `` protect ... indigenous lands . ''\"\n",
    "Y = \"Native American Groups Officially Respond To Leonardo DiCaprio 's Call To Action\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEADS:\n",
      "20.5102219582 How The Golden Globes Will Make You Cry\n"
     ]
    }
   ],
   "source": [
    "samples = gensamples(X, skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sample = samples[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Representatives of the groups depicted in The trainwreck spoke with BuzzFeed News about the actor 's Golden Globes speech calling on listeners to `` protect ... indigenous lands . '' ~ How The Golden Globes Will Make You Cry ~\""
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([idx2word[w] for w in sample])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 75)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = sequence.pad_sequences([sample], maxlen=maxlen, value=empty, padding='post', truncating='post')\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 25, 50)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights = wmodel.predict(data, verbose=0, batch_size=1)\n",
    "weights.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 8)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "startd = np.where(data[0,:] != empty)[0][0]\n",
    "lenh = np.where(data[0,maxlend+1:] == eos)[0][0]\n",
    "startd, lenh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW0AAAEACAYAAAB4ayemAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGrBJREFUeJzt3X1MVff9B/D3pVg3daMi4V61sprSXtFQHvqg60waHoTV\nTVg3l3Uu6Y2z8ZcuyzrTdJCmyU/TZNymWTNdu7/chHalXZMusGX4AKyQYtpacwWJo/hTrE7xXmyl\n+FS1wvn9wS7l4cL9oFy+3w/3/fqLczjH8+0b+vXL23MOLsdxHBARkQoJpgdARERynLSJiBThpE1E\npAgnbSIiRThpExEpwkmbiEiRqJP20aNHkZOTg9zcXOTk5CApKQk7duxAb28vioqK4PV6UVxcjL6+\nvukYLxFRXHNN5j7tgYEB3Hnnnfjwww/xyiuvYMGCBfjNb36DF198Eb29vfD7/bEcKxFR3JtUPdLQ\n0IC7774bS5YsQW1tLXw+HwDA5/OhpqYmJgMkIqKvTGrS/utf/4oNGzYAAEKhENxuNwDA4/Ggp6dn\n6kdHREQjiOuRL7/8EosWLUJHRwdSUlKQnJyM8+fPD31+wYIF+Oyzz2I2UCIiAhKlB+7evRv3338/\nUlJSAABut3totR0MBpGamhrxvJKSEly9ehUejwcAMHfuXKSnpyM7OxsA0NraCgBxvx3eZ8t4bN4+\nduwY1q9fb814bN0e/b1lejw2b4/OzNTXa+/evQAG24usrCw888wzGE280v7pT3+K7373u0M9dllZ\nGZKTk1FWVjbhP0Q+8cQT2L59u+QScc3v96O8vNz0MFRgVjLMSc7GrAKBAAoKCsbsF3XaV65cQUND\nA374wx8O7SsrK0N9fT28Xi8aGxvH/Q8OBoM3OeT4curUKdNDUINZyTAnOU1ZieqROXPm4Ny5cyP2\nJScno6GhISaDIiKiyGL+RGRxcXGsLzEjhO/KoeiYlQxzktOU1aQerrkZjY2NyM3NjeUliIhmnFvq\ntG/F8H+dpfG1tLSYHoIazEqGOclpyoovjCIiUoT1CBGRhYzVI0RENHXYaVtCU6dmGrOSYU5ymrLi\nSpuISBF22kREFmKnTUQ0A7DTtoSmTs00ZiXDnOQ0ZcWVNhGRIuy0iYgsxE6biGgGYKdtCU2dmmnM\nSoY5yWnKiittIiJF2GkTEVlovE5b/It9b8Xl6/0AgNsTXJiVyMU9EdHNmpZOe8s/jmLLP46iq/eL\nWF9OLU2dmmnMSoY5yWnKalpW2p/0XgUA9A9Mx9WIiGaumK+0s7OzY32JGWH16tWmh6AGs5JhTnKa\nsmLBTESkCO/TtoSmTs00ZiXDnOQ0ZSWatPv6+vDjH/8YGRkZWLFiBT788EP09vaiqKgIXq8XxcXF\n6Ovri/VYiYjinmjSfvrpp7F27Vp0dHSgra0Ny5Ytg9/vR2FhITo7O5Gfn4+KioqI57LTltHUqZnG\nrGSYk5ymrKJO2hcuXMB7772HjRs3AgASExORlJSE2tpa+Hw+AIDP50NNTU1sR0pERNEn7RMnTiAl\nJQUbN25Ebm4uNm/ejCtXriAUCsHtdgMAPB4Penp6Ip7PTltGU6dmGrOSYU5ymrKKep/2jRs3EAgE\n8Oqrr+KBBx7Ali1b4Pf74XK5Rhw3ejusubkZXd37MHu+B386uQDpi1KQmZk59ONIOKx43w6zZTw2\nb7e3t1s1Hm7r3w4zOZ6WlhZUV1cDANLS0pCamhrxMfao7x4JhUL49re/ja6urqGL+P1+HD9+HE1N\nTXC73QgGg8jLy0NHR8eY8xsbG1EeGJzQf7/uXix3z53ockREhFt4n7bb7caSJUtw9OhRAIOT8IoV\nK1BSUoLKykoAQFVVFUpLS6d2xERENIbo7pEdO3bgZz/7GbKzs9HW1obnnnsOZWVlqK+vh9frHVxN\nl5dHPJedtoymTs00ZiXDnOQ0ZSV690hWVhY++uijMfsbGhqmfEBERDQ+vnvEEpruEzWNWckwJzlN\nWfHdI0REivDdI5bQ1KmZxqxkmJOcpqy40iYiUoSdtiU0dWqmMSsZ5iSnKSuutImIFGGnbQlNnZpp\nzEqGOclpyoorbSIiRdhpW0JTp2Yas5JhTnKasuJKm4hIEXbaltDUqZnGrGSYk5ymrLjSJiJShJ22\nJTR1aqYxKxnmJKcpK660iYgUYadtCU2dmmnMSoY5yWnKiittIiJF2GlbQlOnZhqzkmFOcpqy4kqb\niEgRdtqW0NSpmcasZJiTnKasuNImIlKEnbYlNHVqpjErGeYkpykrrrSJiBRhp20JTZ2aacxKhjnJ\nacoqUXLQXXfdhaSkJCQkJGDWrFk4cOAAent78ZOf/AQnT57EXXfdhbfffhtJSUmxHi8RUVwTrbQT\nEhLQ1NSEQ4cO4cCBAwAAv9+PwsJCdHZ2Ij8/HxUVFRHPZacto6lTM41ZyTAnOU1ZiSZtx3EwMDAw\nYl9tbS18Ph8AwOfzoaamZupHR0REI4gmbZfLhTVr1uDBBx/Ezp07AQChUAhutxsA4PF40NPTE/Fc\ndtoymjo105iVDHOS05SVqNPev38/Fi5ciHPnzqGoqAherxcul2vEMaO3w5qbm9HVvQ+z53vwp5ML\nkL4oBZmZmUM/joTDivftMFvGY/N2e3u7VePhtv7tMJPjaWlpQXV1NQAgLS0NqampKCgowGgux3Gc\nMXsnsG3bNsybNw87d+5EU1MT3G43gsEg8vLy0NHRMeb4xsZGlAcGJ/Tfr7sXy91zJ3M5IqK4FAgE\nIk7aUeuRK1eu4NKlSwCAy5cvY9++fcjMzERJSQkqKysBAFVVVSgtLZ3aERMR0RhRJ+1QKITVq1cj\nJycHq1atwrp161BUVISysjLU19fD6/UOrqbLyyOez05bRlOnZhqzkmFOcpqyitppL126NOLEm5yc\njIaGhpgMioiIIuO7Ryyh6T5R05iVDHOS05QV3z1CRKQI3z1iCU2dmmnMSoY5yWnKiittIiJF2Glb\nQlOnZhqzkmFOcpqy4kqbiEgRdtqW0NSpmcasZJiTnKasuNImIlKEnbYlNHVqpjErGeYkpykrrrSJ\niBRhp20JTZ2aacxKhjnJacqKK20iIkXYaVtCU6dmGrOSYU5ymrLiSpuISBF22pbQ1KmZxqxkmJOc\npqy40iYiUoSdtiU0dWqmMSsZ5iSnKSuutImIFGGnbQlNnZppzEqGOclpyoorbSIiRdhpW0JTp2Ya\ns5JhTnKasuJKm4hIEXbaltDUqZnGrGSYk5ymrMST9sDAAHJzc1FSUgIA6O3tRVFREbxeL4qLi9HX\n1xezQRIR0SDxpL19+3YsX758aNvv96OwsBCdnZ3Iz89HRUVFxPPYacto6tRMY1YyzElOU1aiSfv0\n6dOoq6vDk08+ObSvtrYWPp8PAODz+VBTUxObERIR0RDRpL1lyxa89NJLcLlcQ/tCoRDcbjcAwOPx\noKenJ+K57LRlNHVqpjErGeYkpymrxGgH/POf/4Tb7UZ2djaamprGPW74hD5cc3Mzurr3YfZ8D/50\ncgHSF6UgMzNz6MeRcFjxvh1my3hs3m5vb7dqPNzWvx1mcjwtLS2orq4GAKSlpSE1NRUFBQUYzeU4\njjNm7zDPPfcc/vKXvyAxMRFffPEFLl68iMceewwHDx5EU1MT3G43gsEg8vLy0NHRMeb8xsZGlAcG\nJ/Tfr7sXy91zJ7ocEREBCAQCESftqPXIb3/7W5w6dQpdXV146623kJ+fj9dffx3r1q1DZWUlAKCq\nqgqlpaVTPmgiIhrppu/TLi8vR319Pbxe7+Bqurw84nHstGU0dWqmMSsZ5iSnKauonfZwjzzyCB55\n5BEAQHJyMhoaGmIyKCIiiozvHrGEpvtETWNWMsxJTlNWfPcIEZEifPeIJTR1aqYxKxnmJKcpK660\niYgUYadtCU2dmmnMSoY5yWnKiittIiJF2GlbQlOnZhqzkmFOcpqy4kqbiEgRdtqW0NSpmcasZJiT\nnKasuNImIlKEnbYlNHVqpjErGeYkpykrrrSJiBRhp20JTZ2aacxKhjnJacqKK20iIkWmudN20NZ9\nEW3dF3H2wrVYX1oVTZ2aacxKhjnJacpqWlfan1+9gWfrjuHZumPouXR9Oi9NRDQjsNO2hKZOzTRm\nJcOc5DRlxU6biEgR3qdtCU2dmmnMSoY5yWnKiittIiJF2GlbQlOnZhqzkmFOcpqy4kqbiEgRdtqW\n0NSpmcasZJiTnKasok7a165dw8qVK5GTk4PMzExs27YNANDb24uioiJ4vV4UFxejr68v5oMlIop3\nUSft2bNn491338WhQ4fQ2tqK3bt348CBA/D7/SgsLERnZyfy8/NRUVER8Xx22jKaOjXTmJUMc5LT\nlJWoHpkzZw6AwVX3jRs34HK5UFtbC5/PBwDw+XyoqamJ3SiJiAiAcNIeGBhATk4OPB4P1qxZgwcf\nfBChUAhutxsA4PF40NPTE/Fcdtoymjo105iVDHOS05RVouSghIQEHDp0CBcuXMBjjz2GI0eOwOVy\njThm9HZYc3Mzurr3YfZ8D944fgeC3V9izqJ0AOkAvgor/ONJvG6H2TIem7fb29utGg+39W+HmRxP\nS0sLqqurAQBpaWlITU1FQUEBRnM5juOM2TuBF154AXPmzMHOnTvR1NQEt9uNYDCIvLw8dHR0jDm+\nsbER5YHBCX3rmqXYWn8CAPDS2nRkLfrGZC5NRBQ3AoFAxEk7aj3y6aefDt0Z8sUXX6C+vh4ZGRko\nKSlBZWUlAKCqqgqlpaVTO2IiIhoj6qR99uxZ5OXlITs7GytXrkRxcTHWrl2LsrIy1NfXw+v1Dq6m\ny8sjns9OW0ZTp2Yas5JhTnKasoraaWdmZiIQCIzZn5ycjIaGhpgMioiIIuO7Ryyh6T5R05iVDHOS\n05QV3z1CRKQI3z1iCU2dmmnMSoY5yWnKiittIiJF2GlbQlOnZhqzkmFOcpqy4kqbiEgRdtqW0NSp\nmcasZJiTnKasuNImIlKEnbYlNHVqpjErGeYkpykrrrSJiBRhp20JTZ2aacxKhjnJacqKK20iIkXY\naVtCU6dmGrOSYU5ymrLiSpuISBF22pbQ1KmZxqxkmJOcpqy40iYiUoSdtiU0dWqmMSsZ5iSnKSuu\ntImIFGGnbQlNnZppzEqGOclpyoorbSIiRdhpW0JTp2Yas5JhTnKasuJKm4hIEXbaltDUqZnGrGSY\nk5ymrKJO2qdPn0Z+fj5WrFiBzMxM7NixAwDQ29uLoqIieL1eFBcXo6+vL+aDJSKKd1En7cTERLz8\n8ss4cuQI3n//fbz66qv4+OOP4ff7UVhYiM7OTuTn56OioiLi+ey0ZTR1aqYxKxnmJKcpq6iTtsfj\nGZp4582bh4yMDJw+fRq1tbXw+XwAAJ/Ph5qamtiOlIiIJtdpf/LJJ2htbcWqVasQCoXgdrsBDE7s\nPT09Ec9hpy2jqVMzjVnJMCc5TVklSg+8dOkS1q9fj+3bt2PevHlwuVwjPj96O6y5uRld3fswe74H\nbxy/A8HuLzFnUTqAdABfhRX+8SRet8NsGY/N2+3t7VaNh9v6t8NMjqelpQXV1dUAgLS0NKSmpqKg\noACjuRzHccbsHeXGjRv4/ve/j0cffRRPP/00ACAjIwNNTU1wu90IBoPIy8tDR0fHmHMbGxtRHhic\n0LeuWYqt9ScAAC+tTUfWom9EuzQRUVwKBAIRJ21RPfLzn/8cy5cvH5qwAaCkpASVlZUAgKqqKpSW\nlk7NSImIaFxRJ+39+/fjjTfewL/+9S/k5OQgNzcXe/bsQVlZGerr6+H1egdX0+XlEc+/1U777IVr\naOu+iLbuizh74dot/Vk209SpmcasZJiTnKasonba3/nOd9Df3x/xcw0NDVM+oNF6Ll3Hs3XHAAxW\nKgu/OTvm1yQishXfPWIJTfeJmsasZJiTnKasxHePxNrZC9fQc+k6ACB13u1cURMRRWDNu0fCNciz\ndceGJu94oqlTM41ZyTAnOU1Z8S1/RESKsNO2hKZOzTRmJcOc5DRlxZU2EZEi1nTa8U5Tp2Yas5Jh\nTnKasuJKm4hIEXbaltDUqZnGrGSYk5ymrLjSJiJShJ22JTR1aqYxKxnmJKcpK660iYgUYadtCU2d\nmmnMSoY5yWnKiittIiJFjHXatyVg6D3Zbd0Xcb1/IOLnhu+fyTR1aqYxKxnmJKcpK2Nv+eu72o9t\nDSeGtv+3cGnEzw3fT0QU79hpW0JTp2Yas5JhTnKasmKnTUSkCO/TtoSmTs00ZiXDnOQ0ZcWVNhGR\nIuy0LaGpUzONWckwJzlNWXGlTUSkCDttS2jq1ExjVjLMSU5TVlEn7U2bNsHtduO+++4b2tfb24ui\noiJ4vV4UFxejr68vpoMkIqJBUSftjRs3Yu/evSP2+f1+FBYWorOzE/n5+aioqBj3fHbaMpo6NdOY\nlQxzktOUVdRJe/Xq1Zg/f/6IfbW1tfD5fAAAn8+Hmpqa2IyOiIhGuKlOu6enB263GwDg8XjQ09Mz\n7rHstGU0dWqmMSsZ5iSnKaspefeIy+Ua93PNzc3o6t6H2fM9eOP4HQh2f4k5i9KB/75T5MLx/07q\nw7bb7wgBGPxLof3gB7hw/Cy+efdgzRION/zjzEzZDrNlPDZvt7e3WzUebuvfDjM5npaWFlRXVwMA\n0tLSkJqaioKCAozmchzHGbN3lJMnT2LdunU4fPgwACAjIwNNTU1wu90IBoPIy8tDR0dHxHMbGxtR\nHhic1LeuWYqt9V+9CGr0C6OGvyQq0scvrU1H1qJvRBsuEZF6gUAg4qQtqkccx8Hwub2kpASVlZUA\ngKqqKpSWlk7NKImIaEJRJ+0NGzbg4YcfxtGjR5GWloZdu3ahvLwc9fX18Hq9gyvp8vJxz2enLaOp\nUzONWckwJzlNWUXttMMdy2gNDQ1TPhgiIpoY3z1iCU33iZrGrGSYk5ymrIz95ppbdfbCNfRcug4A\nSJ13OxZ+c7bhERERxZ7ad4/0XLqOZ+uO4dm6Y0OTt2aaOjXTmJUMc5LTlBXf8kdEpEjM65Hs7Gy8\nFZiaPyv8W9oBRPzt7cDkqxJbahZNnZppzEqGOclpykrVSrvvav9QJXK934m4f7JVyUyrWYhoZlPb\nac80mjo105iVDHOS05SVqpU2EVG8433altDUqZnGrGSYk5ymrLjSJiJShJ22JTR1aqYxKxnmJKcp\nK7VPRNrAltsFiSh+sNO+BVN5u6CmTs00ZiXDnOQ0ZcVOm4hIkRnXaYefjmzrvojjn14Z+vjshWtD\nx5y9cG1o//AnK4cbfszwc2NFU6dmGrOSYU5ymrKacZ1239X+cX9VWbhzDtca4WMiGX7M8HOJiExi\np20JTZ2aacxKhjnJacpqxq20b8V4L6SaTrwjhYgmMuM67Vsx3guppkO4U+MLrKLT1D+axJzkNGXF\nu0eIiBRR9T5tGwyvL6ayQrnZTi0e6xRN/aNJ/J6S0/Q9xZX2JA2vL6a7Qok2HtYpNBX4PWW3W5q0\n9+zZg2XLluHee+/Fiy++GPEYTZ22SZo6NdOYlQxzktOU1U1P2gMDA/jlL3+JvXv34siRI3jzzTfx\n8ccfjznu2LFjtzTAeNHe3m56CGowKxnmJKcpq5vutA8cOIB77rkH3/rWtwAAjz/+OGpra7Fs2bIR\nx12+fPnWRmiByd4KOF4nONH+o6fPRX1CM1rPOHycEx032T93+DHzbr8Nl673i//bJntdyf4zPecn\nNf541dfXZ3oIUQ3/+gHmvobDs7L9e+qmJ+0zZ85gyZIlQ9t33nknDhw4MCWDss3opyyjGe9pyon2\n7/u/8zhSd+yWntAcPs6Jjpvsnzv6CdJoT5neynUl+1f89y+Nm7ku2WX41w+w42to+/dUzO8eCQaD\n+J8nFw9ezOWK9eXUutYbND0ENYJn/mN6CCqcOnXK9BDU0JSVy3Gcm7oF4oMPPsDWrVuxZ88eAIDf\n74fL5UJZWdmI45566qkRFUlWVhYfbY+gtbWVuQgxKxnmJGdDVq2trWhraxvazsrKwjPPPDPmuJue\ntPv7++H1etHY2IiFCxfioYcewptvvomMjIybHzUREU3opuuR2267Da+88gqKioowMDCATZs2ccIm\nIoqxm15pExHR9JuSJyI3bdoEt9uN++67b9xjfvWrX+Gee+5BdnZ23D5wEy2nzs5OPPzww/ja176G\nl19+eZpHZ5doWVVXVyMrKwtZWVlYvXq1qvtsp1K0nP7+978jKysLOTk5eOihh7B///5pHqE9JPMU\nAHz00UeYNWsW/va3v03TyCbJmQLvvfeec+jQISczMzPi5+vq6py1a9c6juM4H3zwgbNy5cqpuKw6\n0XI6d+6cc/DgQef55593fve7303z6OwSLav333/f+fzzzx3HcZzdu3fze2qcnC5fvjz08eHDh51l\ny5ZN19CsEy0rx3Gc/v5+Jz8/3/ne977nvPPOO9M4OrkpWWmvXr0a8+fPH/fztbW1eOKJJwAAK1eu\nRF9fH0Kh0FRcWpVoOaWkpOD+++9HYiJfcx4tq1WrViEpKWno4zNnzkzX0KwSLac5c+YMfXzp0iUk\nJMTv64aiZQUAf/jDH7B+/XqkpqZO06gmb1q+gqMfxFm8eHHc/k9GU2/nzp149NFHTQ/DWjU1NcjI\nyMC6devw5z//2fRwrNXd3Y2amho89dRTcCz+p774/WuXZoR3330Xu3btGveFZQT84Ac/QEdHB2pq\navD888+bHo61fv3rX4/4PrJ14p6Wn8MXL16M//znq6fYTp8+jcWLF0/HpWkGO3z4MDZv3ow9e/ZE\n/bGXBuuBrq4unD9/HsnJyaaHY52DBw/i8ccfh+M4+PTTT7F7927MmjULJSUlpoc2wpSttB3HGfdv\nppKSErz22msABp+kvOOOO+B2u6fq0qpMlNPo4+LdRFmdOnUKP/rRj/D666/j7rvvnuaR2WWinI4f\nPz70cSAQwPXr1+N6wp4oq66uLnR1deHEiRNYv349/vjHP1o3YQNTtNLesGEDmpqa8NlnnyEtLQ3b\ntm3D9evX4XK5sHnzZqxduxZ1dXVIT0/H3LlzsWvXrqm4rDrRcgqFQnjggQdw8eJFJCQkYPv27fj3\nv/+NefPmmR76tIuW1QsvvIDz58/jF7/4BRzHwaxZs2bsC8smEi2nd955B6+99hpuv/12fP3rX8fb\nb79tesjGRMtqOJfF70niwzVERIrwHyKJiBThpE1EpAgnbSIiRThpExEpwkmbiEgRTtpERIpw0iYi\nUoSTNhGRIv8Ppo/auuwbOKEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11f9aa3d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.hist(np.array(weights[0,:lenh,startd:].flatten()+1), bins=100);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from IPython.core.display import display, HTML\n",
    "\n",
    "def heat(sample,weights,dark=0.3):\n",
    "    weights = (weights - weights.min())/(weights.max() - weights.min() + 1e-4)\n",
    "    html = ''\n",
    "    fmt = ' <span style=\"background-color: #{0:x}{0:x}ff\">{1}</span>'\n",
    "    for t,w in zip(sample,weights):\n",
    "        c = int(256*((1.-dark)*(1.-w)+dark))\n",
    "        html += fmt.format(c,idx2word[t])\n",
    "    display(HTML(html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       " <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #100100ff\">_</span> <span style=\"background-color: #fafaff\">Representatives</span> <span style=\"background-color: #f4f4ff\">of</span> <span style=\"background-color: #ffffff\">the</span> <span style=\"background-color: #f0f0ff\">groups</span> <span style=\"background-color: #f7f7ff\">depicted</span> <span style=\"background-color: #ffffff\">in</span> <span style=\"background-color: #fdfdff\">The</span> <span style=\"background-color: #f0f0ff\">trainwreck</span> <span style=\"background-color: #f7f7ff\">spoke</span> <span style=\"background-color: #ffffff\">with</span> <span style=\"background-color: #ececff\">BuzzFeed</span> <span style=\"background-color: #c7c7ff\">News</span> <span style=\"background-color: #fefeff\">about</span> <span style=\"background-color: #ffffff\">the</span> <span style=\"background-color: #b9b9ff\">actor</span> <span style=\"background-color: #ffffff\">'s</span> <span style=\"background-color: #5555ff\">Golden</span> <span style=\"background-color: #4c4cff\">Globes</span> <span style=\"background-color: #d6d6ff\">speech</span> <span style=\"background-color: #f9f9ff\">calling</span> <span style=\"background-color: #fefeff\">on</span> <span style=\"background-color: #f8f8ff\">listeners</span> <span style=\"background-color: #fdfdff\">to</span> <span style=\"background-color: #e8e8ff\">``</span> <span style=\"background-color: #f0f0ff\">protect</span> <span style=\"background-color: #f2f2ff\">...</span> <span style=\"background-color: #f1f1ff\">indigenous</span> <span style=\"background-color: #ffffff\">lands</span> <span style=\"background-color: #fcfcff\">.</span> <span style=\"background-color: #f5f5ff\">''</span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "heat(sample, weights[0,-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "columns = [idx2word[data[0,i]] for i in range(startd,maxlend)]\n",
    "rows = [idx2word[data[0,i]] for i in range(maxlend+1,maxlend+lenh+1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(weights[0,:lenh,startd:],columns=columns,index=rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAccAAAGZCAYAAAD8ePo1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XlcE9f6P/DPBFCRRXAB/eJye6lLK+KCWm0tWpdq3UFE\n1NpWvG4Vra0rKIpVoWrV2qrXevW6oC217oA7ttpaNxRrtXWrtqgoLqissiW/P/gxl0ASZmISkvB5\n+8rrRcLJ5AlO8sw585wzgkqlUoGIiIhEiooOgIiIyNwwORIREZXC5EhERFQKkyMREVEpTI5ERESl\nMDkSERGVYmvsF8i686es9rmPHstq7/jSPyW3zXv2VNa2BRt5xw52TjVktYfMWTQFWRmy2gu28v57\nbapVl9W+IDNdVvuMm8my2rt6e8lqL8eva2JltW85rp+s9veOnpTVfsnyI7LaR/53rKz2VWvWlNVe\nsJG376iUhfK2r7CR3LYw97m8bQuCrPYQ5H3Olbk5strbOjrLai/3eyE/45ms9g71PWW1l8O7UWe9\nn3vx72MGjOTFGT05EhFR5SD7wMSMcViViIgswvHjx9GrVy/07NkTa9eu1dru4sWLaN68OQ4dOiT7\nucWYHImIyCAEQaH3rTxKpRLz58/H+vXrERcXh/j4ePz5Z9nTdkqlEkuXLkWnTp1kP7ckJkciIjJ7\nFy9eRKNGjeDh4QE7Ozv06dMHCQkJZdpFR0ejZ8+eqFniPLvU55bE5EhERAahgKD3rTypqamoV6+e\neN/d3R0PHjwo0+bIkSMYNmyY7OeWxoIcIiIyiIouyImMjMS0adMMsi0mRyIiMgiFzGkxcri7uyMl\nJUW8n5qaCjc3N7U2ly5dwscffwyVSoUnT57g+PHjsLGxkfTc0pgciYjI7LVo0QLJycm4e/cu6tSp\ng/j4eCxbtkytTcnziKGhoXjrrbfQrVs3FBYWlvvc0pgciYjIIIw5rGpjY4Pw8HAEBwdDpVIhICAA\nnp6eiImJgSAIGDJkiOzn6iIpOSYnJ6Nhw4by3gkREZEB+fr6wtfXV+2xoKAgjW2joqLKfa4ukpJj\nWFgY7t+/jxYtWqBt27Zo27YtmjZtKvlFiIjI+gkSqk4thaTkuGXLFuTl5eG3337DmTNnMHbsWGRn\nZ+PMmTPGjo+IiCyEMQtyTE1SckxMTMS5c+eQmJiIjIwMdOnSBW3btjV2bEREZEEqeiqHIUlKju+9\n9x6aN2+OsWPHwtfXF1WqVDF2XERERBVGUnI8deoUzp8/j7Nnz2Lz5s1QKBRo1aoVJk+ebOz4iIjI\nQigqW8/R2dkZDRo0wL1793D//n0kJSWhoKDA2LERERFVCEnJsVu3bvjnP/8JHx8fDB06FFFRURxa\nJSIiNYIVLdctKTkePnwYCoX1vGkiIjI8ayrIkZTxHjx4gAkTJqBjx47o2LEjJk6ciPv37xs7NiIi\nsiAKQdD7Zm4kJcfQ0FB07doVP/30E3766Se89dZbCA0NNXZsREREFUJSckxLS8OgQYNga2sLW1tb\n+Pv7Iy0tzdixERGRBRFe4J+5kZQcXVxcsGfPHhQWFqKwsBB79uyBi4uLsWMjIiKqEJKSY2RkJPbv\n34833ngDnTp1wsGDB8ss6kpERJWbQlDofTM3kqpVPTw8sGbNGmPHQkREFsyaqlV1Jsf58+frfLOz\nZ882eEBERGSZzLHqVF86k6OXl5f481dffYWJEycaPSAiIqKKpjM5+vn5iT9v2rRJ7T4REVFJ5lh1\nqi/JZ0GtaSyZiIhIF0kFOUREROUxx6pTfelMjq1btxZ7jM+fP0ebNm0AACqVCoIg4Pz588aPkIiI\nLII1jTDqTI5JSUmmioOIiCycNVWrWk8fmIiIyEB4zpGIiAyiUlarEhERVRbsORIRkUFUmoIcIiIi\nqaypIIfJkYiIDILnHImIiKyY8XuOMrvZyvwCIwUCqJSFstoLCus5CjIHCjvzGaiwd65q1O0X5OQb\ndfsWv2+qVNKbyvzcQmEjq7kApbztG5lKZV7xyGFNK+RYzzshIiIyEPM5lCciIovGalUiIqJSWK1K\nRERUCqtViYiIrBh7jkREZBDWNKzKniMREVEp7DkSEZFBsFqViIioFGsaVmVyJCIig6hU1aq3bt3C\n+++/j759+wIArly5gtWrVxs9MCIisiwKQdD7Zm7KTY7h4eGYMmUKbG2LOpnNmjXDvn37jB4YERFR\nRSk3Oebk5MDb21vtMRsbeQv7EhERWZJyzzm6uroiOTlZrEI6cOAA6tSpY/TAiIjIslSqatW5c+ci\nPDwcN2/exJtvvon69etjyZIlpoiNiIgsiDmeO9RXucmxQYMG2LhxI7Kzs6FUKuHo6GiKuIiIyMJY\nU7VquckxLy8PBw8exN27d1FQ8L8LEYeEhBg1MCIisiyVquc4fvx4ODk5oXnz5qhSpYopYiIiIqpQ\n5SbH1NRUrF+/3hSxEBERmYVyp3K0bt0aV69eNUUsRERkwQRB0PtmbrT2HPv27QtBEFBYWIidO3ei\nfv36asOqsbGxJgmQiIgsQ6U455iamordu3ebMhYiIrJg5tgD1JfW5Fi/fn14eHiYMhYiIrJgxp7K\ncfz4cURGRkKlUmHQoEEYM2aM2u8TEhKwYsUKKBQK2NraIjQ0FD4+Prh//z6mT5+Ox48fQ6FQYPDg\nwXjvvfd0vpbW5Pj48WNs2LBB6xNHjhwp820RERHpR6lUYv78+di4cSPc3NwQEBCAbt26wdPTU2zz\n+uuvo1u3bgCAq1evYvLkydi/fz9sbGwQGhqKV155BVlZWfD398cbb7yh9tzStCZHpVKJrKwsA741\nIiIi/Vy8eBGNGjUSRzT79OmDhIQEtQRnb28v/pydnQ2FoqjmtE6dOuKypw4ODvD09MSDBw/0S451\n6tThRH8iIpJMYcRR1dTUVNSrV0+87+7ujt9++61MuyNHjmDp0qVIS0vD2rVry/z+zp07uHLlSpkL\napSmdSqHSqWSEzcREVVy5jCVo3v37ti/fz9WrVqFL774Qu13WVlZmDRpEsLCwuDg4KBzO1p7jhs3\nbjRIoEREVDkYcyqHu7s7UlJSxPupqalwc3PT2r5t27a4ffs2nj59ChcXFxQUFGDSpEkYMGAAunfv\nXu7rae05uri4yAydiIgqM2P2HFu0aIHk5GTcvXsXeXl5iI+PF4tviiUnJ4s/X758Gfn5+WIuCwsL\nw8svv4z3339f0nspd/k4IiKiimZjY4Pw8HAEBwdDpVIhICAAnp6eiImJgSAIGDJkCA4ePIg9e/bA\nzs4OVatWFYdVz507h9jYWDRp0gQDBw6EIAj4+OOP4evrq/X1mByJiMgi+Pr6lkloQUFB4s+jR4/G\n6NGjyzzPx8cHf/zxh6zXYnIkIiKDUFSm6zkSERFJUSmWjyMiIpKjUiw8TkREJIcV5cbyr+dIRERU\n2TA5EhERlWL0YVVFlaqy2ldz177igUZy+vEyl8RTVK0mLxYjK8zNldXe1ta4/70qpVJW++cPn8lq\nX+MVWc1lsa9h3P/bZ/czZLW/l/FUVntlfr6s9sYe7xIEmcfZMuIRFDbyNi27vby/jaKqffmNXoDc\n+G2qGTceOXjOkYiIqBRjX8/RlJgciYjIIDiVg4iIqJRKN6yalpaGbdu24e7duygoKBAfj4qKMlpg\nREREFUVScvzwww/h4+ODjh07wsZG3sliIiKqHKyo4ygtOebk5GDatGnGjoWIiMgsSKq/7tKlC44d\nO2bsWIiIyIIpBEHvm7mR1HPcvHkzvv76a9jZ2cHOzg4qlQqCIOD8+fPGjo+IiCxEpZvKkZSUZOw4\niIjIwpljD1BfkoZVVSoV9uzZg1WrVgEA7t27h4sXLxo1MCIioooiKTlGRETgwoULiIuLAwBUr14d\n8+bNM2pgRERkWQRB/5u5kZQcL168iLlz56Jq1aJ1UmvUqIF8uWs7EhERWQhJ5xxtbW1RWFgoLg2U\nlpYGhYIX9CAiov+pdMvHjRgxAhMmTMDjx4+xfPlyHDhwAJMnTzZ2bEREZEGsqSBHUnLs378/mjdv\njlOnTkGlUmH16tXw9PQ0dmxERGRBrCg36k6OT5/+7xpztWrVQp8+fdR+5+LiYrzIiIiIKojO5Ojv\n7w9BEKBSqXDv3j04OzsDANLT01GvXj0cPXrUJEESEZH5qzTDqsXJb/bs2ejRowc6d+4MADh27BgS\nEhKMHx0REVEFkFRy+uuvv4qJEQA6d+7MVXOIiEiN8AL/zI2kghw3NzesXr0a/fv3BwDExsbCzc3N\nqIEREZFlsaapHJJ6jkuXLkVaWhpCQkIQEhKCx48fY+nSpcaOjYiILIhC0P9mbiT1HF1cXDB79mxj\nx0JERGQWdCbHcePG6XzymjVrDBoMERFZLmsaVtWZHIODg00VBxERkdnQmRzbt28v/pyXl4e//voL\nAPDSSy/Bzs7OqIEREZFlqTQ9x2KnT5/GzJkz4eHhIS4IsGjRIrRr187Y8RERkYUwx8IafUlKjosW\nLcL69evxz3/+EwBw69YtTJkyBTt37jRqcEREZDmsqecoaSpHfn6+mBiBomFVXs+RiIislaSeo5eX\nF2bNmiUuArB37154eXkZNTAiIrIsVtRxlJYc582bh61bt2LLli0AAB8fHwwbNsyogREREVUUncOq\nR44cwdatW1GlShWMHDkS9+7dw+XLlxEdHc0rchARkRqFIOh9Mzc6k+O6devQtWtX8X5+fj527tyJ\n6OhofPvtt0YPjoiILEelWXg8Pz8f9erVE+/7+PjAxcUFLi4uyMnJMXpwRERkOcywA6g3nT3H9PR0\ntftz5swRf05LSzNORERERBVMZ3L09vbGtm3byjweExMDb29vowVFRESWx5rOOeocVg0LC8OECRMQ\nGxuL5s2bAwAuX76MvLw8rFq1yiQBEhERmZrO5FirVi3ExMTg5MmTuHHjBgCgc+fO6Nixo0mCIyIi\ny2FNK+RImufYsWNHvROiwlbSS4hyZJ7LrOLiKj0WC18sXZmbJ+8Jjk7GCeT/U1SpIqt9FRcHI0Ui\nX42X3Y26fffmdWW1v/74lqz2T3//W1Z7NxcXWe1tqlWX1V6lUspqL0hbnAuA/O+QwrxcWe1tqlSV\n1V6lLJTVHrDs7x05rCg3SkuORERE5bGmnqP0wzciIqJKgj1HIiIyCGu6ZBV7jkRERKVISo779+9H\nZmYmAGD16tUICQnB5cuXjRoYERFZFkEQ9L6ZG0nJcfXq1XB0dERiYiJOnjyJgIAAREREGDk0IiKy\nJIKg/83cSEqONjY2AIBjx44hMDAQXbp04cWOiYhIjbFXyDl+/Dh69eqFnj17Yu3atWV+Hxsbi/79\n+6N///4YOnQorl69qvZ7pVIJPz8/jBs3rvz3IiUgd3d3zJkzB/v27UPnzp2Rl5cHpVLevCYiIiJ9\nKZVKzJ8/H+vXr0dcXBzi4+Px559/qrVp0KABtm7dir1792L8+PEIDw9X+/3mzZvh6ekp6fUkJccv\nvvgCnTp1wvr16+Hs7IynT59i+vTpEt8SERFVBsY853jx4kU0atQIHh4esLOzQ58+fZCQkKDWplWr\nVnBychJ/Tk1NFX93//59HDt2DIMHD5b0XiQlR3t7e9SsWRPnzp0DANja2qJRo0aSXoCIiOhFpaam\nql1C0d3dHQ8ePNDa/vvvv4evr694PzIyEtOnT5dc/CMpOa5cuRLr1q0Tx3jz8/Mxbdo0SS9ARESV\ng7kU5Jw6dQo7d+7E1KlTAQA//vgjateujVdeeQUqlUrSNiQtAnD48GHs3r0bfn5+AIoydlZWlp5h\nExGRNTLmlAx3d3ekpKSI91NTU+Hm5lam3ZUrVzBnzhysW7cONWrUAACcP38eR48exbFjx5Cbm4us\nrCxMnz4dixcv1vp6knqOdnZ2auPC2dnZst4UERFZP2P2HFu0aIHk5GTcvXsXeXl5iI+PR7du3dTa\npKSkYNKkSVi8eDEaNmwoPv7JJ5/gxx9/REJCApYtW4bXXntNZ2IEJPYc33nnHcyZMwfp6enYtm0b\nduzYgcDAQClPJSIiemE2NjYIDw9HcHAwVCoVAgIC4OnpiZiYGAiCgCFDhmD16tV49uwZ5s2bB5VK\nBVtbW2zfvl2v15OUHEeNGoUTJ07AwcEBt27dwqRJk/DGG2/o9YJERGSdpM5X1Jevr69akQ0ABAUF\niT8vWLAACxYs0LmN9u3bo3379uW+luSFx5s0aYLnz59DEAQ0adJE6tOIiIgsjqRzjt9//z0GDx6M\nw4cP4+DBgxgyZIjeXVUiIrJO5lKtagiSeo7r1q3Drl274OrqCgB48uQJgoKCEBAQYNTgiIjIcpjj\nAuL6kpQcXV1d4eDgIN53cHAQEyURERFgnj1AfelMjhs2bAAANGzYEIGBgejWrRsEQUBCQgKaNm1q\nkgCJiIhMTWdyLJ7o37BhQ7U5I6XnlhAREVWaYdWQkBC1+8XJsuQQKxERkbWRdM7x2rVrmD59Op49\newag6BzkokWL0LhxY6MGR0RElsOKOo7SkuOcOXMwc+ZMdOjQAQBw+vRphIeHIyYmxqjBERGR5TD2\nIgCmJCk5Zmdni4kRAF577TWur0pERGqsKDdKS44NGjTAqlWrMGDAAADA3r170aBBA6MGRkREVFEk\nrZATGRmJJ0+eYOLEiZg4cSLS0tIQGRlp7NiIiMiCFF+9SZ+buZHUc6xRowZmz55t7FiIiIjMgs7k\nOG7cOJ1PXrNmjUGDISIiy2WGHUC96UyOwcHBZR4r7v6qVCrjRERERBbJHIdH9aUzOaanpyM1NRXD\nhw8HAAQEBCAtLQ2CIGDq1KkmCZCIiCyDFeVG3QU569atQ9euXcX7+fn52LFjB6KjoznHkYiIrJbO\nnmN+fj7q1asn3vfx8YGrqytcXV2Rk5Nj9OCIiMhyWNOwqs6eY3p6utr9OXPmiD+npaUZJyIiIqIK\npjM5ent7Y9u2bWUej4mJgbe3t9GCIiIiyyMI+t/Mjc5h1bCwMEyYMAGxsbFo3rw5AODy5cvIy8vD\nqlWrTBIgERFZBmsaVtWZHGvVqoWYmBicPHkSN27cAAB07twZHTt2NElwRERkOawoN0pbIadjx45M\niEREVGlISo6mlJ8h72ofyvx8yW0Lc/Nkbbvw+XNZ7RW168hqD5kLKdi5ushq//z+fVntq9evL6u9\nqqBAVnu571fO/61gI2mZYJHCTt6ur1IWympv51BNVvvYFdNktf/rfIqs9s6e9cpvVILcfaEwR97n\nVlGlqvRtPzduZby8/1n5VIXyPieqQnkRFebK+54ypkp3ySoiIqLyWFFulHZVDiIiosqEPUciIjKI\nSlOtSkREJJUV5UYOqxIREZXGniMRERmEoLCeriOTIxERGQSHVYmIiKwYe45ERGQQrFYlIiIqxYpy\nI4dViYiIStPZc5w/f77ObvLs2bMNHhAREVmmSjOs6uXlZao4iIjIwllRbtSdHP38/EwVBxERkdnQ\nmRzHjRun88lr1qwxaDBERGTBrKjrqDM5BgcHmyoOIiKycJXmnGP79u1NFQcREZHZ0JkcP/roI6xY\nsQL9+vXT+PvY2FijBEVERJbHijqOupPjq6++iosXL2LlypWwteV6AUREpF2lWXj86dOniIyMxM2b\nN9GkSRO0adMGrVu3RuvWreHi4mKqGImIiExKZ3KcMWMGACAvLw+XLl1CUlISdu7cifDwcDg7O2Pf\nvn0mCZKIiMxfpRlWLZabm4vMzExkZGQgIyMDbm5uaNq0qbFjIyIiC1JpqlXDw8Nx/fp1ODg4oGXL\nlmjdujVGjhyJGjVqmCo+IiIik9O58HhKSgry8vJQp04duLu7o27dunB2djZVbEREZEEEQf+budHZ\nc1y/fj1UKhWuX7+OpKQkbNiwAdeuXYOLiwtatWqFSZMmmSpOIiIyc5VmWBUoerNNmjSBs7MznJyc\n4OjoiB9//BEXL15kciQiIqukMzlu3rwZSUlJSEpKgq2trTiNIyAgAE2aNDFVjEREZAGsqOOoOzne\nvXsXvXr1QmhoKNzc3EwVExERWSBjD6seP34ckZGRUKlUGDRoEMaMGaP2+5s3byIsLAyXL1/GJ598\ngpEjR4q/y8jIwKxZs3D9+nUoFApERkaiZcuWWl9LZ3IMDQ19wbdCRET04pRKJebPn4+NGzfCzc0N\nAQEB6NatGzw9PcU2Li4umD17No4cOVLm+QsXLkTnzp3x5ZdfoqCgAM+fP9f5ejqrVYmIiCRTvMCt\nHBcvXkSjRo3g4eEBOzs79OnTBwkJCWptatasCS8vrzLLnWZmZiIxMRGDBg0CANja2sLR0bHct0JE\nRPTCBEHQ+1ae1NRU1KtXT7zv7u6OBw8eSIrrzp07cHV1RWhoKPz8/BAeHs6eIxERVW4FBQX4/fff\nMWzYMOzatQvVqlXD2rVrdT6HyZGIiAzCmIsAuLu7IyUlRbyfmpoquVC0bt26qFu3Llq0aAEA6Nmz\nJ37//Xedz2FyJCIigzDmsGqLFi2QnJyMu3fvIi8vD/Hx8ejWrZvW9iqVSvy5du3aqFevHm7dugUA\nOHXqlFohjya8SCMREZk9GxsbhIeHIzg4GCqVCgEBAfD09ERMTAwEQcCQIUPw6NEjDBo0CFlZWVAo\nFNi8eTPi4+Ph4OCA2bNnY+rUqSgoKECDBg0QFRWl8/WYHImIyCCMvQiAr68vfH191R4LCgoSf65d\nuzaOHTum8bnNmjXDjh07JL8WkyMRERmGFS2RY3bJ0c6putG2rSoskNVeUaWKzBdQld+mZHOlvPaQ\n2d7O2Une9mVS5ufLam/M/1u5f3tjy0x5Iqt94XN5+2aTLrrPl5Rm6+ggq73cfVPuviDYSv/qyX30\nWNa2barby2ovl8LOTlZ7mypVZbVXyvyekvu3J2nMLjkSEZFlEhTsORIREamxolFVTuUgIiIqjT1H\nIiIyiEp1sWMiIiIprCg3cliViIioNPYciYjIMKyo6ygrOebk5MDe3rhziIiIyDJZ01QOScOq58+f\nR+/evfHOO+8AAK5cuYKIiAhjxkVERFRhJCXHqKgorF+/Hi4uLgCK1qhLTEw0amBERGRZjHnJKlOT\nPKxa8grMAKBQsJaHiIhKMMcspydJybFevXo4f/48BEFAfn4+Nm/eXO61sIiIiCyVpO5fREQEtm7d\nitTUVPj6+uKPP/7AnDlzjB0bERFZkEo3rKpQKLB06VK1x27fvg1XV1ejBEVERJan0lWrjhs3DpmZ\nmeL9P//8E+PHjzdaUERERBVJcnIcN24csrKycOnSJUyaNAlLliwxdmxERGRBBEHQ+2ZuJA2rdunS\nBQUFBRg1ahSysrKwcuVKvPTSS8aOjYiILIn55Ti96UyO8+fPV8voGRkZaNCgAbZu3QoAmD17tnGj\nIyIiqgA6k6OXl5fa/ebNmxs1GCIislzmODyqL53J0c/Pz1RxEBGRhas0ybHYX3/9hWXLluHGjRvI\nzc0VH09ISDBaYERERBVFUrVqaGgohg4dChsbG2zevBkDBw5E//79jR0bERFZEsUL3MyMpJByc3PR\nsWNHAICHhwcmTpyIY8eOGTUwIiKyLJVuKkeVKlWgVCrRqFEjbNmyBe7u7sjKyjJ2bERERBVCUs8x\nLCwMOTk5mD17Ni5fvow9e/Zg0aJFxo6NiIgsSKXrOXp7ewMAHBwcEBUVZdSAiIjIQplfjtObzuQ4\nbtw4nU9es2aNQYMhIiIyBzqT44ULF1CvXj306dMHLVu2hEqlMlVcRERkYazpqhw6k+OJEydw4sQJ\nxMfHIy4uDp07d0bfvn3RuHFjU8VHRESWwgzPHepLZ0GOjY0NfH19sWjRImzbtg2NGjXCiBEjsGXL\nFlPFR0REZHLlFuTk5eXhxx9/RFxcHO7evYsRI0agR48epoiNiIgsiBV1HHUnx+nTp+P69evw9fVF\nSEgImjRpYqq4iIjIwpjjlAx96UyOe/fuhb29Pf766y9ER0eLj6tUKgiCgPPnzxs9QCIiIlPTmRyv\nXLliqjiIiMjSVZZqVSIiIqmsaVjVDNdCJyIiqljsORIRkWFYT8fR+MlRpZS3qk41tzqy2itspb8F\n2+rVZW1bsLOT115hI6s9UCivuczxfEW1arLay49fHmWBvPcr5/9WrvzMHFnt5f5trp6+K6u9k3NV\nWe1reTeU1d7WwUFWe7l/e0Fuexvpny1lfoGsbdvKHNqTu1+qbOR9p8nddxQ28v6WKiN+TuTisCoR\nEZEVM59DDiIismiVZm1VIiIiyaxoWJXJkYiIDILnHImIiKwYe45ERGQY1tNxZM+RiIioNPYciYjI\nIFitSkREVJoVFeQwORIRkUGwWpWIiMiKldtzXLNmjcbHx40bZ/BgiIjIghn5nOPx48cRGRkJlUqF\nQYMGYcyYMWXaLFiwAMePH4e9vT0+++wzvPLKKwCAjRs3Yvv27RAEAU2aNEFUVBSqVKmi9bXK7Tkq\nFArxVlBQgCNHjuDvv/9+gbdHREQkj1KpxPz587F+/XrExcUhPj4ef/75p1qbY8eOITk5GYcOHcKn\nn36KuXPnAgBSU1MRHR2NnTt3IjY2FoWFhdi3b5/O1yu351g6M48ePRr/+te/5L4vIiKycsY853jx\n4kU0atQIHh4eAIA+ffogISEBnp6eYpuEhAQMHDgQANCyZUtkZGTg0aNHAIqSa05ODhQKBZ4/fw43\nNzedryf7nGNeXh7u378v92lERGTthBe4lSM1NRX16tUT77u7u+PBgwdqbR48eIC6deuqtUlNTYW7\nuztGjhyJLl26wNfXF05OTnj99dd1vl65PceBAweKRwOFhYV4+PAhxo4dW/47ISKiSsVcq1XT09OR\nkJCAH374AU5OTpg0aRJiY2PRr18/rc8pNzl++eWX4s82NjaoU6eOzpOYREREhubu7o6UlBTxfmpq\napmhUTc3N7WRzfv378Pd3R2//PILGjRoABcXFwBAjx49kJSUpDM5ljus2rBhQ+Tm5uLEiRM4fvw4\nkpOTZb8pIiKqBBSC/rdytGjRAsnJybh79y7y8vIQHx+Pbt26qbXp1q0bdu/eDQC4cOECnJ2dUbt2\nbfzf//35DEQhAAAgAElEQVQffv31V+Tm5kKlUuHUqVNq5yo1KbfnuGXLFnzzzTfo3r07AOCjjz7C\nsGHDMHz48HLfDBERkSHY2NggPDwcwcHBUKlUCAgIgKenJ2JiYiAIAoYMGYLOnTvj2LFj6NGjB+zt\n7REVFQUA8Pb2Rs+ePTFw4EDY2tri1VdfRWBgoM7XE1QqlUpXg379+iEmJgYODg4AgKysLAQFBSE2\nNlbSG3r+WF7xjjI3R1Z7WwcnyW3zM57J2rZgZycvlmrVZbVXKQtltS/My5XVXi658ec9TZPV/vnD\nx7LaOzduLKu9HE9+uyyrvau3l6z2CZ9+J6u9k3NVWe1fHeAtq719vbrlNyrBpqq9rPb5menyti9j\nX8u8eVPWtqu4OMtqryyQ9zlUyDytVK12HVntlfl5stoXPpf3nelQX3eP6UWk/vSj3s91f7OLweIw\nBEnLx9mVSBJ2MhMGERFVEmZakKOPcpNj//79ERgYiLfffhsAcOTIEXEeCRERUTFzrVbVh9aCnNGj\nR+POnTsYPXo05s2bB3t7e9jb2yMiIgKjRo0yZYxEREQmpbXn6O/vj1GjRmHgwIH417/+hZYtW5oy\nLiIisjSV4XqO77zzDnx9fbF69WoMGjQIAwYMgELxv47myJEjTRIgERFZBmsaVtV5ztHOzg729vbI\ny8tDVlaWWnIkIiKyVlqT4/Hjx/HZZ5+ha9eu2LVrF+zt5ZV2ExFRJVMZeo5r1qzBihUr0NiIc82I\niMh6CJXhnOM333xjyjiIiIjMhqRFAIiIiMpVGYZViYiI5LCmalWWnxIREZXCniMRERmGFfUcmRyJ\niMggrKlalcOqREREpbDnSEREhsFhVSIiolKsKDlyWJWIiKgUs+s55j5Ok9Xe1sFJcltlQYGsbSuM\nfRQkd/uFhTK3b9xjH8HGRlZ7ha289sZkW72aUbdfr6GzrPaPUjJltc/PyJbVvpqbUlZ72VQqo21a\n7n6mLJD3OVHYyfsaNPp+LPdza0a9NWua52h2yZGIiCwUq1WJiIisF3uORERkEIKRT+WYEpMjEREZ\nhhWdc7SeNE9ERGQg7DkSEZFBsFqViIioNFarEhERWS/2HImIyCA4rEpERFSaFSVHDqsSERGVIik5\nRkdH49mzZ8aOhYiILJmg0P9mZiQNqz569AgBAQF49dVXMWjQILz55ptWNbZMREQvTqhs1aoff/wx\nDh06hICAAOzatQtvv/02li1bhuTkZGPHR0REZHKS+7KCIKBOnTqoXbs2bGxs8OzZM0yaNAmLFy82\nZnxERGQpBEH/m5mRNKy6adMm7NmzB66urggICMD06dNhZ2cHpVKJt99+G9OnTzd2nEREZOas6XSb\npOSYnp6Or776Ch4eHmqPKxQKfP3110YJjIiIqKKUO6xaWFiI+Pj4MomxmKenp8GDIiIiC2RF1arl\nRmRjY4OXXnoJKSkppoiHiIgslKAQ9L6ZG8nDqn369IG3tzfs7e3Fx9esWWO0wIiIiCqKzuT4999/\n49GjR/joo4/UHk9MTESdOnWMGhgREVmYylKQExkZiU8++QRNmzZVe7xGjRpYvnw5Bg8ebNTgiIjI\nclhTtarOc46PHj0qkxgBoGnTprh7967RgiIiIqpIOnuOGRkZWn/3/PlzgwdDREQWzAyrTvWl8514\neXlh27ZtZR7//vvv0bx5c6MFRUREFkgh6H8zMzp7jmFhYQgJCUFsbKyYDC9duoT8/HysXLnSJAES\nERGZms7kWLt2bcTExODUqVO4fv06AKBz587o2LGjSYIjIiLLYU0FOZLmOXbo0AEdOnQwdixERGTJ\nKss5RyIiospIUs+RiIioPJVuWJWIiKhcHFYlIiKyXuw5EhGRQZjj1TX0xeRIRESGYUXnHDmsSkRE\nVAp7jkREZBCCFRXkMDkSEZFhWNGwqqBSqVQVHQQREZE5sZ4+MBERkYEwORIREZXC5EhERFQKkyMR\nEVEpTI5ERESlMDkSERGVwuRIRERUCpMjERFRKSZNjvv370dmZiYAYPXq1QgJCcHly5fLtJs2bRoA\nYNOmTZK3/ejRI4SFheFf//oXAODGjRv4/vvvtbbftGkTMjMzoVKpEBYWBj8/P/z8889y3o4k58+f\nR2xsLHbv3i3eSrt16xbef/999O3bFwBw5coVrF69Wud2Nb23zz//XGt7uX8fqbHrKzk5GXl5eQCA\n06dPY/PmzUhPT9fafv/+/ZIeA+S918LCQixatEhu+Dh37hyys7MBAHv27EFUVBTu3r0rezuaSP2c\n6Ovp06dlbvn5+Vrb5+XlITY2FmvWrMHKlSvFmzbZ2dlQKpUAivbthIQEndt/9OgRfvjhB/zwww94\n/Pix/m9MC03fI3K+W8ojd1+W216Thw8f6h8wSWLS5Lh69Wo4OjoiMTERJ0+eREBAACIiIsq0u3z5\nMlJTU7Fjxw48e/aszAdZk5kzZ6JTp0548OABAOAf//gHNm/erDWWHTt2wNHRET///DPS09OxePFi\nLF26tEy7+fPnY8GCBVpvukybNg2LFy/GuXPn8Ntvv+G3337DpUuXyrQLDw/HlClTYGtbtJpfs2bN\nsG/fPp3bPnToEPbu3SvenzdvHtLS0rS2l/v3kRp769at0aZNG603bSZOnAiFQoG///4bc+bMwb17\n9zBlyhSt7deuXSvpMbnv1cbGBufOndP6utpERETA3t4eV65cwYYNG9CwYUPMmDFDY9u0tDSsWbMG\n4eHhCA0NFW/aSP2cFLt16xZmz56N4OBgvPfee+JNG39/f3Ts2BE9e/ZEz5490bFjR3Tt2hV+fn4a\n/4/Hjx+PhIQE2NjYoHr16uJNm3fffRe5ublITU3FqFGjsGfPHsycOVNj23379mHw4ME4cOAA9u/f\nL/6sSWFhodbX1EXTQd2uXbv02pYmcvdlue01mTVrVpnHRowYgffeew+TJk2S/R6oLJOurWpjYwMA\nOHbsGAIDA9GlSxd88cUXZdoFBQXhgw8+wO3bt+Hv7y8+rlKpIAgCEhISyjznyZMn6N27t/iFaWtr\nC4VCe+4vXjXv2LFjGDBgABo3bgxNK+l5eXkBKOpF3bhxA7179wYAHDhwAJ6enjrf76VLl7Bv3z4I\n5aw3mJOTA29vb7XHiv9W2nz11VcYP348FAoFfvrpJzg5OSEyMlJre7l/H6mxJyUlAQC++OIL1KlT\nBwMGDAAA7N27V+fRrUKhgK2tLQ4fPox3330XI0aMwMCBA8u0O3bsGI4fP47U1FS1g5HMzEytfyO5\n7/WVV17BuHHj0KtXL7Uv/bffflvrc2xtbSEIAo4cOYLhw4dj8ODB2L59u8a2H374IXx8fNCxY8dy\n/18B6Z+TYh999BGCgoIQGBio830We/3119GzZ0+8+eabAICff/4Zhw4dgr+/P+bNm1eml52amor1\n69eXu91iKpUK9vb22L59O4YOHYrRo0eL+0Vpa9aswfbt21GrVi0ARQcSH3zwAXr16qXW7saNG1i2\nbFm5IyolxcXFIS4uDnfu3MG4cePEx7OyslCjRg3J2wGKPm8TJ07U+Dup+7K+7TXRdGD42WefASj/\nu4OkMWlydHd3x5w5c3DixAmMHj0aeXl54vBLScVHvnPnzsXQoUNx9uxZAEC7du3QrFkzjduuXr06\nnjx5In6ZX7hwAU5OTlpj8fLyQnBwMO7cuYMpU6YgMzNT4xeLn58fAODbb7/FN998I/bugoKCMHz4\ncJ3vt3Hjxnj48CHc3Nx0tnN1dUVycrIY+4EDB1CnTh2NbUv2nBcsWIAJEyagTZs2CAkJwdOnT+Hi\n4qLxeXL/PlJjL3b06FG1nuywYcPQv39/fPTRRxrb29raIi4uDrt378a///1vAEBBQUGZdu7u7vDy\n8sLRo0fRvHlz8XEHBwetvS+57zUvLw+urq44ffq02uO6kqODgwO+/vprxMbGYsuWLVAqlRrjB4oO\nfopPFUgh9XNSzNbWFsOGDZO8/V9//VXtQKNTp05YtGgRPv30U3G4r6TWrVvj6tWraNq0qaTtq1Qq\nJCUlITY2FgsXLgQArfGrVCoxMQKAi4uLxoPUqKgoLFmyRNLrl4y7Tp06ePLkCYKDg8XHHRwcJL+X\nYiX3vdKk7sv6tpfKw8PjhbdB/2PShcdzcnLw008/oUmTJvjHP/6BBw8e4Nq1a+jUqZPG9ps2bcL2\n7dvRo0cPqFQqHDlyBIGBgRgxYkSZtpcvX8b8+fNx/fp1NG7cGE+ePMGKFSu0JlOlUok//vgDDRo0\ngLOzM548eYLU1FSt7Xv27InvvvtOTD7Pnj1DYGAgDh48qPX9jhgxAleuXIG3tzfs7OzEx9esWaPW\n7vbt2wgPD0dSUhKcnZ1Rv359LFmyBPXr1y+zza5du0IQBLEXXfK/T1uvWs7fp/gIOysrS1LsxYKC\ngjBs2DD06dMHgiAgLi4O33zzDWJiYjS2v3HjBmJiYtCqVSv07dsXt2/fxv79+zFmzBiN7QsKCsQD\nk/LI3Rf08fDhQ8TFxaFFixZo27YtUlJScObMGY09gOXLl6NNmzbo3LmzpG1L/ZwUHyhFR0ejZs2a\n6NGjB6pUqSL+XtuBUnBwMDp06IA+ffoAKBra/OWXX7Bu3ToEBASUGXLs3bs3/v77b9SvX19t+7Gx\nsRq3f/bsWaxfvx5t2rTBmDFjcPv2bWzatAmzZ88u03bx4sW4evWqWixNmzYtczBRWFiod4/o9u3b\ncHNzQ9WqVQEAz58/x6NHjzR+vvQhd1+W216q4u+GmjVrlltPQOUz+VU5EhMT8ffff2PQoEFIS0tD\nVlYWGjRooLFtv3798N1334lDXdnZ2RgyZIjWD2VBQQFu3boFlUqFl156Se1LXZNDhw7h3LlzEAQB\nPj4+6NGjh9a2O3bswMqVK/Haa69BpVLh7NmzmDhxotiz1OTMmTMaH2/fvr3Gx4sLGRwdHXXGrQ+V\nSoXCwkK1v49KpVL7stMVczFtsd+5cwcLFy7E+fPnIQgC2rRpg7CwMK1fQCdPnkTr1q1RrVo1SfEX\nf/BL03YwIGdfuH//PubPn4/z588DANq2bYtZs2ahbt26kmIrT+vWrZGTkwM7OzvY2dmJBzbFr6eJ\nlM9JyQOl0nQdKKWlpWHVqlXiudbikQdHR0fcu3cPjRo1Umt/9+5dpKenIzExEUDRCI6Tk5PGnkph\nYSE+//xzredfS1uyZAlatmwpxtK2bVtcuHBBY09b3wTp7++PmJgYcV/Py8vD0KFDsWPHDsnb+OGH\nH/DWW2/Jfm2yXCZNjitXrsSlS5dw69YtHDx4EKmpqfjoo4+09i769euH7du3i0d8ubm5CAgI0Joc\nz58/j7t376qduNc2lh8REYHk5GS1I9aGDRti7ty5WuN/+PAhfv31VwBAy5YttQ59ypWXl4eDBw/i\n7t27asMrISEhWp+zdetW9OvXD87OzgCKerJxcXFah3pDQ0MRFRUl3s/Ozsb48eO1Vu0tWbKkzBeU\npsf0NWPGDFy4cAE1atSAj48P2rVrBx8fH63ngp48eSL+nJeXh/379+PZs2dah23l7AsjR45E3759\n1c6XxsbGYsOGDWXamuLoXO7nxNjkjOAAQGBgILZt2yZp235+fmV6qv369SvzGdfnnGOxAQMGYM+e\nPWqP9e/fX+00QHm+/PJLrYUuUg/c+vXrp/M1tH2vUcUw6TnHw4cPY/fu3WJvy93dHVlZWVrb+/v7\nY/DgwWKP7siRIxg0aJDGttOmTcPt27fRrFkz8ehSEAStX4inTp3C/v37xZ3az89PLLbRRKVS4Zdf\nfsHt27cREhKClJQUXLx4sUwhDQAMHToU3377LVq3bq32odHWYxg/fjycnJzQvHnzMj05bbZt26aW\nCGvUqIHvv/9ea3J0d3dHREQEIiIi8OzZM4wdOxaDBw/Wuv1ffvmlzGPHjx/Xmhxv3bqFiIgIPH78\nGHFxcbhy5QqOHj2KDz/8UGP74ukTqampOHjwID799FM8ePAAv//+u8b2rq6uavc/+OAD+Pv7a0yO\ncveFtLQ0tf3K399f60HD0aNHNT6ui0qlwt69e3Hnzh1MmDAB9+7dw8OHDzXuO4D8z4ncA6Vbt27h\nv//9b5mDMW0Vvdu3b1cbwRk9ejSGDBmiNTlKKXD65ptv8O233+L27dtqSSMrK0tjlbM+5xyL1axZ\nEwkJCejWrRuAou+R0vtTeYYOHar1dyV7oCUP3EorPiWxdetWAFA7GCuv8I1Mz6TJ0c7ODoIgiDtC\n8TwxbUaOHIn27duLQy5RUVF49dVXNbaVWl1ZrFGjRkhJSRGHhu7du4d//OMfWttHRERAoVDg1KlT\nCAkJgYODAyZOnKhxaObbb78F8L9KzvLIrQYEis6ZFidboGjISddcssmTJ2Px4sWYM2cOLl++jDFj\nxqBnz55l2sn90ioWHh6O6dOnY86cOQCKpqNMnTpVa3Lcs2cPzp07h2vXrsHFxQXDhw9H27ZttW6/\n5Dw/pVKJS5cuaS1ikLsvuLi4YM+ePeI807i4OK3n6/RRct+ZMGECqlevjnnz5mkd1pP7OZF7oFRc\n3Tp48GBJ1a2AegVkeUObUgqc+vXrB19fXyxbtkxtGoODg4PGv/3atWv1Puc4b948TJ06FZ9++ikE\nQUDdunVlz22dPXs2vv76a42/k3rgVvxd88svv6hNL5k2bRr8/PwwdepUWTGRcZk0Ob7zzjuYM2cO\n0tPTsW3bNuzYsQOBgYE6n9O8eXOdlWLF5FZXZmVloXfv3uLR+2+//QYvLy+xIKV04cnFixexa9cu\nsfdRo0YNnclIDrnVgEBRheHkyZMRFBQEAIiJiRFL80s6dOiQ+HPLli2xevVqeHt7QxAEHDp0qExF\nptwvrWJyp6NERkaiYcOGCAoKwmuvvVZucURxmTpQVO3n4eGhdXqD3H0hMjIS8+fPR1RUFARBQOvW\nrdVe70XJ3Xfkfk7kHijJrW6VM4IDQG34XhsnJyc4OTlh2bJlkmJ4kekJDRs2xLZt28Tet4ODg+xt\naEuMgLwDN6BoJOHcuXPw8fEBUHQKQFc1MlUMkybHUaNG4cSJE3BwcMCtW7cwadIkvPHGGy+0zZLV\nlX369JFcXSl3oqytrS0KCwvFL6C0tDTJR93a9O3bF4IgoLCwEDt37pRcDQgUHW3GxMSIvdTXX39d\n4zDpDz/8oHb/1VdfRUFBgfh46eQoCALq168v9gBL0jVVRM50FKBoZZDr16/j7NmzWL58Of7++2+8\n9NJLWofOoqOjtW6rmL77wv3798v87ty5c6hXr165rymF3H1H7udE6oFSsbfeegtbt26VXN0qZwQH\nkD/EbmyPHj3CsmXL8ODBA6xbtw43btxAUlKSztMKcsg5cAOAhQsXIiwsTFyhy9nZWeccZaoYJi3I\n2bBhA3r37g13d3eDbfPMmTNQqVT4/PPP1c6HFT9mqKKJvXv3Yt++ffj999/h5+eHAwcO4KOPPtJ5\nnrI87dq107kkW3nzlp4/f46UlBT885//1DuG0saOHYuvv/5aYyWkrgpIOdNRgKJJ/OfOncPZs2eR\nmJiIJ0+eoFWrVlqHuzIyMrBy5Upxzmv79u0xYcIEtfmL+u4LmopCND2mL037zuTJk/HOO+9ofc7D\nhw9x8eJFCIKAFi1a6DzQUCqViImJwalTpwD870BJW2+ra9euZR7T9X8r17vvvisOsRfv33379kVc\nXJxBti/Xv/71L/j7+2PNmjXYu3cvCgoK4OfnV+EFMBkZGQCgcw4uVRyT9hyzsrIQHByMGjVqoHfv\n3ujVqxdq1679QtssnlpQUFBQZprB8+fPtT6vZLFMfn4+CgoKYG9vr7W8vn///mjevDlOnToFlUqF\n1atXl7tCTnnq16+v98TdhIQELF68GPn5+Th69Cj++OMPrFixQmfvSMp0heLhozZt2qB9+/bw8fGR\n9D4bNGiAjRs3Sp6OMnToUPj4+MDHxwfvvvtuudMmwsLC0LhxY6xYsQJA0TnL0NBQtTU+5e4LSUlJ\nSEpKQlpamlplamZmpt5LlWkid9/5/vvvsWrVKnTo0AEqlQoLFizAhx9+iICAAI3tFQoF/P390aFD\nB0kHSvoUFcmhz4pPxiR3xSS5pBy4laRPdTqZnkmTY0hICEJCQnDlyhXs379f/FLcuHGj3tvUt4Ck\nZLGMSqVCQkICLly4oLX9tGnTsGTJErUvteLH9PX48WON0wWKjRw5UuvvVq1ahe3bt4sVg6+88orO\nha9DQ0PRt29fMbns3bsXoaGhWl8/ICAAiYmJWLBgAZKTk9G8eXP4+Pjg/fff19he7tBV8VF7Tk4O\n7O3ttcZdLDk5GV999ZV4PyQkpMySZHL3hfz8fGRnZ6OwsFCtGtTR0RFffvlluTGVp+RqRrVq1RKn\nDRX/Ttsw5rp167Br1y6x0OPJkycICgrSmhylHiidPHkSHTt2VDsPXZKuFYHkkDvEbmxyV0ySS8qB\nW0n6VKeT6Zk0ORarVasWateuDRcXlxdehV/fApKSBEFA9+7dsXLlSq0VYzdu3FC7X1hY+MJXSlAq\nlTpL9HWxtbWV9QGXM10BADp06IB27drht99+w+nTpxETE4Nr165pTY4zZ84Uh66AosW+P/74Y63J\nMSkpCbNmzUJ2djZ+/PFHXLlyBTExMVoX2K5WrRoSExPFitZz586VWUBA7r7Qvn17tG/fHn5+fkZZ\nesvf318cmr5375441SI9PR316tXT2oNzdXVVKxpxcHDQOfVA6oHS2bNn0bFjxzLnoYsZKjnOnTsX\n4eHhuHnzJt58801xiL2izJw5E+PHj0dycjKCgoLEFZMMRcqBW0n6VKeT6Zk0OW7duhUHDhxAWloa\nevXqhQULFuDll19+oW3KrXorVvLoubjCrHixgZK+/vprrFmzBrm5uWLvo3hlmfIqbctTp04dvYdS\nXn75ZcTGxqKwsBB//fUXoqOj0bp1a63t5U5XeP/995GTk4NWrVqhbdu2aotDayJ36CoyMhLr16/H\n+PHjARRN/ShegUWTiIgIzJgxQyxiqFGjRpmKUn33hdmzZ2PFihVq8wQ/+eSTF/4CK05+s2fPRo8e\nPcTl444dO6bx/F5xL75hw4YIDAxEt27dxHOBuiqZpR4oFRehSakmfRFyh9iNrXHjxtiyZUuZ1aEM\nRcqBW0n6VKeT6Zk0Od6/fx9hYWF45ZVXTPmyGpU8eraxsYGHh4fG1TfGjh2LsWPHYunSpbIvK1Oe\nF/mAhoeHY82aNahSpQqmTJmCTp066awGLDldASg6p6jrS7Jp06a4fPkyrl+/LiYdXcu96TN0Vboa\ntLwrZ+zdu1e8zqEhv3CfPHkiJkagaKqFIa8rWHqh786dO2u89mbxKELDhg3RsGFD8fHiyevaSD1Q\n0jWED+gexpfD3M6pDRkyBLt27ULjxo3FxwxZcDVv3jxMnz5d54FbSefOncOuXbvg4eEhuTqdTM8k\nyTEzMxOOjo4YNWoUAJS5JqMhJ1xLJffo2dvbGxkZGeIXfnp6Os6cOYPu3bvrHYO+51oLCwvx5Zdf\nYsaMGfj4448lPcfDw0NrsY4mYWFhAIr+73bt2oWwsDA8fPhQ4/X+APlDV/Xq1RPXYc3Pz8fmzZt1\nFqk8efJEXA+0eO3WCRMmyF7pRBOFQoGUlBT83//9H4CidWINuWKJm5sbVq9ejf79+wMo+hLUNAez\ndPKQOi9P6oGSvkP4cpnLObWHDx8iNTUVz58/x++//y4ejGZmZiInJ8dgr9OsWTNZB27/+c9/DPba\nZDwmmcqh7/QAY5K72LSm9RkHDhyocyqGMclZvxIommqxcOFCXLhwAYIgoFWrVggLC9O66PuWLVuQ\nmJiIy5cvw8PDA23bthWvSaiNnMW+09LSsHDhQpw8eRIqlQpvvPEGZs2apTXZjRw5Em3btlVLMGfO\nnHmhYq5ix48fx5w5c9CuXTtxgvann36qc66gHE+fPsXKlSvFYeO2bdsiJCRE60HhtWvXMH36dHEJ\nMldXVyxatEit56OJ1OImY6vIaRsl7dq1Czt37sSlS5fE67ICRQcb/v7+BjvHqm9P+fHjx8jNzRXv\nFx+ckXkw+VU5zIWcxaYBzYsha3rMVObOnYvU1FTJF+gNDAzEsGHDxHOO8fHx2LJli9a5f+vXr0fb\ntm3RvHlznZeKWrZsGT755BMAwIkTJ154UQdtNH3hGvLv//jxY3z33Xd49dVX8fz5c9SqVQvt2rUz\nyLblCgoKwuTJk9GhQwcARQsmLF++XOvC4+fPn8fs2bPLLW4qObSriaZLSukjPDwc7777rtmcUzt4\n8KDGpRINZdSoUWJPueSUlZLXkCwpISEBixYtwoMHD1CzZk2kpKTA09MT8fHxRouR5DPpOcf333+/\nTIWkpsdMQW71ppeXF6KiosT1Krdu3SppWTtjkXuB3pycHLWFtwcMGKCz4KR4CLw8P/30k5gcP//8\n83KTo7bydqBoFGHChAkaf/fGG28gPj5enDh/4MABrdcBlev777/H5s2bcf/+fTRr1gy//vorWrVq\npXUhbqlKXn1eE23D3NnZ2WJiBIDXXntN5/qqUVFRkoqbTLW/mts5tY4dOyIqKkryPES55Fafrlix\nAt999x1GjhyJ3bt349SpU7KuEEKmYZLkmJubi5ycHDx58gTPnj1TG/tPTU01RQhlyK3eDA8Px+rV\nqzF58mQARV/WmpZYM5WIiAiN1bXa+Pr6Yu3atejduzcEQcC+ffvQuXNn8fyvqc77luzlFsvOzsaO\nHTvw9OnTMsmxeLEGlUqFTZs2Yfr06QCKzrtWr15d8nUDddm8eTO2b9+OwMBAREdH488//8Ty5ctf\neLvaeg7ladCgAVatWqU2qqFt+LuYlOImXdceNSRzO6c2a9YsWfMQ5ZJbfWprawtXV1colUoolUp0\n6NCBy8eZIZMkx5iYGGzatAkPHjyAv7+/mBwdHR3x7rvvmiKEMjQtNq2rSKd69epmtWp+3759UatW\nLcNWZfIAABMYSURBVLRt21Y8H6jrSHj//v0Aiv4viotNVCoV4uPjX+i8b/FCBiqVSuOiBqUrIEsm\njMzMTGzevBk7d+5E7969NSaTkos1PH36FH///bfaeRpDqFKlinigkZeXB09PT9y6deuFt1tylZ68\nvDz89ddfAFDu+djIyEh89dVXmDhxIgDAx8dH55en3OKmtLQ0/Oc//8GNGzfU/pYv2lMu5uHhofFi\nzRVF7jxEueT2lJ2dnZGVlYV27dph6tSpqFmzpsaDRqpYJj3nGB0drfUacKZUWFiI6OhofPDBB+W2\nXbhwIWbNmqV1iExOBaihpaSkIDExEefPn8fx48fh5ORUpmio2L59++Dr6wtHR0esWrUKv//+Oz78\n8MMXHmor7+hbU1HC06dPsWHDBsTGxsLPzw/vvfee1oscF9M09Nm6dWuDDMlPmDABUVFR2LRpE06d\nOgVnZ2cUFBQYrAd0+vRpzJw5Ex4eHuKCAIsWLTLYOc2SxU1KpRKdOnXSWdwUHByMd955B//9738x\nb9487Nq1CzVr1jTYhazN7WLNQ4YMwbRp09TmIS5evBjfffedQbavbWUqbQtLZGdno2rVqlCpVIiN\njUVGRgb69etnkMprMhyTF+Rcu3YNN27cQF5enviYtovQGlNAQAC2b99ebrviSrczZ85o/H3pNTxN\n5f79+0hMTMSZM2dw9epV1KhRAz4+Phg7dqzG9sXFK4mJiVixYgVGjRqFVatWGWxh9nv37pUZ2nv4\n8GGZZcMWLVqEw4cPIzAwEMOHD5d8+aB+/fqJQ5979uwRhz4NNTRW7MyZM8jIyMCbb75psGkI/v7+\n+Pzzz8V1T2/duoUpU6Zg586dau30PUepTzw7d+5UK2gaNGiQ1utLyjVgwADxYs3F1dwVWbx25coV\ncR4iUNRz++yzz9CsWTODvYamnnJ5Q+Fk3kxakLNy5UqcPn0af/75Jzp37ozjx4/Dx8enQpJjmzZt\n8Omnn6J3795q5e+le1LFJeDt27dHXl4ebt68CUEQ8NJLL1XoHK4uXbqgRYsWGDt2LD799NNy2xdX\n0R07dgyBgYHo0qWLzsvqyNW9e3f07NkTCxcuFP+eY8aMKTPResOGDahSpQr+/e9/q33ZF1+PUNvC\n78Ya+izNGAc7+fn5aguCv/TSSxqvt6hpWLnkELg2cqfpFFcfu7m54ccff4Sbm5vGK9frS+7Fmo1J\nqVTi5s2bRltAAlDvKQ8aNAj5+fniJeVKKnmxg5LK2/epYpg0OR48eBB79uzBwIEDERUVhUePHhls\nKEeuP/74AwDEBaaLd1Bt511+/PFHzJ07Fw0bNoRKpcKdO3cwb948cUkwU9u9ezfOnTuHuLg4/Oc/\n/0GjRo3Qrl07rWuZuru7Y86cOThx4gRGjx6NvLw8g15gtXHjxvDx8cGwYcOwYsUK8e9U2pUrV/Ta\nft26dZGeno7u3btj5MiRcHZ2tph5YV5eXpg1a5Y4R3Pv3r1q8+6KpaenIzU1VayIDggIQFpaGgRB\n0Hm+e8qUKRg2bJjYi46Pj8cnn3yidVRg/PjxyMjIwIwZMzB//nxkZWWJiz4YgqaLNRvq2olyKRQK\nrFu3Dr179zbaMnaHDx8We8pA0WdN0znWkufPyfyZNDlWrVoVCoUCtra2yMzMRK1atXDv3j1ThiB6\n66231BYkEAQBjo6O+OOPPzQub/fZZ59h8+bNaNSoEYCik/xjxoypsOTYrFkzNGjQAA0aNMC5c+ew\nd+9enD17VuuX0BdffIGffvoJwcHBcHZ2xoMHD8TKT0MQBAHDhw9Hs2bNMG7cOEydOtWgq8ysWrUK\nADBx4kS89tpr4tCnJZg3bx62bt2KLVu2AIB4EFHaunXr1Kpk8/PzsWPHDuTk5CA0NFTr9R/lTtM5\ncOAAfHx80KRJE0RHR+Pp06dYtGiRxus86sMYFzV/Ea+//jrWr19fZpTIUBXa5tRTJsMxaXL08vJC\neno6Bg8eDH9/f1SvXl3nYtnGdPnyZVy6dAldu3aFSqXCDz/8gKZNmyImJga9evXC6NGj1do7ODiI\niREoKreXer7MGPz9/ZGfn4/WrVvDx8cHW7Zs0XllCXt7e7U5kG5ubhqXMNNX8UGGj48PNm7ciMmT\nJ+PmzZsG235JFXWeV64jR46IPcGRI0ciPj4eaWlp+P3331G3bl306tVLrX1+fr7aeVsfHx+4urrC\n1dVV53JncqfpXL16VW0tWRcXF3EkxRCWLFmCadOmqSXE4scqwr59+yAIAr755hu1xw21MpemnvKL\nXpSAKl6FrZBz584dZGZmGvSkuBzDhw/H2rVrxQSXlZWFsWPHYt26dfD398e+ffvU2s+dOxcpKSl4\n5513IAgCDhw4gHr16uH1118HYLjL/UiVlpaGmjVrmvQ1dSldfFNQUICkpKQKW2XGHAQFBWH58uVi\nwhswYAA2bdqE7OxshIaGlqm07dGjBw4fPqxxW927d8eRI0c0/q5kj6/0OUpN03T69++P6OhosUL4\n6dOnGDFihMEKZjQt6l2RBTnPnz/HN998I67L27ZtWwQFBem8coZcJ06cwM8//wwA6NSpU4X2lMkw\nKmyFnPr165d5zJQeP36sVlBjZ2eHR48eoVq1ahoLbfLy8lC7dm1xlY2aNWsiNzdXvLqHqZOjo6Mj\nYmNjzebKB76+vhg1ahSmTJkCQRBga2uLyMhIg135wBJp6gm6uLjAxcVFY0/Q29sb27ZtK9PriImJ\ngbe3t9bXmTp1qqxpOsHBwRgyZIjYcz1w4EC5lbJS6HvhcWObMWMGHB0dxWlkcXFxmDFjhkGv6fjG\nG28wIVqZSrtCTr9+/cRr5gFF197r27cvsrOzNU6gNvY18OQylysfFHv55ZehVCoRHByM5cuXw8XF\nxaDXzLNE6enpavdLrqiUlpZWpn1YWBgmTJiA2NhYMbFdvnwZeXl54jlXTf7973+jd+/eSExMxKlT\npzBq1ChERERoLcgZOHAgvLy8cOrUKQBF1ZYvel1VwDAXHjeG69evq40EdejQAb179zbY9jVVoTo5\nOcHLywszZ87klA4LVWlXyJkwYQJ8fX3F8ul58+ahRYsWAIClS5eWaX/r1i1ERETg8ePHiIuLw5Ur\nV3D06FGd11A0JnO7mritrS2mT5+Offv2Yfjw4Vi0aJFBC3IskdyeYK1atRATE4OTJ0/ixo0bAIqu\n/ajrSiiAftN0Xn75ZYMkxJKKr/s5efJk1KlTB1WqVMHp06dx9epVDBw4UO08pym9+uqruHDhAlq1\nagWg6PqamqqF9fX++++jbt26aov6Jycno3nz5ggLC0N0dLTBXotMp1KukKOPd999F9OnT8ecOXPE\nic0VeWkec7vyQcnLd127dg1TpkzBvXv3NC6AXVk8fvwYEyZMgN3/a+9uQqL6wjCAP6NlToohVkoa\naJD4EWJGUSYGVovJMhFbSEmLSNGFVmZQOZVKVGRBEphpEJWa+reoTMUoWii2SQ2LggzJSQUzP8bP\nqWbufxFe1LTJ0nNnmucHgo7SfQnlnXPvOc+7cOG0K8GlS5fOyXUSExPh7u6O+vp63L9/H46OjoiN\njVUszHr37t2oqKhAR0cHEhISEBERgdbWVsUyVzUaDdra2uSjP52dnfDx8ZHPe/7ts9CoqKif/q/H\nR9xN9z2yDkKfOcbHx6OxsREdHR0wGo3y60qEAMzW6OjoT+/2J46nEc3SJh9MHIfk6+uL4uJiReZ0\nWpI/XQnO1nwf05mt8eNatbW12LdvH+Lj4xX9Gy8sLJzXf1+tVqOqqmrSM9zxwApbv3tizYQ2x/T0\ndOh0Ovj5+cmNRaVSWUVzdHV1RXt7u/zLXlNT81M0mkiWNvmgtbVVbgA02aZNm+a8IU4038d0ZmvB\nggWorKzEgwcPkJeXBwCTNo2J9qsjTnMhJycHZ8+eRWZmppxQdPHiRYyNjUGr1c7rtWn+CL2tqtFo\n5DNH1kan00Gr1aKpqQkuLi7w8vJCTk7OvP/hTTU0NARnZ2f5DNtUSm18yM7Olj83GAxoaGhAYGCg\nnEBEtqO1tRV3795FcHAwdu7cCZ1Oh+rqaiQkJChdGtFvE9ocU1JSkJGRoei72tmaOoJpbGwMJpNJ\nHjEzdSTTfEtMTER+fj4iIiImJfwA059pU4per8fhw4ctatMQ0VwqKCjAwYMHkZ2dPe0b/oyMDAWq\norki9LZqX18fIiMjERQUNGmenZJjn8wZz0hsa2tDS0sLtm7dCkmS8PDhQ3l3q0j5+fkAfhw9sWRq\ntRqfPn1SugwSKDU1FVeuXJl0xnEipZ6Hz5fxI19zufOVLIfQlaOljX2ajb179yI/P18OLx4aGkJi\nYiKKiooUq2lgYOCn4b9KJdJMPEQuSRJaW1uh0WgsakA0za/u7m4sX7581vMNiSyR0JXjhg0b0NHR\ngY8fPyI0NBSjo6OTdq1asp6enkm7Qh0cHNDT06NYPdMN/w0ODp6zae6zNXHckr29PTw9PeHh4aFI\nLaSM8cclttIERc3fJGUIbY5lZWUoLS3FwMCAHMp8+vRpReLjZis6OhqxsbHYvn07gB+h0jExMYrV\nc+vWLXn47+3bt+Xhv0qZuPrv7e3lVHMbZGvzCsffENbW1qKnp0ceSfb48WO4ubkpWRrNAaHNsaio\nCOXl5XJiiLe397QxWpYoKSkJ4eHh8qH2c+fOISAgQLF6RA3/Nae5uRmXLl3CkiVLkJycjGPHjqGv\nrw8mkwkXLlxAeHi48JpIGbY2r3D8DeH58+dx7949+fWIiAhF3zjT3BDaHB0cHCbdmlTy7NOfCAwM\nnDHMWTRLGf6blZWFI0eOYHBwEPv370dBQQGCg4Px4cMHpKWlsTnSP290dBQ6nU7OUNXpdL8cMUbW\nQWhzXL9+Pa5du4axsTHU19ejuLh4zgas2hpLGf5rNBoRFhYGAMjNzZXzK6cLbyf6Fx0/fhzx8fFY\nuXIlJElCZ2cnMjMzlS6L/pLQ5nj06FH8999/8PX1RWlpKbZs2TLj5HqamdFoRGRkJGpqagAou9vX\nzs5O/nzqfDxrDHsgmq3w8HDU1tbKw71XrVplEZNy6O8oNuy4v79f3mlJs5eUlAStVqvIrdSJ/P39\noVarIUkSDAaD3CAlScLXr1/x5s0bResjEsFaM6NpZsKDx/Py8vD9+3fExMTAzc0Na9euxYkTJ0SW\n8U/Q6/VyoIJarZZfF719/O3bt0KvR2RprDkzmmYmtDkODg7C2dkZ5eXliI6ORkpKyoxpGvRrBoNB\nTssBfqzUcnJyFKyIyDa9fv3aajOjaWZCm6PRaER3dzeqq6tx6NAhkZf+5xiNxp+eNY6NjSlUDZHt\nWr16NT5//mxVmdFkntDmmJycjAMHDiAkJARBQUHQ6XTw9vYWWYLVKy4uRklJCXQ63aRV9/DwMEJC\nQhSsjMg2WWNmNJmn2IYc+jODg4MYGBjA5cuXkZaWJr/u5OSk2LgqIltmzZnRNDOhzbGtrQ1nzpzB\nly9fUFlZiXfv3uHZs2dITk4WVQIREZFZduZ/ZO5otVqkpaVhwYIfd3P9/PxQVVUlsgQiojkRFxcH\n4EembEhIiPwx/jVZN6HPHEdHRxEUFDTptfGtz0RE1qSkpASA7WXK2gqhK0dXV1e0t7fLW55ramqw\nbNkykSUQERGZJfSZo06ng1arRVNTE1xcXODl5YWcnBybmf9GRETWQdhtVZPJhJaWFty8eRMjIyMw\nmUxwdnYWdXkiIqLfJuy2qp2dHQoLCwEAixcvZmMkIiKLJfSZY2hoKG7cuIGuri709/fLH0RERJZE\n6DPH6WY3qlQqPH36VFQJREREZjEhh4iIaAqh5xwNBgOKi4vx8uVLqFQqrFu3DnFxcVi0aJHIMoiI\niH5J6MoxNTUVTk5OiIqKAgBUVlZCr9cjNzdXVAlERERmCV05vn//flJc3MaNG7Fjxw6RJRAREZkl\ndLdqQEAAmpub5a9fvXqFNWvWiCyBiIjILKG3VTUaDdra2rBixQoAQGdnJ3x8fOQg8kePHokqhYiI\naEZCm2NHR8cvv88YOSIisgRCb6t6enqiq6sLL168gKenJ9RqNUwmEzw9PdkYiYjIYghtjlevXkVh\nYSGuX78OAPj27RvS09NFlkBERGSW0Ob45MkT5OXlQa1WAwDc3d0xPDwssgQiIiKzhDbHhQsXQqVS\nyfMcR0ZGRF6eiIjotwg956jRaHDq1Cno9XqUlZWhoqICe/bsEVkCERGRWcKzVevr61FXVwcACAsL\nw+bNm0VenoiIyCxFg8dNJhMqKyvlODkiIiJLIOSZ49DQEPLz85GVlYW6ujpIkoQ7d+5g27ZtqK6u\nFlECERHRbxOyckxKSsKSJUsQHByMhoYG9Pb2QpIknDx5Ev7+/vN9eSIiolkR0hx37dolR8MZjUaE\nhYXh+fPnHFVFREQWScht1fHsVACwt7eHh4cHGyMREVksIStHf39/+eC/JEkwGAxwdHSEJElQqVRo\nbGyc7xKIiIh+m6K7VYmIiCyR0IQcIiIia8DmSERENAWbIxER0RRsjkRERFOwORIREU3xPxL/B6Mc\ndTjeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12ea3d590>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.heatmap(df);"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}