train.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FN = 'train'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "you should use GPU but if it is busy then you always can fall back to your CPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['THEANO_FLAGS'] = 'device=cpu,floatX=float32'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.0.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use indexing of tokens from [vocabulary-embedding](./vocabulary-embedding.ipynb) this does not clip the indexes of the words to `vocab_size`.\n",
    "\n",
    "Use the index of outside words to replace them with several `oov` words (`oov` , `oov0`, `oov1`, ...) that appear in the same description and headline. This will allow headline generator to replace the oov with the same word in the description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FN0 = 'vocabulary-embedding'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "implement the \"simple\" model from http://arxiv.org/pdf/1512.01712v1.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "you can start training from a pre-existing model. This allows you to run this notebooks many times, each time using different parameters and passing the end result of one run to be the input of the next.\n",
    "\n",
    "I've started with `maxlend=0` (see below) in which the description was ignored. I then moved to start with a high `LR` and the manually lowering it. I also started with `nflips=0` in which the original headlines is used as-is and slowely moved to `12` in which half the input headline was fliped with the predictions made by the model (the paper used fixed 10%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "FN1 = 'train'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "input data (`X`) is made from `maxlend` description words followed by `eos`\n",
    "followed by headline words followed by `eos`\n",
    "if description is shorter than `maxlend` it will be left padded with `empty`\n",
    "if entire data is longer than `maxlen` it will be clipped and if it is shorter it will be right padded with empty.\n",
    "\n",
    "labels (`Y`) are the headline words followed by `eos` and clipped or padded to `maxlenh`\n",
    "\n",
    "In other words the input is made from a `maxlend` half in which the description is padded from the left\n",
    "and a `maxlenh` half in which `eos` is followed by a headline followed by another `eos` if there is enough space.\n",
    "\n",
    "The labels match only the second half and \n",
    "the first label matches the `eos` at the start of the second half (following the description in the first half)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "maxlend=25 # 0 - if we dont want to use description at all\n",
    "maxlenh=25\n",
    "maxlen = maxlend + maxlenh\n",
    "rnn_size = 512 # must be same as 160330-word-gen\n",
    "rnn_layers = 3  # match FN1\n",
    "batch_norm=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the out of the first `activation_rnn_size` nodes from the top LSTM layer will be used for activation and the rest will be used to select predicted word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "activation_rnn_size = 40 if maxlend else 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# training parameters\n",
    "seed=42\n",
    "p_W, p_U, p_dense, weight_decay = 0, 0, 0, 0\n",
    "optimizer = 'adam'\n",
    "LR = 1e-4\n",
    "batch_size=64\n",
    "nflips=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nb_train_samples = 30000\n",
    "nb_val_samples = 3000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# read word embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import cPickle as pickle\n",
    "\n",
    "with open('data/%s.pkl'%FN0, 'rb') as fp:\n",
    "    embedding, idx2word, word2idx, glove_idx2idx = pickle.load(fp)\n",
    "vocab_size, embedding_size = embedding.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('data/%s.data.pkl'%FN0, 'rb') as fp:\n",
    "    X, Y = pickle.load(fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_unknown_words = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of examples 684114 684114\n",
      "dimension of embedding space for words 100\n",
      "vocabulary size 40000 the last 10 words can be used as place holders for unknown/oov words\n",
      "total number of different words 523734 523734\n",
      "number of words outside vocabulary which we can substitue using glove similarity 122377\n",
      "number of words that will be regarded as unknonw(unk)/out-of-vocabulary(oov) 361357\n"
     ]
    }
   ],
   "source": [
    "print 'number of examples',len(X),len(Y)\n",
    "print 'dimension of embedding space for words',embedding_size\n",
    "print 'vocabulary size', vocab_size, 'the last %d words can be used as place holders for unknown/oov words'%nb_unknown_words\n",
    "print 'total number of different words',len(idx2word), len(word2idx)\n",
    "print 'number of words outside vocabulary which we can substitue using glove similarity', len(glove_idx2idx)\n",
    "print 'number of words that will be regarded as unknonw(unk)/out-of-vocabulary(oov)',len(idx2word)-vocab_size-len(glove_idx2idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i in range(nb_unknown_words):\n",
    "    idx2word[vocab_size-1-i] = '<%d>'%i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "when printing mark words outside vocabulary with `^` at their end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "oov0 = vocab_size-nb_unknown_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(oov0, len(idx2word)):\n",
    "    idx2word[i] = idx2word[i]+'^'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(681114, 681114, 3000, 3000)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=nb_val_samples, random_state=seed)\n",
    "len(X_train), len(Y_train), len(X_test), len(Y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "del X\n",
    "del Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "empty = 0\n",
    "eos = 1\n",
    "idx2word[empty] = '_'\n",
    "idx2word[eos] = '~'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.preprocessing import sequence\n",
    "from keras.utils import np_utils\n",
    "import random, sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prt(label, x):\n",
    "    print label+':',\n",
    "    for w in x:\n",
    "        print idx2word[w],\n",
    "    print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H: Behold the Fastest CF Card Ever Made ( For Now )\n",
      "D: Are you worried your compact flash card wo n't be able to keep up with the demands of the upcoming 4K revolution ? Toshiba 's got your back with its new Exceria^ Pro series CF cards that boast read and write speeds of 160MB/s^ and 150MB/s^ respectively , making them the world 's fastest -- for the time being .\n"
     ]
    }
   ],
   "source": [
    "i = 334\n",
    "prt('H',Y_train[i])\n",
    "prt('D',X_train[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H: Obama : Bad for Black People\n",
      "D: Are you one of the 48 % of Americans who is `` hearing too much about Barack Obama '' ? Then you certainly wo n't like this Sunday 's story by professional Democratic Party Underminer Matt Bai^ . It 's about how Barack Obama represents the End of Black Politics , because he 's a black person who white people do n't feel threatened by . In the story , Bai^ harangues^ Philadelphia mayor Michael Nutter^ about why he di\n"
     ]
    }
   ],
   "source": [
    "i = 334\n",
    "prt('H',Y_test[i])\n",
    "prt('D',X_test[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation, Dropout, RepeatVector\n",
    "from keras.layers.wrappers import TimeDistributed\n",
    "from keras.layers.recurrent import LSTM\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.regularizers import l2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# seed weight initialization\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "regularizer = l2(weight_decay) if weight_decay else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "start with a standaed stacked LSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(vocab_size, embedding_size,\n",
    "                    input_length=maxlen,\n",
    "                    embeddings_regularizer=regularizer, weights=[embedding], mask_zero=True,\n",
    "                    name='embedding_1'))\n",
    "\n",
    "for i in range(rnn_layers):\n",
    "    lstm = LSTM(rnn_size, return_sequences=True, # batch_norm=batch_norm,\n",
    "                kernel_regularizer=regularizer, recurrent_regularizer=regularizer,\n",
    "                bias_regularizer=regularizer, dropout=p_W, recurrent_dropout=p_U,\n",
    "                name='lstm_%d'%(i+1)\n",
    "                  )\n",
    "    model.add(lstm)\n",
    "    model.add(Dropout(p_dense,name='dropout_%d'%(i+1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A special layer that reduces the input just to its headline part (second half).\n",
    "For each word in this part it concatenate the output of the previous layer (RNN)\n",
    "with a weighted average of the outputs of the description part.\n",
    "In this only the last `rnn_size - activation_rnn_size` are used from each output.\n",
    "The first `activation_rnn_size` output is used to computer the weights for the averaging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.layers.core import Lambda\n",
    "import keras.backend as K\n",
    "\n",
    "def simple_context(X, mask, n=activation_rnn_size, maxlend=maxlend, maxlenh=maxlenh):\n",
    "    desc, head = X[:,:maxlend,:], X[:,maxlend:,:]\n",
    "    head_activations, head_words = head[:,:,:n], head[:,:,n:]\n",
    "    desc_activations, desc_words = desc[:,:,:n], desc[:,:,n:]\n",
    "    \n",
    "    # RTFM http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.batched_tensordot\n",
    "    # activation for every head word and every desc word\n",
    "    activation_energies = K.batch_dot(head_activations, desc_activations, axes=(2,2))\n",
    "    # make sure we dont use description words that are masked out\n",
    "    activation_energies = activation_energies + -1e20*K.expand_dims(1.-K.cast(mask[:, :maxlend],'float32'),1)\n",
    "    \n",
    "    # for every head word compute weights for every desc word\n",
    "    activation_energies = K.reshape(activation_energies,(-1,maxlend))\n",
    "    activation_weights = K.softmax(activation_energies)\n",
    "    activation_weights = K.reshape(activation_weights,(-1,maxlenh,maxlend))\n",
    "\n",
    "    # for every head word compute weighted average of desc words\n",
    "    desc_avg_word = K.batch_dot(activation_weights, desc_words, axes=(2,1))\n",
    "    return K.concatenate((desc_avg_word, head_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if activation_rnn_size:\n",
    "    model.add(Lambda(simple_context,\n",
    "                     mask = lambda inputs, mask: mask[:,maxlend:],\n",
    "                     output_shape = lambda input_shape: (input_shape[0], maxlenh, 2*(rnn_size - activation_rnn_size)),\n",
    "                     name='simplecontext_1'))\n",
    "model.add(TimeDistributed(Dense(vocab_size,\n",
    "                                kernel_regularizer=regularizer, bias_regularizer=regularizer,\n",
    "                                name = 'timedistributed_1')))\n",
    "model.add(Activation('softmax', name='activation_1'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from keras.optimizers import Adam, RMSprop # usually I prefer Adam but article used rmsprop\n",
    "# opt = Adam(lr=LR)  # keep calm and reduce learning rate\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "K.set_value(model.optimizer.lr,np.float32(LR))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 50, 100)           4000000   \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 50, 512)           1255424   \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 50, 512)           0         \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 50, 512)           2099200   \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 50, 512)           0         \n",
      "_________________________________________________________________\n",
      "lstm_3 (LSTM)                (None, 50, 512)           2099200   \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 50, 512)           0         \n",
      "_________________________________________________________________\n",
      "simplecontext_1 (Lambda)     (None, 25, 944)           0         \n",
      "_________________________________________________________________\n",
      "time_distributed_1 (TimeDist (None, 25, 40000)         37800000  \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 25, 40000)         0         \n",
      "=================================================================\n",
      "Total params: 47,253,824.0\n",
      "Trainable params: 47,253,824.0\n",
      "Non-trainable params: 0.0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if FN1:\n",
    "    model.load_weights('data/%s.hdf5'%FN1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def lpadd(x, maxlend=maxlend, eos=eos):\n",
    "    \"\"\"left (pre) pad a description to maxlend and then add eos.\n",
    "    The eos is the input to predicting the first word in the headline\n",
    "    \"\"\"\n",
    "    assert maxlend >= 0\n",
    "    if maxlend == 0:\n",
    "        return [eos]\n",
    "    n = len(x)\n",
    "    if n > maxlend:\n",
    "        x = x[-maxlend:]\n",
    "        n = maxlend\n",
    "    return [empty]*(maxlend-n) + x + [eos]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "samples = [lpadd([3]*26)]\n",
    "# pad from right (post) so the first maxlend will be description followed by headline\n",
    "data = sequence.pad_sequences(samples, maxlen=maxlen, value=empty, padding='post', truncating='post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.all(data[:,maxlend] == eos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1, 50), [26])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape,map(len, samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 25, 40000)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probs = model.predict(data, verbose=0, batch_size=1)\n",
    "probs.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this section is only used to generate examples. you can skip it if you just want to understand how the training works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# variation to https://github.com/ryankiros/skip-thoughts/blob/master/decoding/search.py\n",
    "def beamsearch(predict, start=[empty]*maxlend + [eos],\n",
    "               k=1, maxsample=maxlen, use_unk=True, empty=empty, eos=eos, temperature=1.0):\n",
    "    \"\"\"return k samples (beams) and their NLL scores, each sample is a sequence of labels,\n",
    "    all samples starts with an `empty` label and end with `eos` or truncated to length of `maxsample`.\n",
    "    You need to supply `predict` which returns the label probability of each sample.\n",
    "    `use_unk` allow usage of `oov` (out-of-vocabulary) label in samples\n",
    "    \"\"\"\n",
    "    def sample(energy, n, temperature=temperature):\n",
    "        \"\"\"sample at most n elements according to their energy\"\"\"\n",
    "        n = min(n,len(energy))\n",
    "        prb = np.exp(-np.array(energy) / temperature )\n",
    "        res = []\n",
    "        for i in xrange(n):\n",
    "            z = np.sum(prb)\n",
    "            r = np.argmax(np.random.multinomial(1, prb/z, 1))\n",
    "            res.append(r)\n",
    "            prb[r] = 0. # make sure we select each element only once\n",
    "        return res\n",
    "\n",
    "    dead_k = 0 # samples that reached eos\n",
    "    dead_samples = []\n",
    "    dead_scores = []\n",
    "    live_k = 1 # samples that did not yet reached eos\n",
    "    live_samples = [list(start)]\n",
    "    live_scores = [0]\n",
    "\n",
    "    while live_k:\n",
    "        # for every possible live sample calc prob for every possible label \n",
    "        probs = predict(live_samples, empty=empty)\n",
    "\n",
    "        # total score for every sample is sum of -log of word prb\n",
    "        cand_scores = np.array(live_scores)[:,None] - np.log(probs)\n",
    "        cand_scores[:,empty] = 1e20\n",
    "        if not use_unk:\n",
    "            for i in range(nb_unknown_words):\n",
    "                cand_scores[:,vocab_size - 1 - i] = 1e20\n",
    "        live_scores = list(cand_scores.flatten())\n",
    "        \n",
    "\n",
    "        # find the best (lowest) scores we have from all possible dead samples and\n",
    "        # all live samples and all possible new words added\n",
    "        scores = dead_scores + live_scores\n",
    "        ranks = sample(scores, k)\n",
    "        n = len(dead_scores)\n",
    "        ranks_dead = [r for r in ranks if r < n]\n",
    "        ranks_live = [r - n for r in ranks if r >= n]\n",
    "        \n",
    "        dead_scores = [dead_scores[r] for r in ranks_dead]\n",
    "        dead_samples = [dead_samples[r] for r in ranks_dead]\n",
    "        \n",
    "        live_scores = [live_scores[r] for r in ranks_live]\n",
    "\n",
    "        # append the new words to their appropriate live sample\n",
    "        voc_size = probs.shape[1]\n",
    "        live_samples = [live_samples[r//voc_size]+[r%voc_size] for r in ranks_live]\n",
    "\n",
    "        # live samples that should be dead are...\n",
    "        # even if len(live_samples) == maxsample we dont want it dead because we want one\n",
    "        # last prediction out of it to reach a headline of maxlenh\n",
    "        zombie = [s[-1] == eos or len(s) > maxsample for s in live_samples]\n",
    "        \n",
    "        # add zombies to the dead\n",
    "        dead_samples += [s for s,z in zip(live_samples,zombie) if z]\n",
    "        dead_scores += [s for s,z in zip(live_scores,zombie) if z]\n",
    "        dead_k = len(dead_samples)\n",
    "        # remove zombies from the living \n",
    "        live_samples = [s for s,z in zip(live_samples,zombie) if not z]\n",
    "        live_scores = [s for s,z in zip(live_scores,zombie) if not z]\n",
    "        live_k = len(live_samples)\n",
    "\n",
    "    return dead_samples + live_samples, dead_scores + live_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# !pip install python-Levenshtein"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def keras_rnn_predict(samples, empty=empty, model=model, maxlen=maxlen):\n",
    "    \"\"\"for every sample, calculate probability for every possible label\n",
    "    you need to supply your RNN model and maxlen - the length of sequences it can handle\n",
    "    \"\"\"\n",
    "    sample_lengths = map(len, samples)\n",
    "    assert all(l > maxlend for l in sample_lengths)\n",
    "    assert all(l[maxlend] == eos for l in samples)\n",
    "    # pad from right (post) so the first maxlend will be description followed by headline\n",
    "    data = sequence.pad_sequences(samples, maxlen=maxlen, value=empty, padding='post', truncating='post')\n",
    "    probs = model.predict(data, verbose=0, batch_size=batch_size)\n",
    "    return np.array([prob[sample_length-maxlend-1] for prob, sample_length in zip(probs, sample_lengths)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def vocab_fold(xs):\n",
    "    \"\"\"convert list of word indexes that may contain words outside vocab_size to words inside.\n",
    "    If a word is outside, try first to use glove_idx2idx to find a similar word inside.\n",
    "    If none exist then replace all accurancies of the same unknown word with <0>, <1>, ...\n",
    "    \"\"\"\n",
    "    xs = [x if x < oov0 else glove_idx2idx.get(x,x) for x in xs]\n",
    "    # the more popular word is <0> and so on\n",
    "    outside = sorted([x for x in xs if x >= oov0])\n",
    "    # if there are more than nb_unknown_words oov words then put them all in nb_unknown_words-1\n",
    "    outside = dict((x,vocab_size-1-min(i, nb_unknown_words-1)) for i, x in enumerate(outside))\n",
    "    xs = [outside.get(x,x) for x in xs]\n",
    "    return xs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def vocab_unfold(desc,xs):\n",
    "    # assume desc is the unfolded version of the start of xs\n",
    "    unfold = {}\n",
    "    for i, unfold_idx in enumerate(desc):\n",
    "        fold_idx = xs[i]\n",
    "        if fold_idx >= oov0:\n",
    "            unfold[fold_idx] = unfold_idx\n",
    "    return [unfold.get(x,x) for x in xs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import Levenshtein\n",
    "\n",
    "def gensamples(skips=2, k=10, batch_size=batch_size, short=True, temperature=1., use_unk=True):\n",
    "    i = random.randint(0,len(X_test)-1)\n",
    "    print 'HEAD:',' '.join(idx2word[w] for w in Y_test[i][:maxlenh])\n",
    "    print 'DESC:',' '.join(idx2word[w] for w in X_test[i][:maxlend])\n",
    "    sys.stdout.flush()\n",
    "\n",
    "    print 'HEADS:'\n",
    "    x = X_test[i]\n",
    "    samples = []\n",
    "    if maxlend == 0:\n",
    "        skips = [0]\n",
    "    else:\n",
    "        skips = range(min(maxlend,len(x)), max(maxlend,len(x)), abs(maxlend - len(x)) // skips + 1)\n",
    "    for s in skips:\n",
    "        start = lpadd(x[:s])\n",
    "        fold_start = vocab_fold(start)\n",
    "        sample, score = beamsearch(predict=keras_rnn_predict, start=fold_start, k=k, temperature=temperature, use_unk=use_unk)\n",
    "        assert all(s[maxlend] == eos for s in sample)\n",
    "        samples += [(s,start,scr) for s,scr in zip(sample,score)]\n",
    "\n",
    "    samples.sort(key=lambda x: x[-1])\n",
    "    codes = []\n",
    "    for sample, start, score in samples:\n",
    "        code = ''\n",
    "        words = []\n",
    "        sample = vocab_unfold(start, sample)[len(start):]\n",
    "        for w in sample:\n",
    "            if w == eos:\n",
    "                break\n",
    "            words.append(idx2word[w])\n",
    "            code += chr(w//(256*256)) + chr((w//256)%256) + chr(w%256)\n",
    "        if short:\n",
    "            distance = min([100] + [-Levenshtein.jaro(code,c) for c in codes])\n",
    "            if distance > -0.6:\n",
    "                print score, ' '.join(words)\n",
    "        #         print '%s (%.2f) %f'%(' '.join(words), score, distance)\n",
    "        else:\n",
    "                print score, ' '.join(words)\n",
    "        codes.append(code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HEAD: Top 10 Reasons Why Ines Sainz Is A Story\n",
      "DESC: Can we be honest ? Let 's be honest . The real reason the Ines Sainz story has gained traction is because media outlets can\n",
      "HEADS:\n",
      "18.7137032747 The Internet Goes bonkers\n"
     ]
    }
   ],
   "source": [
    "gensamples(skips=2, batch_size=batch_size, k=10, temperature=1.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data generator generates batches of inputs and outputs/labels for training. The inputs are each made from two parts. The first maxlend words are the original description, followed by `eos` followed by the headline which we want to predict, except for the last word in the headline which is always `eos` and then `empty` padding until `maxlen` words.\n",
    "\n",
    "For each, input, the output is the headline words (without the start `eos` but with the ending `eos`) padded with `empty` words up to `maxlenh` words. The output is also expanded to be y-hot encoding of each word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be more realistic, the second part of the input should be the result of generation and not the original headline.\n",
    "Instead we will flip just `nflips` words to be from the generator, but even this is too hard and instead\n",
    "implement flipping in a naive way (which consumes less time.) Using the full input (description + eos + headline) generate predictions for outputs. For nflips random words from the output, replace the original word with the word with highest probability from the prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def flip_headline(x, nflips=None, model=None, debug=False):\n",
    "    \"\"\"given a vectorized input (after `pad_sequences`) flip some of the words in the second half (headline)\n",
    "    with words predicted by the model\n",
    "    \"\"\"\n",
    "    if nflips is None or model is None or nflips <= 0:\n",
    "        return x\n",
    "    \n",
    "    batch_size = len(x)\n",
    "    assert np.all(x[:,maxlend] == eos)\n",
    "    probs = model.predict(x, verbose=0, batch_size=batch_size)\n",
    "    x_out = x.copy()\n",
    "    for b in range(batch_size):\n",
    "        # pick locations we want to flip\n",
    "        # 0...maxlend-1 are descriptions and should be fixed\n",
    "        # maxlend is eos and should be fixed\n",
    "        flips = sorted(random.sample(xrange(maxlend+1,maxlen), nflips))\n",
    "        if debug and b < debug:\n",
    "            print b,\n",
    "        for input_idx in flips:\n",
    "            if x[b,input_idx] == empty or x[b,input_idx] == eos:\n",
    "                continue\n",
    "            # convert from input location to label location\n",
    "            # the output at maxlend (when input is eos) is feed as input at maxlend+1\n",
    "            label_idx = input_idx - (maxlend+1)\n",
    "            prob = probs[b, label_idx]\n",
    "            w = prob.argmax()\n",
    "            if w == empty:  # replace accidental empty with oov\n",
    "                w = oov0\n",
    "            if debug and b < debug:\n",
    "                print '%s => %s'%(idx2word[x_out[b,input_idx]],idx2word[w]),\n",
    "            x_out[b,input_idx] = w\n",
    "        if debug and b < debug:\n",
    "            print\n",
    "    return x_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def conv_seq_labels(xds, xhs, nflips=None, model=None, debug=False):\n",
    "    \"\"\"description and hedlines are converted to padded input vectors. headlines are one-hot to label\"\"\"\n",
    "    batch_size = len(xhs)\n",
    "    assert len(xds) == batch_size\n",
    "    x = [vocab_fold(lpadd(xd)+xh) for xd,xh in zip(xds,xhs)]  # the input does not have 2nd eos\n",
    "    x = sequence.pad_sequences(x, maxlen=maxlen, value=empty, padding='post', truncating='post')\n",
    "    x = flip_headline(x, nflips=nflips, model=model, debug=debug)\n",
    "    \n",
    "    y = np.zeros((batch_size, maxlenh, vocab_size))\n",
    "    for i, xh in enumerate(xhs):\n",
    "        xh = vocab_fold(xh) + [eos] + [empty]*maxlenh  # output does have a eos at end\n",
    "        xh = xh[:maxlenh]\n",
    "        y[i,:,:] = np_utils.to_categorical(xh, vocab_size)\n",
    "        \n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def gen(Xd, Xh, batch_size=batch_size, nb_batches=None, nflips=None, model=None, debug=False, seed=seed):\n",
    "    \"\"\"yield batches. for training use nb_batches=None\n",
    "    for validation generate deterministic results repeating every nb_batches\n",
    "    \n",
    "    while training it is good idea to flip once in a while the values of the headlines from the\n",
    "    value taken from Xh to value generated by the model.\n",
    "    \"\"\"\n",
    "    c = nb_batches if nb_batches else 0\n",
    "    while True:\n",
    "        xds = []\n",
    "        xhs = []\n",
    "        if nb_batches and c >= nb_batches:\n",
    "            c = 0\n",
    "        new_seed = random.randint(0, sys.maxint)\n",
    "        random.seed(c+123456789+seed)\n",
    "        for b in range(batch_size):\n",
    "            t = random.randint(0,len(Xd)-1)\n",
    "\n",
    "            xd = Xd[t]\n",
    "            s = random.randint(min(maxlend,len(xd)), max(maxlend,len(xd)))\n",
    "            xds.append(xd[:s])\n",
    "            \n",
    "            xh = Xh[t]\n",
    "            s = random.randint(min(maxlenh,len(xh)), max(maxlenh,len(xh)))\n",
    "            xhs.append(xh[:s])\n",
    "\n",
    "        # undo the seeding before we yield inorder not to affect the caller\n",
    "        c+= 1\n",
    "        random.seed(new_seed)\n",
    "\n",
    "        yield conv_seq_labels(xds, xhs, nflips=nflips, model=model, debug=debug)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((64, 50), (64, 25, 40000), 2)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = next(gen(X_train, Y_train, batch_size=batch_size))\n",
    "r[0].shape, r[1].shape, len(r)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_gen(gen, n=5):\n",
    "    Xtr,Ytr = next(gen)\n",
    "    for i in range(n):\n",
    "        assert Xtr[i,maxlend] == eos\n",
    "        x = Xtr[i,:maxlend]\n",
    "        y = Xtr[i,maxlend:]\n",
    "        yy = Ytr[i,:]\n",
    "        yy = np.where(yy)[1]\n",
    "        prt('L',yy)\n",
    "        prt('H',y)\n",
    "        if maxlend:\n",
    "            prt('D',x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L: Rockets GM Daryl O'Keefe Has A implausible And geometry Reason For Giving Money To Mitt Romney ~ _ _ _ _ _ _ _ _\n",
      "H: ~ Rockets GM Daryl O'Keefe Has A implausible And geometry Reason For Giving Money To Mitt Romney _ _ _ _ _ _ _ _\n",
      "D: personal views , but no one really seems to like Mitt for his essential <0>^ . But Daryl O'Keefe , the Houston Rockets general manager\n",
      "L: Today 's Celebrity Twitter Chatter ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ Today 's Celebrity Twitter Chatter _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "D: _ _ _ _ _ _ _ _ _ _ _ _ _ Your daily look into the <0>^ lives of our favorite celebs .\n",
      "L: Natalie Wood Case : Boat Captain Did n't Hear Her Cries For Help ( VIDEO ) ~ _ _ _ _ _ _ _ _\n",
      "H: ~ Natalie Wood Case : Boat Captain Did n't Hear Her Cries For Help ( VIDEO ) _ _ _ _ _ _ _ _\n",
      "D: Many questions still remain in the case surrounding actress Natalie Wood 's mysterious death , but one person who was on the boat is opening\n",
      "L: The Beautiful Water Cube In Beijing Is Now A Water Theme Park ~ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ The Beautiful Water Cube In Beijing Is Now A Water Theme Park _ _ _ _ _ _ _ _ _ _ _ _\n",
      "D: They 're marveled and <0>^ at during the games , but after the festivities , they 're forgotten and left to Rust . China is\n",
      "L: Real Housewives of New Jersey : The revelation of Kim G . ~ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ Real Housewives of New Jersey : The revelation of Kim G . _ _ _ _ _ _ _ _ _ _ _ _\n",
      "D: that the fights and clashes on the Real Housewives of New Jersey come out of nowhere , but they have all been resurrection by a\n"
     ]
    }
   ],
   "source": [
    "test_gen(gen(X_train, Y_train, batch_size=batch_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "test fliping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L: Rockets GM Daryl O'Keefe Has A implausible And geometry Reason For Giving Money To Mitt Romney ~ _ _ _ _ _ _ _ _\n",
      "H: ~ The GM Daryl O'Keefe Has A <0>^ And geometry Reason For Giving Him To Mitt Romney _ _ _ _ _ _ _ _\n",
      "D: personal views , but no one really seems to like Mitt for his essential <0>^ . But Daryl O'Keefe , the Houston Rockets general manager\n",
      "L: Today 's Celebrity Twitter Chatter ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ Today 's Celebrity Twitter Chatter _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "D: _ _ _ _ _ _ _ _ _ _ _ _ _ Your daily look into the <0>^ lives of our favorite celebs .\n",
      "L: Natalie Wood Case : Boat Captain Did n't Hear Her Cries For Help ( VIDEO ) ~ _ _ _ _ _ _ _ _\n",
      "H: ~ Natalie Wood : : Natalie Captain ~ n't Hear Her Cries For Help ( VIDEO ) _ _ _ _ _ _ _ _\n",
      "D: Many questions still remain in the case surrounding actress Natalie Wood 's mysterious death , but one person who was on the boat is opening\n",
      "L: The Beautiful Water Cube In Beijing Is Now A Water Theme Park ~ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ The World Water Cube In Beijing Is Now A ~ Theme Park _ _ _ _ _ _ _ _ _ _ _ _\n",
      "D: They 're marveled and <0>^ at during the games , but after the festivities , they 're forgotten and left to Rust . China is\n",
      "L: Real Housewives of New Jersey : The revelation of Kim G . ~ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ Real Housewives of New Jersey : The revelation of Kim Kardashian . _ _ _ _ _ _ _ _ _ _ _ _\n",
      "D: that the fights and clashes on the Real Housewives of New Jersey come out of nowhere , but they have all been resurrection by a\n"
     ]
    }
   ],
   "source": [
    "test_gen(gen(X_train, Y_train, nflips=6, model=model, debug=False, batch_size=batch_size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "valgen = gen(X_test, Y_test,nb_batches=3, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "check that valgen repeats itself after nb_batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L: Dell Releases Draft 802.11n Wireless Card ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ Dell Releases Draft 802.11n Wireless Card _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "D: the all the problems that draft N has been facing , we 're glad they 're releasing this as an WhatsApp card and not embedding\n",
      "L: Can You Identify a Mystery Cocoon Which Has the Whole Internet stumped ? ~ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ Can You Identify a Mystery Cocoon Which Has the Whole Internet stumped ? _ _ _ _ _ _ _ _ _ _ _\n",
      "D: Usually , throw the internet an image of something you ca n't identify and it 's only a couple of minutes until you 're bombarded\n",
      "L: <0>^ ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ <0>^ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "D: _ _ _ _ _ _ _ Whoa whoa whoa . Wait a second . You 're telling me there 's no liquid cooling ?\n",
      "L: Dell Releases Draft 802.11n Wireless Card ~ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "H: ~ Dell Releases Draft 802.11n Wireless Card _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n",
      "D: the all the problems that draft N has been facing , we 're glad they 're releasing this as an WhatsApp card and not embedding\n"
     ]
    }
   ],
   "source": [
    "for i in range(4):\n",
    "    test_gen(valgen, n=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "history = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "traingen = gen(X_train, Y_train, batch_size=batch_size, nflips=nflips, model=model)\n",
    "valgen = gen(X_test, Y_test, nb_batches=nb_val_samples//batch_size, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((64, 50), (64, 25, 40000), 2)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = next(traingen)\n",
    "r[0].shape, r[1].shape, len(r)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for iteration in range(500):\n",
    "    print 'Iteration', iteration\n",
    "    h = model.fit_generator(traingen, steps_per_epoch=nb_train_samples//batch_size,\n",
    "                        epochs=1, validation_data=valgen, validation_steps=nb_val_samples\n",
    "                           )\n",
    "    for k,v in h.history.iteritems():\n",
    "        history[k] = history.get(k,[]) + v\n",
    "    with open('data/%s.history.pkl'%FN,'wb') as fp:\n",
    "        pickle.dump(history,fp,-1)\n",
    "    model.save_weights('data/%s.hdf5'%FN, overwrite=True)\n",
    "    gensamples(batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}