Miniproyecto_PLN_final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I3OxiAdhFemO"
   },
   "source": [
    "# Miniproyecto PLN\n",
    "\n",
    "## Problema 7\n",
    "\n",
    "### Autores:\n",
    "- Moisés Barrios Torres\n",
    "- Cecilia Diana Albelda\n",
    "- Elena Marrero Castellano\n",
    "- Irina Filimonova Sevcenco"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PxwsF_UpFr_C"
   },
   "source": [
    "### Carga de librerías\n",
    "\n",
    "Importamos todas las librerías necesarias para la carga de datos, preprocesado, modelado y evaluación de modelos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "38jQdTwSlg-t"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import html, keras, math, nltk, random, re, spacy, string, torch\n",
    "\n",
    "from collections import Counter, defaultdict\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.utils import tokenize\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from nltk.corpus import stopwords, wordnet\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "from scipy.special import softmax\n",
    "from sklearn import metrics\n",
    "from sklearn.decomposition import LatentDirichletAllocation, TruncatedSVD\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer, TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression, SGDClassifier\n",
    "from sklearn.metrics import accuracy_score, classification_report\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import MultinomialNB, GaussianNB\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import Normalizer, LabelEncoder\n",
    "from sklearn.svm import SVC\n",
    "from transformers import (AutoModel, AutoModelForSequenceClassification,\n",
    "                          AutoTokenizer, BertForSequenceClassification,\n",
    "                          BertTokenizerFast, BertTokenizer,\n",
    "                          TFAutoModelForSequenceClassification, Trainer,\n",
    "                          TrainingArguments)\n",
    "from transformers.file_utils import (is_tf_available, is_torch_available,\n",
    "                                     is_torch_tpu_available)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vztSX48nF0lj"
   },
   "source": [
    "### Carga de datos \n",
    "Cargamos los datos a partir del fichero csv que previamente hemos creado. El archivo consta de 2989 artículos extraídos mediante técnicas de webscrapping del periódico digital \"EL PAÍS\".\n",
    "\n",
    "Recordamos que los artículos están etiquetados cada uno con su categoría correspondientes, siendo estas: \n",
    "\n",
    "INTERNACIONAL, DEPORTES, CIENCIA, SOCIEDAD, ECONOMÍA y POLÍTICA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 195
    },
    "id": "_jyvCWzeN80o",
    "outputId": "cfb592b9-45e6-4047-f507-118171cc989a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CATEGORÍA</th>\n",
       "      <th>ENLACE</th>\n",
       "      <th>TEXTO</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>INTERNACIONAL</td>\n",
       "      <td>https://elpais.com/television/2021-05-20/la-di...</td>\n",
       "      <td>El Festival de Eurovisión es una tradición mus...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>INTERNACIONAL</td>\n",
       "      <td>https://elpais.com/television/2021-05-20/eurov...</td>\n",
       "      <td>La recta final del Festival de Eurovisión 2021...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>INTERNACIONAL</td>\n",
       "      <td>https://elpais.com/economia/2021-05-20/crecen-...</td>\n",
       "      <td>Se dice que el mercado de divisas nunca duerme...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>INTERNACIONAL</td>\n",
       "      <td>https://elpais.com/economia/2021-05-20/la-puja...</td>\n",
       "      <td>El precio de las casas se ha disparado en país...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>INTERNACIONAL</td>\n",
       "      <td>https://elpais.com/opinion/2021-05-20/drama-y-...</td>\n",
       "      <td>Me ha sorprendido el tono dramático que ha uti...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       CATEGORÍA  ...                                              TEXTO\n",
       "0  INTERNACIONAL  ...  El Festival de Eurovisión es una tradición mus...\n",
       "1  INTERNACIONAL  ...  La recta final del Festival de Eurovisión 2021...\n",
       "2  INTERNACIONAL  ...  Se dice que el mercado de divisas nunca duerme...\n",
       "3  INTERNACIONAL  ...  El precio de las casas se ha disparado en país...\n",
       "4  INTERNACIONAL  ...  Me ha sorprendido el tono dramático que ha uti...\n",
       "\n",
       "[5 rows x 3 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tabla = pd.read_table('articulos.csv', sep=',', index_col=0)\n",
    "tabla.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uOOYVssWF_7R"
   },
   "source": [
    "Nos guardamos los artículos y las etiquetas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aHEoH3LKPvLq"
   },
   "outputs": [],
   "source": [
    "articulos, labels = tabla['TEXTO'], tabla['CATEGORÍA']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uYZBz4awP-F-",
    "outputId": "995ca80c-4640-4aa7-f80d-bb2a281338ac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       El Festival de Eurovisión es una tradición mus...\n",
       "1       La recta final del Festival de Eurovisión 2021...\n",
       "2       Se dice que el mercado de divisas nunca duerme...\n",
       "3       El precio de las casas se ha disparado en país...\n",
       "4       Me ha sorprendido el tono dramático que ha uti...\n",
       "                              ...                        \n",
       "2984    Una investigación sorprendente ha unido en el ...\n",
       "2985    Hace más de 25 años, un grupo de arqueólogos y...\n",
       "2986    Los dinosaurios, aunque sean fósiles o recreac...\n",
       "2987    En el interior de una piedra que no mide mucho...\n",
       "2988    En lo que hoy es el norte de Myanmar se han en...\n",
       "Name: TEXTO, Length: 2989, dtype: object"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articulos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SCQcitNKQILu",
    "outputId": "51f2692d-cd96-4d8c-f495-d93103325a20"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       INTERNACIONAL\n",
       "1       INTERNACIONAL\n",
       "2       INTERNACIONAL\n",
       "3       INTERNACIONAL\n",
       "4       INTERNACIONAL\n",
       "            ...      \n",
       "2984          CIENCIA\n",
       "2985          CIENCIA\n",
       "2986          CIENCIA\n",
       "2987          CIENCIA\n",
       "2988          CIENCIA\n",
       "Name: CATEGORÍA, Length: 2989, dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X5NT13-CfYqC"
   },
   "source": [
    "### Preprocesado"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UGEux6NWk78W"
   },
   "source": [
    "Los algoritmos de ML funcionan sobre un conjunto de características de entrada definido (features), normalmente numérico. Para procesar texto es necesario convertirlo a una matriz numérica (vector space model). Es conveniente limpiar y normalizar el texto en lenguaje natural antes de convertirlo a una matriz numérica.\n",
    "\n",
    "Nosotros hemos probado los siguientes prerocesados: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yTGU3HlYfcPQ"
   },
   "source": [
    "- **PREPROCESADO 1** (*preprocesado*): quitamos los dígitos, los signos de puntuación, las stopwords y las palabras que tengan una longitud de 1. Además nos quedamos con los lemas en minúscula. Finalmente, la salida es una cadena de texto con los tokens filtrados.\n",
    "\n",
    "- **PREPROCESADO 2** (normaliza_bis): añadimos un espacio después de \".\" y \"?\". Quitamos los signos de puntuación y las stopwords con la librería sklearn. Además nos quedamos con los lemas en minúscula. Finalmente, la salida es una cadena de texto con los tokens filtrados.\n",
    "\n",
    "- **PREPROCESADO 3** (extraer_adj): añadimos un espacio después de \".\" y \"?\" y obtenemos el lema de todos los tokens de tipo ADJ. Finalmente, la salida es una cadena de texto con los tokens filtrados.\n",
    "\n",
    "- **PREPROCESADO 4** (extraer_noun): añadimos un espacio después de \".\" y \"?\" y obtenemos el lema de todos los tokens de tipo NOUN. Finalmente, la salida es una cadena de texto con los tokens filtrados.\n",
    "\n",
    "- **PREPROCESADO 5** (normaliza_bis_mezcla): añadimos un espacio después de \".\" y \"?\". Posteriormente, quitamos los dígitos, los signos de puntuación, las stopwords y las palabras que tengan una longitud de 1. Finalmente, la salida es una cadena de texto con los tokens filtrados lematizados y en minúsculas.\n",
    "\n",
    "- **PREPROCESADO 6** (simple_preprocess): separa el texto en tokens, filtra tokens de longitud 2, elimina acentos y signos de puntuación. Finalmente, la salida es el texto en minúsculas.\n",
    "\n",
    "- **PREPROCESADO 7** (tokenize): genera de forma iterativa tokens como cadenas Unicode, eliminando las marcas de acento y, opcionalmente, poniendo en minúscula. El texto de entrada puede ser unicode o una cadena de bytes codificada en utf8. Los tokens en la salida son secuencias contiguas máximas de caracteres alfabéticos (sin dígitos).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nw9acJDalkle"
   },
   "source": [
    "#### Definimos las funciones de preprocesado que hemos creado nosotros:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5b93PL7QhfQ4"
   },
   "outputs": [],
   "source": [
    "def preprocesado(texto):\n",
    "  '''\n",
    "   Quitamos dígitos, signos de puntuación, stopwords y las palabras que tengan una longitud de 1. \n",
    "   Además, nos quedamos con los lemas en minúscula y devolvemos una cadena de texto \n",
    "   con los tokens filtrados.\n",
    "  '''\n",
    "  doc = nlp(texto) # objeto nlp\n",
    "  tokens = [token.lemma_.lower() for token in doc if not token.is_digit and not \n",
    "            token.is_punct and not token.is_stop and len(token.text) > 1]\n",
    "  return ' '.join(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CbuNDkMOcMEx"
   },
   "outputs": [],
   "source": [
    "def normaliza_bis(texto):\n",
    "  '''\n",
    "   Añadimos un espacio después de \".\" y \"?\", quitamos signos de puntuación y stopwords\n",
    "   con la librería sklearn. Además, nos quedamos con los lemas en minúscula y \n",
    "   devolvemos una cadena de texto con los tokens filtrados.\n",
    "  '''\n",
    "    texto = re.sub(r\"([\\?\\.])\", r\"\\1 \", texto) \n",
    "    doc = nlp(texto)\n",
    "    tokens = [t for t in doc if not t.is_stop and not t.is_punct and (len(t) > 1)] \n",
    "    palabras = []\n",
    "    for t in tokens:\n",
    "        if t.ent_iob_=='B' and t.ent_type_=='PER':\n",
    "            palabras.append('persona')\n",
    "        elif t.ent_iob_=='I' and t.ent_type_=='PER':\n",
    "            continue\n",
    "        else:\n",
    "            palabras.append(t.lower_) \n",
    "    salida = ' '.join(palabras) \n",
    "    return salida"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eT6kWqtDhNsF"
   },
   "outputs": [],
   "source": [
    "def extraer_adj(texto):\n",
    "  '''\n",
    "   Añadimos un espacio después de \".\" y \"?\", obtenemos el lema de todos los tokens de tipo ADJ\n",
    "   y devolvemos una cadena de texto con los tokens filtrados.\n",
    "  '''\n",
    "    texto = re.sub(r\"([\\?\\.])\", r\"\\1 \", texto)\n",
    "    doc = nlp(texto)\n",
    "    tokens = [t.lemma_ for t in doc if t.pos_=='ADJ'] \n",
    "    return ' '.join(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eNKA1z7-muZi"
   },
   "outputs": [],
   "source": [
    "def extraer_noun(texto):\n",
    "  '''\n",
    "   Añadimos un espacio después de \".\" y \"?\" y obtenemos el lema de todos los tokens de tipo NOUN.\n",
    "   Devolvemos una cadena de texto con los tokens filtrados\n",
    "  '''\n",
    "    texto = re.sub(r\"([\\?\\.])\", r\"\\1 \", texto) \n",
    "    doc = nlp(texto)\n",
    "    tokens = [t.lemma_ for t in doc if t.pos_=='NOUN'] \n",
    "    return ' '.join(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WKn4hcZyoooM"
   },
   "outputs": [],
   "source": [
    "def normaliza_bis_mezcla(texto):\n",
    "  '''\n",
    "  Añadimos un espacio después de \".\" y \"?\", quitamos dígitos, signos de puntuación, stopwords y\n",
    "  palabras que tengan una longitud de 1. Devolvemos una cadena de texto con los\n",
    "  tokens filtrados lematizados y en minúsculas.\n",
    "  '''\n",
    "    texto = re.sub(r\"([\\?\\.])\", r\"\\1 \", texto) \n",
    "    doc = nlp(texto)\n",
    "    tokens = [token for token in doc if not token.is_digit and \n",
    "              not token.is_punct and not token.is_stop and len(token.text) > 1]\n",
    "    palabras = []\n",
    "    for t in tokens:\n",
    "        if t.ent_iob_=='B' and t.ent_type_=='PER':\n",
    "            palabras.append('persona')\n",
    "        elif t.ent_iob_=='I' and t.ent_type_=='PER':\n",
    "            continue\n",
    "        else:\n",
    "            palabras.append(t.lower_) \n",
    "    salida = ' '.join(palabras) \n",
    "    return salida"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1JQAQ8LEjiF6"
   },
   "source": [
    "#### Cargamos el modelo en español de spaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "u300fT5qEfzn",
    "outputId": "7109812f-8a6d-4444-c4c4-361e4249ae70"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the model via spacy.load('es_core_news_sm')\n"
     ]
    }
   ],
   "source": [
    "spacy.cli.download(\"es_core_news_sm\")\n",
    "nlp = spacy.load('es_core_news_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4-nVIYeslw4H"
   },
   "source": [
    "#### Aplicamos las distintas funciones de preprocesado:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "A7oCvk7wtsV_"
   },
   "outputs": [],
   "source": [
    "corpus_preprocesado = [preprocesado(t) for t in articulos] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KNfKMgUGla3Z"
   },
   "outputs": [],
   "source": [
    "corpus_normaliza_bis = [normaliza_bis(t) for t in articulos]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bTYQSK0Uttmz"
   },
   "outputs": [],
   "source": [
    "corpus_extraer_adj = [extraer_adj(t) for t in articulos]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hR92AgU_t0Du"
   },
   "outputs": [],
   "source": [
    "corpus_extraer_noun = [extraer_noun(t) for t in articulos]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CwciTGf4t4kn"
   },
   "outputs": [],
   "source": [
    "corpus_normaliza_bis_mezcla = [normaliza_bis_mezcla(t) for t in articulos]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "L3KJBsfYuZkv"
   },
   "outputs": [],
   "source": [
    "corpus_simple_preprocess = [' '.join(a) for a in [simple_preprocess(t) for t in articulos]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WVcrQsYlu6DL"
   },
   "outputs": [],
   "source": [
    "corpus_tokenize = [' '.join(a) for a in [list(tokenize(t, deacc=True, lowercase=True)) for t in articulos]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j3TZlQI1mipI"
   },
   "source": [
    "#### Dividimos el conjunto de datos en entrenamiento y test\n",
    "\n",
    "Procedemos a dividir el conjunto inicial. Por un lado, cogemos 2/3 del conjunto de datos inicial para entrenar, es decir, el conjunto de entrenamiento (siendo train_corpus, train_labels los nombres de dichas variables). Por otro lado, el tercio restante (conjunto de test) para comprobar el correcto funcionamiento del modelo (siendo test_corpus, test_labels los nombres de dichas variables). \n",
    "\n",
    "Con el conjunto de test no se trabaja en el entrenamiento, asegurándonos así que el modelo no conozca los datos \"reales\".\n",
    "\n",
    "Vamos a realizar la división de los datos para cada tipo de preprocesado definido anteriormente."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8NnK2F8XFYCd"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 1: preprocesado\n",
    "train_corpus_p, test_corpus_p, train_labels_p, test_labels_p = train_test_split(corpus_preprocesado,\n",
    "                                             labels,\n",
    "                                             test_size=1/3,\n",
    "                                             random_state=666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QommMh2NxVrY"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 2: normaliza_bis\n",
    "train_corpus_nb, test_corpus_nb, train_labels_nb, test_labels_nb = train_test_split(corpus_normaliza_bis,\n",
    "                                             labels,\n",
    "                                             test_size=1/3,\n",
    "                                             random_state=666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qwLN8cR7zUcg"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 3: extraer_adj\n",
    "train_corpus_ea, test_corpus_ea, train_labels_ea, test_labels_ea = train_test_split(corpus_extraer_adj,\n",
    "                                             labels,\n",
    "                                             test_size=1/3,\n",
    "                                             random_state=666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CYcSBbQQzUpz"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 4: extraer_noun\n",
    "train_corpus_en, test_corpus_en, train_labels_en, test_labels_en = train_test_split(corpus_extraer_noun,\n",
    "                                             labels,\n",
    "                                             test_size=1/3,\n",
    "                                             random_state=666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "W5dwu6qWzU2q"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 5: normaliza_bis_mezcla\n",
    "train_corpus_nbm, test_corpus_nbm, train_labels_nbm, test_labels_nbm = train_test_split(corpus_normaliza_bis_mezcla,\n",
    "                                             labels,\n",
    "                                             test_size=1/3,\n",
    "                                             random_state=666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "B6dZa6XYzVAX"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 6: simple_preprocess\n",
    "train_corpus_s, test_corpus_s, train_labels_s, test_labels_s = train_test_split(corpus_simple_preprocess,\n",
    "                                             labels,\n",
    "                                             test_size=1/3,\n",
    "                                             random_state=666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VKCKYd4TzVJo"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 7: tokenize\n",
    "train_corpus_t, test_corpus_t, train_labels_t, test_labels_t = train_test_split(corpus_tokenize,\n",
    "                                             labels,\n",
    "                                             test_size=1/3,\n",
    "                                             random_state=666)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HCaHNfAT2bat"
   },
   "source": [
    "### Vectorizadores principales\n",
    "\n",
    "- Matrices sparse:\n",
    "  - BOW (Bag Of Words): Cada texto se representa por una fila de la matriz, donde cada elemento (columna) representa la frecuencia de aparición de un término del vocabulario en el texto\n",
    "  - TF-IDF (Term Frequency–Inverse Document Frequency): Representación sobre un vector BoW donde se aplica una ponderación para descontar los términos más comunes y promover los raros.\n",
    "- Matrices densas:\n",
    "  - LSA (Latent Semantic Analysis) para extraer un conjunto de conceptos latentes relacionando documentos.\n",
    "  - LDA (Latent Dirichlet Allocation), que obtiene la distribución de probabilidad de Dirichlet de las combinaciones de (articulo, categoría)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jw9_X3xl7Zun"
   },
   "source": [
    "#### Definimos todos los vectorizadores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oXbSMtVe4G7q"
   },
   "outputs": [],
   "source": [
    "def modelo(train_corpus, test_corpus):\n",
    "  # Modelo BOW\n",
    "  matrix_features_train = []\n",
    "  matrix_features_test = []\n",
    "  bow_vectorizer = CountVectorizer(min_df = 0.1)\n",
    "  ## características bag of words\n",
    "  bow_train_features = bow_vectorizer.fit_transform(train_corpus)  \n",
    "  bow_test_features = bow_vectorizer.transform(test_corpus)\n",
    "  matrix_features_train.append(bow_train_features)\n",
    "  matrix_features_test.append(bow_test_features)\n",
    "\n",
    "\n",
    "  # Modelo TF-IDF\n",
    "  tfidf_vectorizer = TfidfTransformer()\n",
    "  ## características tfidf (a partir del BoW)\n",
    "  tfidf_train_features = tfidf_vectorizer.fit_transform(bow_train_features)\n",
    "  tfidf_test_features = tfidf_vectorizer.transform(bow_test_features)  \n",
    "  matrix_features_train.append(tfidf_train_features)\n",
    "  matrix_features_test.append(tfidf_test_features)\n",
    "  \n",
    "\n",
    "  # Modelo LSA\n",
    "  tv = TfidfVectorizer()\n",
    "  svd = TruncatedSVD(n_components=100)\n",
    "  lsa = make_pipeline(tv, svd, Normalizer(copy=False))\n",
    "  lsa_train_matrix = lsa.fit_transform(train_corpus)\n",
    "  lsa_test_matrix = lsa.transform(test_corpus)\n",
    "  matrix_features_train.append(lsa_train_matrix)\n",
    "  matrix_features_test.append(lsa_test_matrix)\n",
    "\n",
    "\n",
    "  # Modelo LDA\n",
    "  tf_vectorizer = CountVectorizer(min_df = 0.01)\n",
    "  tf_train = tf_vectorizer.fit_transform(train_corpus)\n",
    "  tf_test = tf_vectorizer.transform(test_corpus)\n",
    "  lda = LatentDirichletAllocation(n_components = 6, random_state = 666)\n",
    "  lda_train_matrix = lda.fit_transform(tf_train)\n",
    "  lda_test_matrix = lda.transform(tf_test)\n",
    "  matrix_features_train.append(lda_train_matrix)\n",
    "  matrix_features_test.append(lda_test_matrix)\n",
    "  return matrix_features_train, matrix_features_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y1A1JBsDtZmN"
   },
   "source": [
    "### Modelos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BXNRDenttbg1"
   },
   "source": [
    "#### Definimos funciones para la creación y evaluación de modelos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WQeQlWvyOAf-"
   },
   "outputs": [],
   "source": [
    "def get_metrics(true_labels, predicted_labels):\n",
    "    \"\"\"\n",
    "    Calculamos distintas métricas sobre el\n",
    "    rendimiento del modelo. Devuelve un diccionario\n",
    "    con los parámetros medidos\n",
    "    \"\"\"\n",
    "\n",
    "    return {\n",
    "        'Accuracy': np.round(\n",
    "                        metrics.accuracy_score(true_labels, \n",
    "                                               predicted_labels),\n",
    "                        3),\n",
    "        'Precision': np.round(\n",
    "                        metrics.precision_score(true_labels, \n",
    "                                               predicted_labels,\n",
    "                                               average='weighted',\n",
    "                                               zero_division=0),\n",
    "                        3),\n",
    "    'Recall': np.round(\n",
    "                        metrics.recall_score(true_labels, \n",
    "                                               predicted_labels,\n",
    "                                               average='weighted',\n",
    "                                               zero_division=0),\n",
    "                        3),\n",
    "    'F1 Score': np.round(\n",
    "                        metrics.f1_score(true_labels, \n",
    "                                               predicted_labels,\n",
    "                                               average='weighted',\n",
    "                                               zero_division=0),\n",
    "                        3)}              "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "daeM6SGzuA7V"
   },
   "outputs": [],
   "source": [
    "def train_predict_evaluate_model(classifier, train_features, train_labels, \n",
    "                                 test_features, test_labels):\n",
    "    \"\"\"\n",
    "    Función que entrena un modelo de clasificación sobre\n",
    "    un conjunto de entrenamiento, lo aplica sobre un conjunto\n",
    "    de test y devuelve la predicción sobre el conjunto de test\n",
    "    y las métricas de rendimiento\n",
    "    \"\"\"\n",
    "\n",
    "    # genera modelo    \n",
    "    classifier.fit(train_features, train_labels)\n",
    "    # predice usando el modelo sobre test\n",
    "    predictions = classifier.predict(test_features) \n",
    "    # evalúa rendimiento de la predicción   \n",
    "    metricas = get_metrics(true_labels=test_labels, \n",
    "                predicted_labels=predictions)\n",
    "    \n",
    "    return predictions, metricas  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nkd7W18EOEIO"
   },
   "outputs": [],
   "source": [
    "def calcular_metricas(lmf_train, lmf_test, train_labels, test_labels):\n",
    "  \"\"\"\n",
    "    Esta función aplica \"train_predict_evaluate_model\" a cada uno de los modelos,\n",
    "    realizando así la predicción y evaliación de todos ellos\n",
    "  \"\"\"\n",
    "\n",
    "  modelLR = LogisticRegression(solver='liblinear')\n",
    "  modelNB = GaussianNB()\n",
    "  modelSVM = SGDClassifier(loss='hinge', max_iter=1000)\n",
    "  modelRBFSVM = SVC(gamma='scale', C=2)\n",
    "\n",
    "  modelos = [('Logistic Regression', modelLR),\n",
    "             ('Naive Bayes', modelNB),\n",
    "             ('Linear SVM', modelSVM),\n",
    "             ('Gauss kernel SVM', modelRBFSVM)]\n",
    "\n",
    "  metricas = []\n",
    "  resultados = []\n",
    "\n",
    "\n",
    "  # Modelos con características BoW\n",
    "  bow_train_f = lmf_train[0].toarray()\n",
    "  bow_test_f = lmf_test[0].toarray()\n",
    "  for m, clf in modelos:\n",
    "      prediccion, metrica = train_predict_evaluate_model(classifier=clf,\n",
    "                                            train_features=bow_train_f,\n",
    "                                            train_labels=train_labels,\n",
    "                                            test_features=bow_test_f,\n",
    "                                            test_labels=test_labels)\n",
    "      metrica['modelo']=f'{m} BoW'\n",
    "      resultados.append(prediccion)\n",
    "      metricas.append(metrica)\n",
    "\n",
    "\n",
    "  # Modelos con características TF-IDF\n",
    "  tfidf_train_f = lmf_train[1].toarray()\n",
    "  tfidf_test_f = lmf_test[1].toarray()\n",
    "  for m, clf in modelos:\n",
    "      prediccion, metrica = train_predict_evaluate_model(classifier=clf,\n",
    "                                            train_features=tfidf_train_f,\n",
    "                                            train_labels=train_labels,\n",
    "                                            test_features=tfidf_test_f,\n",
    "                                            test_labels=test_labels)\n",
    "      metrica['modelo']=f'{m} TF-IDF'\n",
    "      resultados.append(prediccion)\n",
    "      metricas.append(metrica)\n",
    "\n",
    "\n",
    "  # Modelos con características LSA\n",
    "  lsa_train_features = lmf_train[2]\n",
    "  lsa_test_features = lmf_test[2]\n",
    "  for m, clf in modelos:\n",
    "      prediccion, metrica = train_predict_evaluate_model(classifier=clf,\n",
    "                                            train_features=lsa_train_features,\n",
    "                                            train_labels=train_labels,\n",
    "                                            test_features=lsa_test_features,\n",
    "                                            test_labels=test_labels)\n",
    "      metrica['modelo']=f'{m} LSA'\n",
    "      resultados.append(prediccion)\n",
    "      metricas.append(metrica)\n",
    "\n",
    "\n",
    "  # Modelos con características LDA\n",
    "  lda_train_features = lmf_train[3]\n",
    "  lda_test_features = lmf_test[3]\n",
    "  for m, clf in modelos:\n",
    "      prediccion, metrica = train_predict_evaluate_model(classifier=clf,\n",
    "                                            train_features=lda_train_features,\n",
    "                                            train_labels=train_labels,\n",
    "                                            test_features=lda_test_features,\n",
    "                                            test_labels=test_labels)\n",
    "      metrica['modelo']=f'{m} LDA'\n",
    "      resultados.append(prediccion)\n",
    "      metricas.append(metrica)\n",
    "\n",
    "  return metricas, resultados"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NzMQcgWuzdP-"
   },
   "source": [
    "#### Aplicamos los modelos para cada preprocesado\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QG8JWvvKIMPk"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 1: preprocesado\n",
    "mf_train_p, mf_test_p = modelo(train_corpus_p, test_corpus_p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VlFIO5vrIMSR"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 2: normaliza_bis\n",
    "mf_train_nb, mf_test_nb = modelo(train_corpus_nb, test_corpus_nb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ewxzeoWgIMUp"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 3: extraer_adj\n",
    "mf_train_ea, mf_test_ea = modelo(train_corpus_ea, test_corpus_ea)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9vHrz0-7IMXM"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 4: extraer_noun\n",
    "mf_train_en, mf_test_en = modelo(train_corpus_en, test_corpus_en)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "E7XL0TxRIMZu"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 5: normaliza_bis_mezcla\n",
    "mf_train_nbm, mf_test_nbm = modelo(train_corpus_nbm, test_corpus_nbm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xsxmkVVSIkAu"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 6: simple_preprocess\n",
    "mf_train_s, mf_test_s = modelo(train_corpus_s, test_corpus_s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kicCPjGXImLL"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 7: tokenize\n",
    "mf_train_t, mf_test_t = modelo(train_corpus_t, test_corpus_t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eZFwYseUHKq0"
   },
   "source": [
    "#### Evaluación de los modelos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gVJ1KnxlRI8g"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 1: preprocesado\n",
    "metricas_p, resultados_p = calcular_metricas(mf_train_p, \n",
    "                                             mf_test_p, \n",
    "                                             train_labels_p, \n",
    "                                             test_labels_p)\n",
    "metricas_df_p = pd.DataFrame(metricas_p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IimUzF_uSmjb"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 2: normaliza_bis\n",
    "metricas_nb, resultados_nb = calcular_metricas(mf_train_nb, \n",
    "                                             mf_test_nb, \n",
    "                                             train_labels_nb, \n",
    "                                             test_labels_nb)\n",
    "metricas_df_nb = pd.DataFrame(metricas_nb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VXdn-utwSmnG"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 3: extraer_adj\n",
    "metricas_ea, resultados_ea = calcular_metricas(mf_train_ea, \n",
    "                                             mf_test_ea, \n",
    "                                             train_labels_ea, \n",
    "                                             test_labels_ea)\n",
    "metricas_df_ea = pd.DataFrame(metricas_ea)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ol6kp4WNSmrf"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 4: extraer_noun\n",
    "metricas_en, resultados_en = calcular_metricas(mf_train_en, \n",
    "                                             mf_test_en, \n",
    "                                             train_labels_en, \n",
    "                                             test_labels_en)\n",
    "metricas_df_en = pd.DataFrame(metricas_en)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8Ozyy7wASmv9"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 5: normaliza_bis_mezcla\n",
    "metricas_nbm, resultados_nbm = calcular_metricas(mf_train_nbm, \n",
    "                                             mf_test_nbm, \n",
    "                                             train_labels_nbm, \n",
    "                                             test_labels_nbm)\n",
    "metricas_df_nbm = pd.DataFrame(metricas_nbm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "V21R_zwtTYeG"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 6: simple_preprocess\n",
    "metricas_s, resultados_s = calcular_metricas(mf_train_s, \n",
    "                                             mf_test_s, \n",
    "                                             train_labels_s, \n",
    "                                             test_labels_s)\n",
    "metricas_df_s = pd.DataFrame(metricas_s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GVIyr11oTYwR"
   },
   "outputs": [],
   "source": [
    "# Preprocesamiento 7: tokenize\n",
    "metricas_t, resultados_t = calcular_metricas(mf_train_t, \n",
    "                                             mf_test_t, \n",
    "                                             train_labels_t, \n",
    "                                             test_labels_t)\n",
    "metricas_df_t = pd.DataFrame(metricas_t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f7mBKJZefJAb"
   },
   "source": [
    "Mostramos las métricas de los tres mejores modelos obtenidas con cada uno de los preprocesados:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_qR5PGgQX-VA",
    "outputId": "4c171b5f-2014-45da-d567-cc64422c8a0e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "preprocesado \n",
      "\n",
      "    Accuracy  Precision  Recall  F1 Score                   modelo\n",
      "11     0.849      0.850   0.849     0.847     Gauss kernel SVM LSA\n",
      "8      0.845      0.846   0.845     0.842  Logistic Regression LSA\n",
      "10     0.839      0.846   0.839     0.837           Linear SVM LSA\n",
      "\n",
      "\n",
      "normaliza_bis \n",
      "\n",
      "    Accuracy  Precision  Recall  F1 Score                   modelo\n",
      "10     0.853      0.854   0.853     0.850           Linear SVM LSA\n",
      "11     0.846      0.846   0.846     0.843     Gauss kernel SVM LSA\n",
      "8      0.842      0.842   0.842     0.839  Logistic Regression LSA\n",
      "\n",
      "\n",
      "extraer_adj \n",
      "\n",
      "    Accuracy  Precision  Recall  F1 Score                   modelo\n",
      "11     0.759      0.759   0.759     0.756     Gauss kernel SVM LSA\n",
      "10     0.742      0.740   0.742     0.734           Linear SVM LSA\n",
      "8      0.734      0.727   0.734     0.727  Logistic Regression LSA\n",
      "\n",
      "\n",
      "extraer_noun \n",
      "\n",
      "    Accuracy  Precision  Recall  F1 Score                   modelo\n",
      "11     0.823      0.825   0.823     0.821     Gauss kernel SVM LSA\n",
      "10     0.813      0.815   0.813     0.809           Linear SVM LSA\n",
      "8      0.812      0.812   0.812     0.808  Logistic Regression LSA\n",
      "\n",
      "\n",
      "normaliza_bis_mezcla \n",
      "\n",
      "    Accuracy  Precision  Recall  F1 Score                   modelo\n",
      "10     0.845      0.849   0.845     0.842           Linear SVM LSA\n",
      "8      0.844      0.843   0.844     0.841  Logistic Regression LSA\n",
      "11     0.840      0.840   0.840     0.837     Gauss kernel SVM LSA\n",
      "\n",
      "\n",
      "simple_preprocess \n",
      "\n",
      "    Accuracy  Precision  Recall  F1 Score                   modelo\n",
      "11     0.835      0.836   0.835     0.833     Gauss kernel SVM LSA\n",
      "10     0.818      0.820   0.818     0.811           Linear SVM LSA\n",
      "8      0.817      0.815   0.817     0.814  Logistic Regression LSA\n",
      "\n",
      "\n",
      "tokenize \n",
      "\n",
      "    Accuracy  Precision  Recall  F1 Score                   modelo\n",
      "10     0.836      0.832   0.836     0.830           Linear SVM LSA\n",
      "11     0.836      0.836   0.836     0.834     Gauss kernel SVM LSA\n",
      "8      0.816      0.814   0.816     0.812  Logistic Regression LSA\n"
     ]
    }
   ],
   "source": [
    "print(\"preprocesado \\n\")\n",
    "print(metricas_df_p.sort_values(\"Accuracy\", ascending=False).head(3))\n",
    "print(\"\\n\")\n",
    "print(\"normaliza_bis \\n\")\n",
    "print(metricas_df_nb.sort_values(\"Accuracy\", ascending=False).head(3))\n",
    "print(\"\\n\")\n",
    "print(\"extraer_adj \\n\")\n",
    "print(metricas_df_ea.sort_values(\"Accuracy\", ascending=False).head(3))\n",
    "print(\"\\n\")\n",
    "print(\"extraer_noun \\n\")\n",
    "print(metricas_df_en.sort_values(\"Accuracy\", ascending=False).head(3))\n",
    "print(\"\\n\")\n",
    "print(\"normaliza_bis_mezcla \\n\")\n",
    "print(metricas_df_nbm.sort_values(\"Accuracy\", ascending=False).head(3))\n",
    "print(\"\\n\")\n",
    "print(\"simple_preprocess \\n\")\n",
    "print(metricas_df_s.sort_values(\"Accuracy\", ascending=False).head(3))\n",
    "print(\"\\n\")\n",
    "print(\"tokenize \\n\")\n",
    "print(metricas_df_t.sort_values(\"Accuracy\", ascending=False).head(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XHoQRL2ak9lC"
   },
   "source": [
    "Como podemos observar, todas las combinaciones preprocesado-modelo logran valores de \"accuracy\" muy similares entre sí. Sabemos que esta métrica (el \"accuracy\") se obtiene dividiendo el total de predicciones correctas de nuestro modelo entre el total de predicciones realizadas, y suele ser la métrica que más se considera en la evaluación de modelos.\n",
    "\n",
    "Dicha métrica va entre 0.759 (para el modelo Gauss kernel SVM, con el vectorizador LSA y el preprocesado extraer-adj) y  0.853 (para el modelo  Linear SVM, con el vectorizador LSA y el preprocesado normaliza-bis).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kabhuSCBlcna"
   },
   "source": [
    "#### Conclusiones\n",
    "\n",
    "Mostramos la matriz de confusión para la mejor combinación de preprocesado y modelo: Linear SVM, con el vectorizador LSA y el preprocesado normaliza-bis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 225
    },
    "id": "W9ZDZr9raUe_",
    "outputId": "0c00c1d4-9c33-4b05-87d1-d8963fa04a9f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CIENCIA</th>\n",
       "      <th>DEPORTES</th>\n",
       "      <th>ECONOMÍA</th>\n",
       "      <th>INTERNACIONAL</th>\n",
       "      <th>POLÍTICA</th>\n",
       "      <th>SOCIEDAD</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CIENCIA</th>\n",
       "      <td>158</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DEPORTES</th>\n",
       "      <td>0</td>\n",
       "      <td>175</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ECONOMÍA</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>70</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>INTERNACIONAL</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "      <td>169</td>\n",
       "      <td>10</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>POLÍTICA</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>192</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SOCIEDAD</th>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>35</td>\n",
       "      <td>22</td>\n",
       "      <td>6</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               CIENCIA  DEPORTES  ECONOMÍA  INTERNACIONAL  POLÍTICA  SOCIEDAD\n",
       "CIENCIA            158         0         1              5         1         5\n",
       "DEPORTES             0       175         2              1         0         1\n",
       "ECONOMÍA             2         1        70              8         1        10\n",
       "INTERNACIONAL        6         1        16            169        10         7\n",
       "POLÍTICA             0         0         2              1       192         2\n",
       "SOCIEDAD             5         3        35             22         6        79"
      ]
     },
     "execution_count": 61,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "modelSVM = SGDClassifier(loss='hinge', max_iter=1000)\n",
    "prediccion, metrica = train_predict_evaluate_model(classifier=modelSVM,\n",
    "                                            train_features=mf_train_nb[2],\n",
    "                                            train_labels=train_labels_nb,\n",
    "                                            test_features=mf_test_nb[2],\n",
    "                                            test_labels=test_labels_nb)\n",
    "\n",
    "cm = metrics.confusion_matrix(test_labels_p, prediccion)\n",
    "pd.DataFrame(cm, index=modelSVM.classes_, columns=modelSVM.classes_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HMPF2qYol3Z4"
   },
   "source": [
    "En un caso ideal, toda la matriz de confusión estaría formada por ceros exceptuando la diagonal principal. Esto implicaría que hemos acertado el 100% de predicciones. En dicha matriz, las filas corresponden a las etiquetas reales de los artículos (clasificadas por el propio periódico) y las columnas hacen referencia a las predicciones de las etiquetas realizadas por nuestro modelo.\n",
    "\n",
    "Como podemos apreciar, en general la matriz evidencia la buena predicción obtenida, pues concentra sus valores más grandes en la diagonal principal. No obstante, detectamos también que el peor funcionamiento del modelo surge con los artículos de 'SOCIEDAD'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rxnCXPxvmCTm"
   },
   "source": [
    "### Otros modelos\n",
    "\n",
    "Además de los modelos vistos en la asignatura, vamos a probar también con dos nuevos modelos."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uF083OdsKadg"
   },
   "source": [
    "#### Modelo MultinomialNaiveBayes \n",
    "\n",
    "En este apartado, vamos a implementar la clase \"MultinomialNaiveBayes\" a partir de la librería NLTK (Natural Languange Toolkit), muy reconocida en el mundo del NLP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oVl1NKzMClDg"
   },
   "outputs": [],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ih1e6Nt_KY9w"
   },
   "outputs": [],
   "source": [
    "class MultinomialNaiveBayes:\n",
    "  \"\"\"\n",
    "  Esta clase servirá para definir nuestro modelo MultinomialNaiveBayes con la \n",
    "  librería nltk. A partir de las funciones de esta clase podemos tanto ajustar el \n",
    "  modelo como predecir los resultados una vez ajustado.\n",
    "  \"\"\"\n",
    "  def __init__(self, classes, tokenizer):\n",
    "    self.tokenizer = tokenizer\n",
    "    self.classes = classes\n",
    "  \n",
    "  def group_by_class(self, X, y):\n",
    "    y = [y for y in y]\n",
    "    data = dict()\n",
    "    for c in self.classes:\n",
    "      textos = []\n",
    "      for labell in range(len(y)):\n",
    "        if y[labell] == c:\n",
    "          textos.append(X[labell])\n",
    "      data[c] = textos\n",
    "    return data\n",
    "  \n",
    "  def fit(self, X, y):\n",
    "    self.n_class_items = {}\n",
    "    self.log_class_priors = {}\n",
    "    self.word_counts = {}\n",
    "    self.vocab = vocab\n",
    "    n = len(X)\n",
    "    grouped_data = self.group_by_class(X, y)\n",
    "    for c, data in grouped_data.items():\n",
    "      self.n_class_items[c] = len(data)\n",
    "      self.log_class_priors[c] = math.log(self.n_class_items[c]/n)\n",
    "      self.word_counts[c] = defaultdict(lambda: 0)\n",
    "\n",
    "      for text in data:\n",
    "        counts = Counter(nltk.word_tokenize(text))\n",
    "        for word, count in counts.items():\n",
    "          self.word_counts[c][word] += count\n",
    "    return self\n",
    "\n",
    "  def laplace_smoothing(self, word, text_class):\n",
    "    num = self.word_counts[text_class][word] + 1\n",
    "    denom = self.n_class_items[text_class] + len(self.vocab)\n",
    "    return math.log(num/denom)\n",
    "\n",
    "  def predict(self, X):\n",
    "    result = []\n",
    "    for text in X:\n",
    "      class_scores = {c: self.log_class_priors[c] for c in self.classes}\n",
    "      words = set(nltk.word_tokenize(text))\n",
    "      for word in words:\n",
    "        if word not in self.vocab: continue\n",
    "        for c in self.classes:\n",
    "          log_w_given_c = self.laplace_smoothing(word, c)\n",
    "          class_scores[c] += log_w_given_c\n",
    "      result.append(max(class_scores, key = class_scores.get))\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "p0Ht3myemluH"
   },
   "source": [
    "Probamos nuestro modelo con el mejor preprocesado obtenido anteriormente (normaliza-bis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qhgmwSAt7jg4"
   },
   "source": [
    "- Vectorizamos el corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fV0xNcGc6aUk"
   },
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(max_features=3000)\n",
    "sents_encoded = vectorizer.fit_transform(train_corpus_nb)\n",
    "counts = sents_encoded.sum(axis = 0).A1\n",
    "vocab = list(vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DOTyDdBw7m8-"
   },
   "source": [
    "- Definimos el modelo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cmY0dpqHKe8f"
   },
   "outputs": [],
   "source": [
    "MNB = MultinomialNaiveBayes(classes = np.unique(labels),\n",
    "                            tokenizer = Tokenizer()).fit(train_corpus_nb,\n",
    "                                                         train_labels_nb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dFWmx3iO7qBe"
   },
   "source": [
    "- Realizamos las predicciones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZQJqcaB18PDr"
   },
   "outputs": [],
   "source": [
    "nb = MNB.predict(test_corpus_nb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BAhi1bh87wGu"
   },
   "source": [
    "- Mostramos el \"accuracy\" del modelo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rJdVgGapKfEh",
    "outputId": "8d8c019f-92ba-415b-f060-a5e0362d7e9c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy of the MNB classifier is 0.6529588766298897\n"
     ]
    }
   ],
   "source": [
    "print(\"The accuracy of the MNB classifier is\",\n",
    "      accuracy_score(test_labels_nb, nb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ehQKYsVDKrUv",
    "outputId": "3837a92e-1930-4596-da2d-6cc789221e41"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The classification report with metrics -\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      "      CIENCIA       0.95      0.81      0.88       170\n",
      "     DEPORTES       0.99      0.84      0.91       179\n",
      "     ECONOMÍA       0.79      0.12      0.21        92\n",
      "INTERNACIONAL       0.38      0.98      0.55       209\n",
      "     POLÍTICA       0.95      0.72      0.82       197\n",
      "     SOCIEDAD       1.00      0.04      0.08       150\n",
      "\n",
      "     accuracy                           0.65       997\n",
      "    macro avg       0.84      0.58      0.57       997\n",
      " weighted avg       0.83      0.65      0.62       997\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\nThe classification report with metrics -\\n\",\n",
    "      classification_report(test_labels_nb, nb))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gkeY4DTl-wPK"
   },
   "source": [
    "#### Conclusiones:\n",
    "\n",
    "Parece que este modelo no funciona bien en comparación al resto que hemos probado anteriormente.\n",
    "\n",
    "De nuevo, observamos que el modelo trabaja especialmente mal con el conjunto de artículos de 'SOCIEDAD' y, en menor medida, con los de 'ECONOMÍA'. Por una parte, en ambas categorías observamos un valor muy alto de 'precision', lo cual quiere decir que no predecimos mal los artículos que clasificamos con estas etiquetas. En cambio, cuando nos fijamos en su 'recall' correspondiente, vemos que este tiene un valor muy bajo, lo cual indica que a pesar de tener una precisión alta, hay muchos artículos pertenecientes a estas categorías que hemos clasificado de manera errónea."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HFOZVqGXdR00"
   },
   "source": [
    "#### Modelo bert-base-spanish-cased\n",
    "\n",
    "Usaremos bert-base-spanish-cased porque es uno de los mejores transformers para español."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dEiUIWRf-OEh"
   },
   "source": [
    "Probamos nuestro modelo sin ninguno de los preprocesados definidos anteriormente, ya que el transformer preprocesa los datos internamente."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L3aCYbsC9Xgr"
   },
   "source": [
    "- Instalamos la librería"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "--janza_rXgQ"
   },
   "outputs": [],
   "source": [
    "!pip install transformers numpy torch sklearn emoji"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l-XAKn2Z9bDE"
   },
   "source": [
    "- Definimos el modelo y el tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 212,
     "referenced_widgets": [
      "8edb3d6300a44fe6af994330c3b4e685",
      "05b35338591a4d8aa1ccef081d16d8c8",
      "d85a5d13db8f411f981f6800464bd8ed",
      "dfd45196e772431cbc776343590842df",
      "104ecad68ef9439e9f786f95e3342959",
      "4bfa72bf59434cfda04b82a6c5d52daf",
      "d7e1db8662944ca58bd420abe19e9186",
      "9e1ba3590dbb4f68b59ebd69774508ef",
      "36de27cc23a24eacb84f13eb852661d0",
      "78c19c89717c4e72bca859f123be477c",
      "48dd604eaa934acb9fde1fd6ca7a04ec",
      "f7b02d68eb2142d99cd068b8e6d99b2c",
      "61ce072092464effb2b317d19a620cce",
      "acc53157ee8e464fb53c725443e19d96",
      "dab8d6df95a940dab2e83ba15fbfbe2e",
      "129d55ebc44049dbaddf1efb6089fab7",
      "21ac8d5e100445cb97904ab383541b4b",
      "8e3b86ec162b418ebcb9c389fd7ac904",
      "db1d600a1d934296b3e0f38c1d965231",
      "222ed53fd6b441e88130612438d8efc8",
      "e833a36f3cbb4c8aaee7c48ff3960e76",
      "49f35aa94cb64388a981be09cceeefd8",
      "d6558298d72545d782082e310be697a4",
      "4cde5d4857c14980bf07cbfc662858d0",
      "0bf6ba38640f4e488f0df8f3aa0c89b8",
      "53284984acbb4f989087715e661b24bf",
      "d1e3b1369c7d4398aa3f0b0be77fddb7",
      "2739afbb030e43e995caa8d5d755aa69",
      "43fa536db384489eb35eeb8f992a98c8",
      "d0a3df9b4d0549d88c75610d56d6f096",
      "35a3df50ab114afdbd4df612d4e1e3d8",
      "2df5d72bd0794ae58c07d2f1f782909e"
     ]
    },
    "id": "KE9ddoJWrRFP",
    "outputId": "075d2db6-11a9-4d41-b474-72ddd8e03862"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8edb3d6300a44fe6af994330c3b4e685",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=241796.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "36de27cc23a24eacb84f13eb852661d0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=480199.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "21ac8d5e100445cb97904ab383541b4b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=134.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0bf6ba38640f4e488f0df8f3aa0c89b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=364.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "model_name = \"dccuchile/bert-base-spanish-wwm-cased\"\n",
    "tokenizer = BertTokenizerFast.from_pretrained(model_name, do_lower_case=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ydu3ArOgoELy"
   },
   "source": [
    "- Transformamos la columna CATEGORÍA a numérico para poder trabajar con ella"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 402
    },
    "id": "S2sglPGj_zLT",
    "outputId": "1aeca764-17d0-4d3f-8c72-417054a1a9b3"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CATEGORÍA</th>\n",
       "      <th>ENLACE</th>\n",
       "      <th>TEXTO</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>https://elpais.com/television/2021-05-20/la-di...</td>\n",
       "      <td>El Festival de Eurovisión es una tradición mus...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>https://elpais.com/television/2021-05-20/eurov...</td>\n",
       "      <td>La recta final del Festival de Eurovisión 2021...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>https://elpais.com/economia/2021-05-20/crecen-...</td>\n",
       "      <td>Se dice que el mercado de divisas nunca duerme...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>https://elpais.com/economia/2021-05-20/la-puja...</td>\n",
       "      <td>El precio de las casas se ha disparado en país...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>https://elpais.com/opinion/2021-05-20/drama-y-...</td>\n",
       "      <td>Me ha sorprendido el tono dramático que ha uti...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2984</th>\n",
       "      <td>5</td>\n",
       "      <td>https://elpais.com/espana/catalunya/2020-04-20...</td>\n",
       "      <td>Una investigación sorprendente ha unido en el ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2985</th>\n",
       "      <td>5</td>\n",
       "      <td>https://elpais.com/ciencia/2020-04-01/recupera...</td>\n",
       "      <td>Hace más de 25 años, un grupo de arqueólogos y...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2986</th>\n",
       "      <td>5</td>\n",
       "      <td>https://elpais.com/economia/2020-03-30/los-din...</td>\n",
       "      <td>Los dinosaurios, aunque sean fósiles o recreac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2987</th>\n",
       "      <td>5</td>\n",
       "      <td>https://elpais.com/ciencia/2020-03-18/un-fosil...</td>\n",
       "      <td>En el interior de una piedra que no mide mucho...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2988</th>\n",
       "      <td>5</td>\n",
       "      <td>https://elpais.com/ciencia/2020-03-11/encontra...</td>\n",
       "      <td>En lo que hoy es el norte de Myanmar se han en...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2989 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      CATEGORÍA  ...                                              TEXTO\n",
       "0             0  ...  El Festival de Eurovisión es una tradición mus...\n",
       "1             0  ...  La recta final del Festival de Eurovisión 2021...\n",
       "2             0  ...  Se dice que el mercado de divisas nunca duerme...\n",
       "3             0  ...  El precio de las casas se ha disparado en país...\n",
       "4             0  ...  Me ha sorprendido el tono dramático que ha uti...\n",
       "...         ...  ...                                                ...\n",
       "2984          5  ...  Una investigación sorprendente ha unido en el ...\n",
       "2985          5  ...  Hace más de 25 años, un grupo de arqueólogos y...\n",
       "2986          5  ...  Los dinosaurios, aunque sean fósiles o recreac...\n",
       "2987          5  ...  En el interior de una piedra que no mide mucho...\n",
       "2988          5  ...  En lo que hoy es el norte de Myanmar se han en...\n",
       "\n",
       "[2989 rows x 3 columns]"
      ]
     },
     "execution_count": 91,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapping = {value:i for i, value in enumerate(tabla[\"CATEGORÍA\"].unique())}\n",
    "tabla[\"CATEGORÍA\"] = [mapping[cat] for cat in tabla[\"CATEGORÍA\"]]\n",
    "tabla"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J0MceYT5oWmV"
   },
   "source": [
    "- Dividimos los datos en train y test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OrU4HxZh_7JK"
   },
   "outputs": [],
   "source": [
    "train_text = df[\"TEXTO\"].values\n",
    "train_labels = df[\"CATEGORÍA\"].values\n",
    "train_texts, valid_texts, train_labels, valid_labels = train_test_split(list(train_text),\n",
    "                                                                     list(train_labels),\n",
    "                                                                     test_size = 0.2)\n",
    "texts = list(df[\"TEXTO\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "luunJSfWocup"
   },
   "source": [
    "- Sacamos la media de la longitud de todos los textos para optimizar el modelo. Como mucho medirá el 400 (hemos comprobado que es el máximo que modelo soporta) y si se queda corto hará un padding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fm9IKnjE_9g-",
    "outputId": "0cfe868d-3069-46ff-c275-144857d9ac4c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "400"
      ]
     },
     "execution_count": 93,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "leng = [len(txt) for txt in texts]\n",
    "max_length = 400\n",
    "max_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1NG9F71AofpJ"
   },
   "source": [
    "- Convertimos los textos a secuencia de tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5gI-WGXTrRIP"
   },
   "outputs": [],
   "source": [
    "train_encodings = tokenizer(train_texts, truncation=True, padding=True, max_length=max_length)\n",
    "valid_encodings = tokenizer(valid_texts, truncation=True, padding=True, max_length=max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E2aKd4DKoixI"
   },
   "source": [
    "- Empaquetamos nuestros datos de texto tokenizados en un objeto Torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SciYqDfNrRK-"
   },
   "outputs": [],
   "source": [
    "class DetoxisDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self, encodings, labels):\n",
    "        self.encodings = encodings\n",
    "        self.labels = labels\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}\n",
    "        item[\"labels\"] = torch.tensor([self.labels[idx]])\n",
    "        return item\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YgVBtgc1rRNI"
   },
   "outputs": [],
   "source": [
    "train_dataset = DetoxisDataset(train_encodings, train_labels)\n",
    "valid_dataset = DetoxisDataset(valid_encodings, valid_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c_3q_OnMom0r"
   },
   "source": [
    "- Descargamos y cargamos el modelo BERT y sus pesos previamente entrenados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lRd9NTwZrRPv",
    "outputId": "de9d8117-9b38-4caf-deb8-7039a901a32a"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at dccuchile/bert-base-spanish-wwm-cased were not used when initializing BertForSequenceClassification: ['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.predictions.decoder.bias', 'cls.predictions.transform.dense.weight']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at dccuchile/bert-base-spanish-wwm-cased and are newly initialized: ['classifier.weight', 'bert.pooler.dense.bias', 'classifier.bias', 'bert.pooler.dense.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = BertForSequenceClassification.from_pretrained(model_name, num_labels=6).to(\"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nqw4fYUx8aIE"
   },
   "outputs": [],
   "source": [
    "def compute_metrics(pred):\n",
    "  labels = pred.label_ids\n",
    "  preds = pred.predictions.argmax(-1)\n",
    "  acc = accuracy_score(labels, preds)\n",
    "  return {'accuracy': acc,}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yo2Q8yv4otoK"
   },
   "source": [
    "- Elegimos los parámetros de entrenamiento para nuestro modelo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2I45LSdY8c5V"
   },
   "outputs": [],
   "source": [
    "training_args = TrainingArguments(\n",
    "    output_dir='./results',          # output directory\n",
    "    num_train_epochs=4,             # total number of training epochs\n",
    "    per_device_train_batch_size=16,  # batch size per device during training\n",
    "    per_device_eval_batch_size=20,   # batch size for evaluation\n",
    "    warmup_steps=500,                # number of warmup steps for learning rate scheduler\n",
    "    weight_decay=0.01,               # strength of weight decay\n",
    "    logging_dir='./logs',            # directory for storing logs\n",
    "    load_best_model_at_end=True,     # load the best model when finished training (default metric is loss)\n",
    "    # but you can specify `metric_for_best_model` argument to change to accuracy or other metric\n",
    "    logging_steps=200,               # log & save weights each logging_steps\n",
    "    evaluation_strategy=\"steps\",     # evaluate each `logging_steps`\n",
    "    learning_rate = 0.00001\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AQyDd6yj8iqF"
   },
   "outputs": [],
   "source": [
    "trainer = Trainer(\n",
    "    model=model,                         # the instantiated Transformers model to be trained\n",
    "    args=training_args,                  # training arguments, defined above\n",
    "    train_dataset=train_dataset,         # training dataset\n",
    "    eval_dataset=valid_dataset,          # evaluation dataset\n",
    "    com pute_metrics=compute_metrics,     # the callback that computes metrics of interest\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OIOQHfK49vOr"
   },
   "source": [
    "- Entrenamos el modelo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 199
    },
    "id": "H-RHfjNN8t7S",
    "outputId": "6e237b5d-bd24-4268-dba7-dff9709834d6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='600' max='600' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [600/600 15:24, Epoch 4/4]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>1.575000</td>\n",
       "      <td>0.991071</td>\n",
       "      <td>0.702341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>0.645200</td>\n",
       "      <td>0.547331</td>\n",
       "      <td>0.826087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>0.405900</td>\n",
       "      <td>0.525983</td>\n",
       "      <td>0.841137</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=600, training_loss=0.8753809102376302, metrics={'train_runtime': 926.45, 'train_samples_per_second': 0.648, 'total_flos': 0, 'epoch': 4.0, 'init_mem_cpu_alloc_delta': 0, 'init_mem_gpu_alloc_delta': 0, 'init_mem_cpu_peaked_delta': 0, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': -454930432, 'train_mem_gpu_alloc_delta': 1323734528, 'train_mem_cpu_peaked_delta': 459337728, 'train_mem_gpu_peaked_delta': 8898516992})"
      ]
     },
     "execution_count": 101,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YH_mLC7y9zjd"
   },
   "source": [
    "- Evaluamos el modelo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 188
    },
    "id": "d3aauR08-CK0",
    "outputId": "edda5985-5ee9-4752-e844-741fb2339e13"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='30' max='30' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [30/30 00:16]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'epoch': 4.0,\n",
       " 'eval_accuracy': 0.8411371237458194,\n",
       " 'eval_loss': 0.5259833335876465,\n",
       " 'eval_mem_cpu_alloc_delta': 0,\n",
       " 'eval_mem_cpu_peaked_delta': 0,\n",
       " 'eval_mem_gpu_alloc_delta': 0,\n",
       " 'eval_mem_gpu_peaked_delta': 507055104,\n",
       " 'eval_runtime': 17.1893,\n",
       " 'eval_samples_per_second': 34.789}"
      ]
     },
     "execution_count": 102,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7UqL55OpBjk"
   },
   "source": [
    "- Creamos una función que recibirá un texto como entrada y devolverá la predicción del modelo (la categoría) como salida."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "n4_dn3cl-F6u"
   },
   "outputs": [],
   "source": [
    "def get_prediction(text,max_length):\n",
    "    # prepare our text into tokenized sequence\n",
    "    inputs = tokenizer(text, padding=True, truncation=True, max_length=max_length, return_tensors=\"pt\").to(\"cuda\")\n",
    "    # perform inference to our model\n",
    "    outputs = model(**inputs)\n",
    "    # get output probabilities by doing softmax\n",
    "    probs = outputs[0].softmax(1)\n",
    "    # executing argmax function to get the candidate label\n",
    "    return probs.argmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zIGI9Tnk-IE0",
    "outputId": "eb9543d9-600d-4081-d307-47be06b5209a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8323836685023083\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "pred = [int(get_prediction(text, max_length)) for text in valid_texts]\n",
    "score = f1_score(valid_labels, pred, average='macro')\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SHKtOULOm4SI"
   },
   "source": [
    "#### Conclusiones:\n",
    "\n",
    "Con este modelo obtenemos un \"accuracy\" bastante alto pero, al igual que el modelo creado por nosotros, no supera al \"Linear SVM Model\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DpsmcGw9_J7t"
   },
   "source": [
    "### Conclusión final \n",
    "\n",
    "Recordamos que, tras haber probado distintas combinaciones de preprocesados, vectorizadores y modelos, el mejor resultado ha sido un \"accuracy\" de 0.853 obtenido con el modelo Linear SVM, el vectorizador LSA y el preprocesado normaliza-bis. Cabe mencionar que los demás modelos obtenidos también han proporcionado un \"accuracy\" cercano al anterior, alrededor de 0.83 aproximadamente.\n",
    "\n",
    "Además, destacamos el problema encontrado con la categoría 'SOCIEDAD' en los modelos: \"MultinomialNaiveBayes\" con NLTK y \"Linear SVM, con el vectorizador LSA y el preprocesado normaliza-bis\". En ambos hemos encontrado dificultades para clasificar aquellos artículos con esta categoría. Es probable que, aunque no lo hemos comprobado, el resto de modelos también sufrieran en parte una bajada del \"accuracy\" debido a la dificultad de clasificar los artículos de dicha categoría."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Miniproyecto_PLN_final.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "05b35338591a4d8aa1ccef081d16d8c8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0bf6ba38640f4e488f0df8f3aa0c89b8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d1e3b1369c7d4398aa3f0b0be77fddb7",
       "IPY_MODEL_2739afbb030e43e995caa8d5d755aa69"
      ],
      "layout": "IPY_MODEL_53284984acbb4f989087715e661b24bf"
     }
    },
    "104ecad68ef9439e9f786f95e3342959": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "129d55ebc44049dbaddf1efb6089fab7": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "21ac8d5e100445cb97904ab383541b4b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_db1d600a1d934296b3e0f38c1d965231",
       "IPY_MODEL_222ed53fd6b441e88130612438d8efc8"
      ],
      "layout": "IPY_MODEL_8e3b86ec162b418ebcb9c389fd7ac904"
     }
    },
    "222ed53fd6b441e88130612438d8efc8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4cde5d4857c14980bf07cbfc662858d0",
      "placeholder": "​",
      "style": "IPY_MODEL_d6558298d72545d782082e310be697a4",
      "value": " 134/134 [00:05&lt;00:00, 23.3B/s]"
     }
    },
    "2739afbb030e43e995caa8d5d755aa69": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2df5d72bd0794ae58c07d2f1f782909e",
      "placeholder": "​",
      "style": "IPY_MODEL_35a3df50ab114afdbd4df612d4e1e3d8",
      "value": " 364/364 [00:05&lt;00:00, 70.9B/s]"
     }
    },
    "2df5d72bd0794ae58c07d2f1f782909e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "35a3df50ab114afdbd4df612d4e1e3d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "36de27cc23a24eacb84f13eb852661d0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_48dd604eaa934acb9fde1fd6ca7a04ec",
       "IPY_MODEL_f7b02d68eb2142d99cd068b8e6d99b2c"
      ],
      "layout": "IPY_MODEL_78c19c89717c4e72bca859f123be477c"
     }
    },
    "43fa536db384489eb35eeb8f992a98c8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "48dd604eaa934acb9fde1fd6ca7a04ec": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_acc53157ee8e464fb53c725443e19d96",
      "max": 480199,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_61ce072092464effb2b317d19a620cce",
      "value": 480199
     }
    },
    "49f35aa94cb64388a981be09cceeefd8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4bfa72bf59434cfda04b82a6c5d52daf": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4cde5d4857c14980bf07cbfc662858d0": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "53284984acbb4f989087715e661b24bf": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "61ce072092464effb2b317d19a620cce": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "78c19c89717c4e72bca859f123be477c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8e3b86ec162b418ebcb9c389fd7ac904": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8edb3d6300a44fe6af994330c3b4e685": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d85a5d13db8f411f981f6800464bd8ed",
       "IPY_MODEL_dfd45196e772431cbc776343590842df"
      ],
      "layout": "IPY_MODEL_05b35338591a4d8aa1ccef081d16d8c8"
     }
    },
    "9e1ba3590dbb4f68b59ebd69774508ef": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "acc53157ee8e464fb53c725443e19d96": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d0a3df9b4d0549d88c75610d56d6f096": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d1e3b1369c7d4398aa3f0b0be77fddb7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d0a3df9b4d0549d88c75610d56d6f096",
      "max": 364,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_43fa536db384489eb35eeb8f992a98c8",
      "value": 364
     }
    },
    "d6558298d72545d782082e310be697a4": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d7e1db8662944ca58bd420abe19e9186": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d85a5d13db8f411f981f6800464bd8ed": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4bfa72bf59434cfda04b82a6c5d52daf",
      "max": 241796,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_104ecad68ef9439e9f786f95e3342959",
      "value": 241796
     }
    },
    "dab8d6df95a940dab2e83ba15fbfbe2e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "db1d600a1d934296b3e0f38c1d965231": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_49f35aa94cb64388a981be09cceeefd8",
      "max": 134,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e833a36f3cbb4c8aaee7c48ff3960e76",
      "value": 134
     }
    },
    "dfd45196e772431cbc776343590842df": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9e1ba3590dbb4f68b59ebd69774508ef",
      "placeholder": "​",
      "style": "IPY_MODEL_d7e1db8662944ca58bd420abe19e9186",
      "value": " 242k/242k [00:00&lt;00:00, 306kB/s]"
     }
    },
    "e833a36f3cbb4c8aaee7c48ff3960e76": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "f7b02d68eb2142d99cd068b8e6d99b2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_129d55ebc44049dbaddf1efb6089fab7",
      "placeholder": "​",
      "style": "IPY_MODEL_dab8d6df95a940dab2e83ba15fbfbe2e",
      "value": " 480k/480k [00:06&lt;00:00, 70.4kB/s]"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}