finetune/PyTorch/notebooks/BERT_Eval_GLUE.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Pretrained BERT on AzureML with GLUE Dataset\n",
    "\n",
    "In this notebook, you will find the following contents:\n",
    "- Download GLUE dataset on the remote compute and store them in Azure storage\n",
    "- Speed-up fine-tuning BERT for GLUE dataset on AzureML GPU clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "Follow instructions in BERT_pretraining.ipynb notebook for setting up AzureML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check core SDK version number\n",
    "import azureml.core\n",
    "\n",
    "print(\"SDK version:\", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize workspace\n",
    "\n",
    "To create or access an Azure ML Workspace, you will need to import the AML library and the following information:\n",
    "* A name for your workspace\n",
    "* Your subscription id\n",
    "* The resource group name\n",
    "\n",
    "Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step or create a new one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.workspace import Workspace\n",
    "ws = Workspace.setup()\n",
    "ws_details = ws.get_details()\n",
    "print('Name:\\t\\t{}\\nLocation:\\t{}'\n",
    "      .format(ws_details['name'],\n",
    "              ws_details['location']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an experiment\n",
    "Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace for this distributed PyTorch tutorial. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download GLUE dataset on the remote compute\n",
    "\n",
    "Before we start to fine-tune the pretained BERT model, we need to download the [GLUE data](https://gluebenchmark.com/tasks) by running the [script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e) and unpack it to an Azure Blob container."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define AzureML datastore to collect training dataset\n",
    "\n",
    "To make data accessible for remote training, AML provides a convenient way to do so via a [Datastore](https://docs.microsoft.com/azure/machine-learning/service/how-to-access-data). The datastore provides a mechanism for you to upload/download data to Azure Storage, and interact with it from your remote compute targets.\n",
    "\n",
    "Each workspace is associated with a default Azure Blob datastore named `'workspaceblobstore'`. In this work, we use this default datastore to collect the GLUE training dataset ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Datastore\n",
    "ds = ws.get_default_datastore()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a project directory\n",
    "Create a directory that will contain all the necessary code from your local machine that you will need access to on the remote resource. This includes the training script and any additional files your training script depends on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import os.path as path\n",
    "project_root = path.abspath(path.join(os.getcwd(),\"../../../\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download GLUE dataset in BingBert/ directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.upload(src_dir=os.path.join(project_root,'data','glue_data'), target_path='glue_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a folder named \"bert-large-checkpoints\" which contains the .pt bert checkpoint file against which you want to run your eval tasks. The following code will upload the folder to the datastore. The URL for the checkpoint is: https://bertonazuremlwestus2.blob.core.windows.net/public/models/bert_large_uncased_original/bert_encoder_epoch_200.pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.upload(src_dir=os.path.join(project_root,'data','bert-large-checkpoints') , target_path='bert-large-checkpoints')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uploading bert-large config file to datastore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ds.upload(src_dir=os.path.join(project_root,'pretrain','configs'), target_path='config')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Remove /data folder to avoid uploading folder greater than 300MB.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning BERT with Distributed Training\n",
    "As our `GLUE` dataset are ready in Azure storage, we can start the fine-tune the model by exploting the power of distributed training. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a GPU remote compute target\n",
    "\n",
    "We need to create a GPU [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) to perform the fine-tuning. In this example, we create an AmlCompute cluster as our training compute resource.\n",
    "\n",
    "This code creates a cluster for you if it does not already exist in your workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# choose a name for your cluster\n",
    "gpu_cluster_name = \"bertcodetesting\"\n",
    "\n",
    "try:\n",
    "    gpu_compute_target = ComputeTarget(workspace=ws, name=gpu_cluster_name)\n",
    "    print('Found existing compute target.')\n",
    "except ComputeTargetException:\n",
    "    print('Creating a new compute target...')\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC24', max_nodes=4)\n",
    "\n",
    "    # create the cluster\n",
    "    gpu_compute_target = ComputeTarget.create(ws, gpu_cluster_name, compute_config)\n",
    "    gpu_compute_target.wait_for_completion(show_output=True)\n",
    "\n",
    "# Use the 'status' property to get a detailed status for the current cluster. \n",
    "print(gpu_compute_target.status.serialize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a PyTorch estimator for fine-tuning\n",
    "Let us create a new PyTorch estimator to run the fine-tuning script `run_classifier.py`, that is already provided at [the original repository](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py). Please refer [here](https://github.com/huggingface/pytorch-pretrained-BERT#fine-tuning-with-bert-running-the-examples) for more detail about the script. \n",
    "\n",
    "The original `run_classifier.py` script uses PyTorch distributed launch untility to launch multiple processes across nodes and GPUs. We prepared a modified version [run_classifier_azureml.py](./run_classifier_azureml.py) so that we can launch it based on AzureML build-in MPI backend.\n",
    "\n",
    "To use AML's tracking and metrics capabilities, we need to add a small amount of AzureML code inside the training script.\n",
    "\n",
    "In `run_classifier_azureml.py`, we will log some metrics to our AML run. To do so, we will access the AML run object within the script:\n",
    "```Python\n",
    "from azureml.core.run import Run\n",
    "run = Run.get_context()\n",
    "```\n",
    "Further within `run_classifier_azureml.py`, we log learning rate, training loss and evaluation accuracy the model achieves as:\n",
    "```Python\n",
    "run.log('lr', np.float(args.learning_rate))\n",
    "...\n",
    "\n",
    "for step, batch in enumerate(tqdm(train_dataloader, desc=\"Iteration\")): \n",
    "    ...\n",
    "    run.log('train_loss', np.float(loss))\n",
    "\n",
    "...\n",
    "\n",
    "result = {'eval_loss': eval_loss,\n",
    "          'eval_accuracy': eval_accuracy}\n",
    "for key in sorted(result.keys()):\n",
    "    run.log(key, str(result[key]))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code runs GLUE RTE task against a bert-large checkpoint with the parameters used by Huggingface for finetuning.\n",
    "- num_train_epochs = 3\n",
    "- max_seq_length = 128\n",
    "- train_batch_size = 8\n",
    "- learning_rate = 2e-5\n",
    "- grad_accumulation_step = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.train.dnn import PyTorch\n",
    "from azureml.core.runconfig import RunConfiguration\n",
    "from azureml.core.container_registry import ContainerRegistry\n",
    "\n",
    "run_user_managed = RunConfiguration()\n",
    "run_user_managed.environment.python.user_managed_dependencies = True\n",
    "\n",
    "# Using a pre-defined public docker image published on AzureML\n",
    "image_name = 'mcr.microsoft.com/azureml/bert:pretrain-openmpi3.1.2-cuda10.0-cudnn7-ubuntu16.04'\n",
    "\n",
    "estimator = PyTorch(source_directory='../../../',\n",
    "                    compute_target=gpu_compute_target,\n",
    "                     #Docker image\n",
    "                    use_docker=True,\n",
    "                    custom_docker_image=image_name,\n",
    "                    user_managed=True,\n",
    "                    \n",
    "                    script_params = {\n",
    "                          '--bert_model':'bert-large-uncased',\n",
    "                          \"--model_file_location\": ds.path('bert-large-checkpoints/').as_mount(),\n",
    "                          '--task_name': 'RTE',\n",
    "                          '--data_dir': ds.path('glue_data/RTE/').as_mount(),\n",
    "                          '--do_train' : '',\n",
    "                          '--do_eval': '',                      \n",
    "                          '--do_lower_case': '',\n",
    "                          '--max_seq_length': 128,\n",
    "                          '--train_batch_size': 8,\n",
    "                          '--gradient_accumulation_steps': 2,\n",
    "                          '--learning_rate': 2e-5,\n",
    "                          '--num_train_epochs': 3.0,\n",
    "                          '--output_dir': ds.path('output/').as_mount(),\n",
    "                          '--model_file': 'bert_encoder_epoch_245.pt',\n",
    "                          '--fp16': \"\"\n",
    "                    },\n",
    "                    entry_script='./finetune/run_classifier_azureml.py',\n",
    "                    node_count=1,\n",
    "                    process_count_per_node=4,\n",
    "                    distributed_backend='mpi',\n",
    "                    use_gpu=True)\n",
    "\n",
    "# path to the Python environment in the custom Docker image\n",
    "estimator._estimator_config.environment.python.interpreter_path = '/opt/miniconda/envs/amlbert/bin/python'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit and Monitor your run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Experiment\n",
    "\n",
    "experiment_name = 'bert-large-RTE'\n",
    "experiment = Experiment(ws, name=experiment_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run = experiment.submit(estimator)\n",
    "from azureml.widgets import RunDetails\n",
    "RunDetails(run).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#run.cancel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "aagarg"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "msauthor": "aagarg"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}