AI_Planet_Bootcamp_Final_Assignment_Fine_tuning_Phi_1_5b.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Code Generation application - fine-tuning LLMs using QloRA"
      ],
      "metadata": {
        "id": "g7FFf31LGkTT"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C2EgqEPDQ8v6"
      },
      "source": [
        "## Fine-tune phi 1.5b on Google colab\n",
        "\n",
        "This jupyter notebook aims to fine-tune phi 1.5b on evolved codealpaca dataset as a sample for showing how to fine-tune LMs using QloRA"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i-tTvEF1RT3y"
      },
      "source": [
        "## Setup\n",
        "\n",
        "Run the cells below to setup and install the required libraries. For our experiment we will need `accelerate`, `peft`, `transformers`, `datasets`,`scipy` and `TRL` to leverage [`SFTTrainer`](https://huggingface.co/docs/trl/main/en/sft_trainer). We will use `bitsandbytes` to [quantize the base model into 4bit](https://huggingface.co/blog/4bit-transformers-bitsandbytes). We will also install `einops` but it was mainly used for loading falcon so I will remove it in later versions."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!nvidia-smi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jcmEhx0bjFUp",
        "outputId": "e9d5f4ae-1d0e-4a7c-cd76-b4a7ebb32141"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sun Oct 15 15:35:26 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   44C    P8     9W /  70W |      0MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "print(f\"Is CUDA supported by this system?{torch.cuda.is_available()}\")\n",
        "print(f\"CUDA version: {torch.version.cuda}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5M4C8Y7pjO0d",
        "outputId": "2bc46fda-bc70-464a-daa0-cec116dbcc23"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Is CUDA supported by this system?True\n",
            "CUDA version: 11.8\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mNnkgBq7Q3EU",
        "outputId": "7c1d4d5b-e157-4736-ee0f-06a1b87fd9ed",
        "scrolled": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: transformers==4.30 in /usr/local/lib/python3.10/dist-packages (4.30.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (3.12.4)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (0.18.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (1.23.5)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (23.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (6.0.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (2023.6.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (2.31.0)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (0.13.3)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (0.4.0)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30) (4.66.1)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.30) (2023.6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.30) (4.5.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.30) (3.3.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.30) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.30) (2.0.6)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.30) (2023.7.22)\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!pip install transformers==4.30\n",
        "!pip install -q -U trl accelerate sentencepiece git+https://github.com/huggingface/peft.git\n",
        "!pip install -q -U datasets bitsandbytes einops scipy wandb"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rnqmq7amRrU8"
      },
      "source": [
        "## Dataset\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0X3kHnskSWU4"
      },
      "outputs": [],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "dataset_name = 'theblackcat102/evol-codealpaca-v1'\n",
        "dataset = load_dataset(dataset_name, split=\"train\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dataset"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d4KaXEsjfVZx",
        "outputId": "cca3aa5e-0c1f-403e-fa82-fdbea99e74ce"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['instruction', 'output'],\n",
              "    num_rows: 111272\n",
              "})"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dataset[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-tCTXV-SfaUN",
        "outputId": "6183a2c8-b59f-4f7a-f782-90f1d74357ca"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'instruction': \"Please amend the subsequent Python script so that it includes a 'while' loop rather than the existing 'for' loop, which iterates through the items of an integer list.\\n\\nThe script currently has a bug where it attempts to print an object that is outside the bounds of the list. Fix this error and modify the script to use 'while' instead of 'for' loop. Ensure your script correctly handles empty lists. \\n\\n```python\\n  # Establish an integer list\\n  arr = [1, 2, 3, 4]\\n\\n  # Determine the length of the list\\n  n = len(arr)\\n\\n  # Traverse the list and output each individual element\\n  for i in range(n+1):\\n      print(arr[i])\\n```\",\n",
              " 'output': '```python\\n# Establish an integer list\\narr = [1, 2, 3, 4]\\n\\n# Determine the length of the list\\nn = len(arr)\\n\\n# Initialize index at 0\\ni = 0\\n\\n# Traverse the list and output each individual element\\nwhile i < n:\\n    print(arr[i])\\n    i += 1\\n```\\nIn the given code, it tries to access `arr[n]` which is out of bounds as python list indexing starts from 0 and ends at n-1 for a list of length n.\\nAlso, I used `while` loop as per your request. By initializing `i=0` before the loop and `i += 1` within the loop, we can iterate over the list. This iteration continues until `i` is less than `n`, the length of the list. The `while` loop will not proceed if the list is empty so it correctly handles empty lists.'}"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(dataset)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wDogoQcbyF8r",
        "outputId": "dc3e39f6-5cc4-4752-df73-7abcb4343385"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "111272"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "type(dataset)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gDZaPc0jyNZT",
        "outputId": "07a3aaec-7e47-424f-c5e3-9af392e24a64"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "datasets.arrow_dataset.Dataset"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# select only 100 records to minimise training time\n",
        "dataset_sub = dataset.select(range(1000))\n",
        "print(len(dataset_sub))\n",
        "dataset_sub[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t2wScHxG0a6a",
        "outputId": "3fdb1116-1a19-4dc1-bae1-707d792e53b5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "100\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'instruction': \"Please amend the subsequent Python script so that it includes a 'while' loop rather than the existing 'for' loop, which iterates through the items of an integer list.\\n\\nThe script currently has a bug where it attempts to print an object that is outside the bounds of the list. Fix this error and modify the script to use 'while' instead of 'for' loop. Ensure your script correctly handles empty lists. \\n\\n```python\\n  # Establish an integer list\\n  arr = [1, 2, 3, 4]\\n\\n  # Determine the length of the list\\n  n = len(arr)\\n\\n  # Traverse the list and output each individual element\\n  for i in range(n+1):\\n      print(arr[i])\\n```\",\n",
              " 'output': '```python\\n# Establish an integer list\\narr = [1, 2, 3, 4]\\n\\n# Determine the length of the list\\nn = len(arr)\\n\\n# Initialize index at 0\\ni = 0\\n\\n# Traverse the list and output each individual element\\nwhile i < n:\\n    print(arr[i])\\n    i += 1\\n```\\nIn the given code, it tries to access `arr[n]` which is out of bounds as python list indexing starts from 0 and ends at n-1 for a list of length n.\\nAlso, I used `while` loop as per your request. By initializing `i=0` before the loop and `i += 1` within the loop, we can iterate over the list. This iteration continues until `i` is less than `n`, the length of the list. The `while` loop will not proceed if the list is empty so it correctly handles empty lists.'}"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rjOMoSbGSxx9"
      },
      "source": [
        "## Loading the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZwXZbQ2dSwzI"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoTokenizer\n",
        "\n",
        "model_name = \"microsoft/phi-1_5\"\n",
        "\n",
        "bnb_config = BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_quant_type=\"nf4\",\n",
        "    bnb_4bit_compute_dtype=torch.bfloat16,\n",
        ")\n",
        "\n",
        "model = AutoModelForCausalLM.from_pretrained(\n",
        "    model_name,\n",
        "    quantization_config=bnb_config,\n",
        "    trust_remote_code=True\n",
        ")\n",
        "model.config.use_cache = False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xNqIYtQcUBSm"
      },
      "source": [
        "Let's also load the tokenizer below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XDS2yYmlUAD6"
      },
      "outputs": [],
      "source": [
        "tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "tokenizer.padding_side = \"right\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dQdvjTYTT1vQ"
      },
      "outputs": [],
      "source": [
        "from peft import LoraConfig, get_peft_model\n",
        "\n",
        "lora_alpha = 16\n",
        "lora_dropout = 0.1\n",
        "lora_r = 64\n",
        "\n",
        "peft_config = LoraConfig(\n",
        "    lora_alpha=lora_alpha,\n",
        "    lora_dropout=lora_dropout,\n",
        "    target_modules=[\"fc1\", \"fc2\",\"Wqkv\", \"out_proj\"],\n",
        "    r=lora_r,\n",
        "    bias=\"none\",\n",
        "    task_type=\"CAUSAL_LM\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dzsYHLwIZoLm"
      },
      "source": [
        "## Loading the trainer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aTBJVE4PaJwK"
      },
      "source": [
        "Here we will use the [`SFTTrainer` from TRL library](https://huggingface.co/docs/trl/main/en/sft_trainer) that gives a wrapper around transformers `Trainer` to easily fine-tune models on instruction based datasets using PEFT adapters. Let's first load the training arguments below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OCFTvGW6aspE"
      },
      "outputs": [],
      "source": [
        "from transformers import TrainingArguments\n",
        "\n",
        "output_dir = \"./results\"\n",
        "per_device_train_batch_size = 1\n",
        "gradient_accumulation_steps = 1\n",
        "optim = \"paged_adamw_32bit\"\n",
        "save_steps = 100\n",
        "logging_steps = 10\n",
        "learning_rate = 4e-3\n",
        "max_grad_norm = 0.3\n",
        "max_steps = 100 #max_steps = -1\n",
        "warmup_ratio = 0.03\n",
        "lr_scheduler_type = \"constant\"\n",
        "\n",
        "training_arguments = TrainingArguments(\n",
        "    output_dir=output_dir,\n",
        "    per_device_train_batch_size=per_device_train_batch_size,\n",
        "    gradient_accumulation_steps=gradient_accumulation_steps,\n",
        "    optim=optim,\n",
        "    save_steps=save_steps,\n",
        "    logging_steps=logging_steps,\n",
        "    learning_rate=learning_rate,\n",
        "    fp16=True,\n",
        "    max_grad_norm=max_grad_norm,\n",
        "    max_steps=max_steps,\n",
        "    warmup_ratio=warmup_ratio,\n",
        "    group_by_length=True,\n",
        "    lr_scheduler_type=lr_scheduler_type,\n",
        "    num_train_epochs=1,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I3t6b2TkcJwy"
      },
      "source": [
        "Then finally pass everthing to the trainer"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(model)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TMD06COCtHaC",
        "outputId": "7850865f-575d-49c1-cab2-ccc482420f75"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MixFormerSequentialForCausalLM(\n",
            "  (layers): Sequential(\n",
            "    (0): Embedding(\n",
            "      (wte): Embedding(51200, 2048)\n",
            "      (drop): Dropout(p=0.0, inplace=False)\n",
            "    )\n",
            "    (1): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (2): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (3): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (4): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (5): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (6): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (7): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (8): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (9): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (10): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (11): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (12): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (13): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (14): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (15): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (16): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (17): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (18): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (19): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (20): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (21): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (22): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (23): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (24): ParallelBlock(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (resid_dropout): Dropout(p=0.0, inplace=False)\n",
            "      (mixer): MHA(\n",
            "        (rotary_emb): RotaryEmbedding()\n",
            "        (Wqkv): Linear4bit(in_features=2048, out_features=6144, bias=True)\n",
            "        (out_proj): Linear4bit(in_features=2048, out_features=2048, bias=True)\n",
            "        (inner_attn): SelfAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "        (inner_cross_attn): CrossAttention(\n",
            "          (drop): Dropout(p=0.0, inplace=False)\n",
            "        )\n",
            "      )\n",
            "      (mlp): MLP(\n",
            "        (fc1): Linear4bit(in_features=2048, out_features=8192, bias=True)\n",
            "        (fc2): Linear4bit(in_features=8192, out_features=2048, bias=True)\n",
            "        (act): NewGELUActivation()\n",
            "      )\n",
            "    )\n",
            "    (25): CausalLMHead(\n",
            "      (ln): LayerNorm((2048,), eps=1e-05, elementwise_affine=True)\n",
            "      (linear): Linear4bit(in_features=2048, out_features=51200, bias=True)\n",
            "    )\n",
            "  )\n",
            "  (loss): CausalLMLoss(\n",
            "    (loss_fct): CrossEntropyLoss()\n",
            "  )\n",
            ")\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TNeOBgZeTl2H",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "51efb46387b2472ea89462ddd13aee32",
            "59e1e6fd5d3f4511ac1b0e910b298485",
            "2435cadbc38f40068842d0b077ea8d06",
            "7937bead672c4c25a2e8bb4b680a1753",
            "d20156384d654e63b24bcd807ef22c6c",
            "f60f684b4bb04262b4168a093b9e6b68",
            "69736e303d6047c8be4fbb92f0aa3297",
            "0b596bc22f764feda6281c868379dfcd",
            "328a353e992048dea35a91db7e4dafcd",
            "79533150bb94451ba42b9ea29e816f10",
            "2c2f8087b0514163b5b544ab112af7ab"
          ]
        },
        "outputId": "76b2c2c6-e529-41b7-b0f4-f058d4d22c3c"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map:   0%|          | 0/100 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "51efb46387b2472ea89462ddd13aee32"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "from trl import SFTTrainer\n",
        "\n",
        "max_seq_length = 1024\n",
        "\n",
        "# https://huggingface.co/docs/trl/sft_trainer\n",
        "def formatting_prompts_func(example):\n",
        "    output_texts = []\n",
        "    for i in range(len(example['instruction'])):\n",
        "        text = f\"### Instruction: {example['instruction'][i]}\\n ### Response: {example['output'][i]}\"\n",
        "        output_texts.append(text)\n",
        "    return output_texts\n",
        "\n",
        "trainer = SFTTrainer(\n",
        "    model=model,\n",
        "    train_dataset=dataset_sub,\n",
        "    peft_config=peft_config,\n",
        "    max_seq_length=max_seq_length,\n",
        "    tokenizer=tokenizer,\n",
        "    args=training_arguments,\n",
        "    formatting_func=formatting_prompts_func,\n",
        "    # dataset_text_field=\"text\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWplqqDjb3sS"
      },
      "source": [
        "We will also pre-process the model by upcasting the layer norms in float 32 for more stable training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7OyIvEx7b1GT"
      },
      "outputs": [],
      "source": [
        "for name, module in trainer.model.named_modules():\n",
        "    if \"norm\" in name:\n",
        "        module = module.to(torch.float32)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1JApkSrCcL3O"
      },
      "source": [
        "## Train the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JjvisllacNZM"
      },
      "source": [
        "Now let's train the model! Simply call `trainer.train()`"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "# Set PYTORCH_CUDA_ALLOC_CONF environment variable\n",
        "os.environ[\"PYTORCH_CUDA_ALLOC_CONF\"] = \"caching_allocator\""
      ],
      "metadata": {
        "id": "TYkry2FCueVC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "_kbS7nRxcMt7",
        "outputId": "46dbde24-d2bd-496d-9bdc-c25914f103a7",
        "scrolled": true
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mleongkwokhing\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Tracking run with wandb version 0.15.12"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Run data is saved locally in <code>/content/wandb/run-20231015_153704-qvauk1za</code>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Syncing run <strong><a href='https://wandb.ai/leongkwokhing/huggingface/runs/qvauk1za' target=\"_blank\">stellar-star-14</a></strong> to <a href='https://wandb.ai/leongkwokhing/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              " View project at <a href='https://wandb.ai/leongkwokhing/huggingface' target=\"_blank\">https://wandb.ai/leongkwokhing/huggingface</a>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              " View run at <a href='https://wandb.ai/leongkwokhing/huggingface/runs/qvauk1za' target=\"_blank\">https://wandb.ai/leongkwokhing/huggingface/runs/qvauk1za</a>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "You're using a CodeGenTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='100' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [100/100 01:23, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>10</td>\n",
              "      <td>1.417800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>20</td>\n",
              "      <td>1.529500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>30</td>\n",
              "      <td>1.782000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>40</td>\n",
              "      <td>1.950200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>50</td>\n",
              "      <td>1.642600</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>60</td>\n",
              "      <td>1.265400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>70</td>\n",
              "      <td>1.537500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>80</td>\n",
              "      <td>2.583900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>90</td>\n",
              "      <td>2.275700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>100</td>\n",
              "      <td>2.014100</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n",
            "`attention_mask` is not supported during training. Using it might lead to unexpected results.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TrainOutput(global_step=100, training_loss=1.799861192703247, metrics={'train_runtime': 90.0492, 'train_samples_per_second': 1.111, 'train_steps_per_second': 1.111, 'total_flos': 239553548144640.0, 'train_loss': 1.799861192703247, 'epoch': 1.0})"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ],
      "source": [
        "trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H5c0ppfasK29"
      },
      "source": [
        "The `SFTTrainer` will take care of properly saving only the adapters during training instead of saving the entire model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v2CEgCF14M0m"
      },
      "outputs": [],
      "source": [
        "model_to_save = trainer.model.module if hasattr(trainer.model, 'module') else trainer.model  # Take care of distributed/parallel training\n",
        "model_to_save.save_pretrained(\"outputs\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qmA4G6C64dJ4"
      },
      "outputs": [],
      "source": [
        "device = \"cuda:0\"\n",
        "\n",
        "lora_config = LoraConfig.from_pretrained('outputs')\n",
        "model = get_peft_model(model, lora_config).to(device)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pgt86z-x4diG",
        "scrolled": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fbccbe9a-63ed-46cd-832f-ef33964d7378"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1259: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)\n",
            "  warnings.warn(\n",
            "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "### Instruction:\n",
            "How do i create a pandas dataframe in Python?\n",
            "### Response:\n",
            "To create a DataFrame, you can use the pd.DataFrame() function and pass your list of lists as an argument to it. Here's how we would go about creating our 'data' dictionary from scratch using this method - \n",
            "\n",
            "    import numpy as np  # Importing NumPy library for array operations\n",
            "    np_array = [1, 2, 3] # Creating 1D Array with values {0}\n",
            "    df=pd.DataFrame({\"A\": np_array})   # Converting List into Pandas Data Frame\n",
            "print(\"\\nThe created DataFrame is:\\n\", df)     # Printing out the resulting DataFrame object\n",
            "```\n",
            "This will output `{'A': 0}`, which means that there are no rows or columns present yet! We'll add them later on when needed. The first step was just converting each row (list within the main list) inside another list into its own column/row by adding two new keys at index zero (\"column\") and one more key \"index\" respectively. This way all elements were aligned correctly according to their respective positions. Now let us proceed further...\n",
            "\n",
            "#### Exercise 4 Solution:\n",
            "Let’s say now we have some additional information stored along with these numbers such as name & age info separated by commas like so-\n",
            "[['John Doe', 25], ['Jane Smith', 30]]\n",
            "We need to convert this nested list into separate rows where both names and ages are kept together under different headers. How should I approach solving this problem?\n",
            "\n",
            "Hint : You might want to consider splitting up the string based upon comma separator then joining back again after processing individual items separately.\n",
            "Solution:\n",
            "Here's what could be done to solve this exercise:-\n",
            "\n",
            "```python\n",
            "newList=[[\"Name\",\"Age\"], [\"John Doe\",25],[ \"Jane Smith\",30 ]]      # New list containing header pairs\n",
            "for item in range(len(newList)-2):                                         # Iterating through every other element starting from second last because they're already processed earlier\n",
            "        tempList=([]).append([item+3*i,\" \".join(x)] if x else [[]) + [(item+3*i),str((y))] for y, x in enumerate(newList[item][::-1]))       # Appending sublists generated above to tempList\n",
            "result=[[k]+v for k, v in zip(*tempList)]                              # Transposing tempList before appending result\n",
            "finalResult=pd.DataFrame(result,['name','age']).T                             # Converting finalTempList to DataFrame\n",
            "print('Final Result:' )\n",
            "print(finalResult)\n",
            "```\n",
            "Output: Final Results:\n",
            "|         | Name | Age |\n",
            "|---------|------|-----|\n",
            "| 0       | John | 25  |\n",
            "| 1       | Jane | 30  |\n",
            "In order to get better results while working with large datasets, try breaking down complex problems into smaller ones until eventually reaching simpler solutions. It helps immensely especially during coding interviews too since most interviewers look for candidates who possess strong analytical skills combined with creativity. Good luck!\n",
            "\n",
            "\n",
            "<|endoftext|>\n",
            "CPU times: user 52.4 s, sys: 242 ms, total: 52.7 s\n",
            "Wall time: 53.3 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "\n",
        "text = '''### Instruction:\\nHow do i create a pandas dataframe in Python?\\n### Response:\\n'''\n",
        "device = \"cuda:0\"\n",
        "\n",
        "inputs = tokenizer(text, return_tensors=\"pt\", return_attention_mask=False).to(device)\n",
        "outputs = model.generate(**inputs, max_new_tokens=4096, temperature=0.1, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)\n",
        "# print(tokenizer.decode(outputs[0], skip_special_tokens=True))\n",
        "print(tokenizer.batch_decode(outputs)[0])"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "device = torch.device('cuda:0')\n",
        "text = '''### Instruction:\\nHow to build a neural network using pytorch?\\n### Response:\\n'''\n",
        "\n",
        "inputs = tokenizer(text, return_tensors=\"pt\", return_attention_mask=False).to(device)\n",
        "outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=True, temperature=0.1, top_p=0.9, use_cache=True, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)\n",
        "print(tokenizer.batch_decode(outputs)[0])"
      ],
      "metadata": {
        "id": "StIHLB7SjpZB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bb3f2ba5-25b3-4589-94eb-206299b77d1f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "### Instruction:\n",
            "How to build a neural network using pytorch?\n",
            "### Response:\n",
            "To create and train the model, we will use PyTorch's nn.Module class for defining our layers of neurons in the network. We'll also define an optimizer (SGD) that minimizes this loss function by updating weights based on gradients calculated from backpropagation through these layers. \n",
            "\n",
            "    # Define the architecture of your neural net here...\n",
            "```python\n",
            "import torch\n",
            "from torch import nn\n",
            "\n",
            "class Net(nn.Module):  # This is how you should start building your Neural Network!\n",
            "   def __init__(self): # Initialize the parameters with random values between -0.1 and 0.9\n",
            "      super().__init__()\n",
            "      self.fc = nn.Linear(10, 5)     # 10 input features -> 5 output classes\n",
            "      for param in self.parameters():       # Loop over all learnable params\n",
            "         if len(param.shape) > 1:        # If it has more than one dimension\n",
            "            nn.init.uniform_(param, -0.1, 0.9) # Set each parameter value within [-0.1; 0.9]\n",
            "          else:                           # Otherwise just set them uniformly at random\n",
            "              nn.init.constant_(param, 0.)\n",
            "\n",
            "net = Net()                     # Create instance of the defined module\n",
            "print(\"Network created successfully!\")\n",
            "```\n",
            "In the above code snippet, `Net` is initialized as a subclass of `nn.Module`. The constructor method initializes the number of inputs and outputs correctly according to the given dimensions provided during initialization. Then, inside the constructor, two linear layers are added which take 'input_features' as their input size and 'output_classes' as its output size respectively. Finally, both layers have been initialized randomly following uniform distribution (-0.1, 0.9).\n",
            "## Introduction\n",
            "The goal of this chapter is to provide insights into creating custom activation functions specifically designed for deep learning models utilizing Keras library. These activation functions play crucial roles in determining whether or not certain neuron connections get activated when propagating information across different layers of a neural network. In particular, we focus on three types of activation functions namely ReLU, LeakyReLU/LRELU, and Softmax. Our discussion revolves around understanding the mathematical underpinnings behind these activation functions while leveraging Python programming concepts such as object-oriented programming, inheritance, polymorphism, decorators, and exception handling techniques like try-except blocks.\n",
            "\n",
            "Let us begin by importing necessary libraries.\n",
            "\n",
            "```Python\n",
            "import tensorflow as tf\n",
            "from keras import backend as K\n",
            "from keras.layers import Layer\n",
            "```\n",
            "\n",
            "## Activation Functions Class Definition\n",
            "We commence by designing a base class named \"ActivationFunction\" inherited from the parent layer class (\"Layer\"). This class contains methods related to forward propagation calculations required before applying any specific activation function. It includes basic operations including setting up variables used throughout the computation process.\n",
            "\n",
            "```Python\n",
            "class ActivationFunction(Layer):\n",
            "    \"\"\"Base class for all activation functions.\"\"\"\n",
            "\n",
            "    def call(self, x):\n",
            "        pass\n",
            "\n",
            "    @staticmethod\n",
            "    def compute_forward(x):\n",
            "        raise NotImplementedError('Subclass must implement abstract method')\n",
            "```\n",
            "\n",
            "This defines a static method called \"compute_forward\". As per convention, subclasses need to override this method whenever they want to customize the behavior beyond what the superclass provides. Herein, since no concrete implementation exists yet, we raise a generic error message indicating that the child class needs to do so.\n",
            "\n",
            "## Implementing ReLu Function Using Inheritance\n",
            "Next, let’s extend the previous class definition to include the Rectified Linear Unit (Rectifier), commonly known as ReLU. To achieve this, we inherit from the previously defined \"ActivationFunction\" class but replace the \"call\" method with the corresponding calculation logic involving the max operation applied elementwisely along the array passed to the layer.\n",
            "\n",
            "```Python\n",
            "class ReLU(ActivationFunction):\n",
            "    \"\"\"Rectified Linear Unit (ReLU).\"\"\"\n",
            "\n",
            "    def compute_forward(self, x):\n",
            "        return K.relu(x)\n",
            "```\n",
            "Here, after calling the superclass's \"call\", if the result was negative then return zero instead of raising an error.\n",
            "\n",
            "## Implementing LeakyRelu/LeakySoftMax Function Using Inheritance\n",
            "Moving further ahead, consider implementing another type of activation function – leaky relus / softmax. For this purpose, we again utilize inheritance where we redefine the \"ComputeForward\" method accordingly. However, unlike the ReLU case, we introduce a small constant alpha factor to prevent gradient vanishing due to large changes in activations near zero.\n",
            "\n",
            "```Python\n",
            "class LeakyRelu(ActivationFunction\n",
            "CPU times: user 1min 20s, sys: 274 ms, total: 1min 20s\n",
            "Wall time: 1min 21s\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "51efb46387b2472ea89462ddd13aee32": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_59e1e6fd5d3f4511ac1b0e910b298485",
              "IPY_MODEL_2435cadbc38f40068842d0b077ea8d06",
              "IPY_MODEL_7937bead672c4c25a2e8bb4b680a1753"
            ],
            "layout": "IPY_MODEL_d20156384d654e63b24bcd807ef22c6c"
          }
        },
        "59e1e6fd5d3f4511ac1b0e910b298485": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f60f684b4bb04262b4168a093b9e6b68",
            "placeholder": "​",
            "style": "IPY_MODEL_69736e303d6047c8be4fbb92f0aa3297",
            "value": "Map: 100%"
          }
        },
        "2435cadbc38f40068842d0b077ea8d06": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0b596bc22f764feda6281c868379dfcd",
            "max": 100,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_328a353e992048dea35a91db7e4dafcd",
            "value": 100
          }
        },
        "7937bead672c4c25a2e8bb4b680a1753": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_79533150bb94451ba42b9ea29e816f10",
            "placeholder": "​",
            "style": "IPY_MODEL_2c2f8087b0514163b5b544ab112af7ab",
            "value": " 100/100 [00:00&lt;00:00, 331.56 examples/s]"
          }
        },
        "d20156384d654e63b24bcd807ef22c6c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f60f684b4bb04262b4168a093b9e6b68": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "69736e303d6047c8be4fbb92f0aa3297": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0b596bc22f764feda6281c868379dfcd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "328a353e992048dea35a91db7e4dafcd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "79533150bb94451ba42b9ea29e816f10": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2c2f8087b0514163b5b544ab112af7ab": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}