3.Guanaco Chatbot Demo with LLaMA-7B Model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Exploring the Guanaco Chatbot Demo with LLaMA-7B Model\n",
        "\n",
        "![](https://image.lexica.art/full_jpg/3f6788a2-6772-4791-8342-5378c36ba155)\n",
        "\n",
        "we will dive into the code of a chatbot demo that utilizes the LLaMA-7B model for generating human-like responses. The chatbot, named Guanaco, is designed to interact with users, answer their queries, and provide insights using natural language generation. We will break down the code into several sections to understand its functionality and purpose."
      ],
      "metadata": {
        "id": "k5zwd0Ey697X"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction\n",
        "The code presented here is a modified version of a Jupyter Notebook available on GitHub. It showcases the implementation of a chatbot using the LLaMA-7B language model and is integrated into a graphical user interface (GUI) using the Gradio library. The chatbot's responses are generated by predicting the next words based on the input conversation history."
      ],
      "metadata": {
        "id": "sjB8Athu7K_c"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2oFDunic55pn",
        "outputId": "44a9dd78-2fe0-467b-8036-6a2d46b89136"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.6/92.6 MB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m107.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m56.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for transformers (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m251.2/251.2 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for peft (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for accelerate (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting gradio\n",
            "  Downloading gradio-3.41.1-py3-none-any.whl (20.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.1/20.1 MB\u001b[0m \u001b[31m72.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting aiofiles<24.0,>=22.0 (from gradio)\n",
            "  Downloading aiofiles-23.2.1-py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (4.2.2)\n",
            "Collecting fastapi (from gradio)\n",
            "  Downloading fastapi-0.101.1-py3-none-any.whl (65 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.8/65.8 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ffmpy (from gradio)\n",
            "  Downloading ffmpy-0.3.1.tar.gz (5.5 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting gradio-client==0.5.0 (from gradio)\n",
            "  Downloading gradio_client-0.5.0-py3-none-any.whl (298 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m298.2/298.2 kB\u001b[0m \u001b[31m37.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpx (from gradio)\n",
            "  Downloading httpx-0.24.1-py3-none-any.whl (75 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.4/75.4 kB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: huggingface-hub>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (0.16.4)\n",
            "Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio) (6.0.1)\n",
            "Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (3.1.2)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (2.1.3)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (3.7.1)\n",
            "Requirement already satisfied: numpy~=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (1.23.5)\n",
            "Collecting orjson~=3.0 (from gradio)\n",
            "  Downloading orjson-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (139 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m139.9/139.9 kB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio) (23.1)\n",
            "Requirement already satisfied: pandas<3.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (1.5.3)\n",
            "Requirement already satisfied: pillow<11.0,>=8.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (9.4.0)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from gradio) (2.2.0)\n",
            "Collecting pydub (from gradio)\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Collecting python-multipart (from gradio)\n",
            "  Downloading python_multipart-0.0.6-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.7/45.7 kB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml<7.0,>=5.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (6.0.1)\n",
            "Requirement already satisfied: requests~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (2.31.0)\n",
            "Collecting semantic-version~=2.0 (from gradio)\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (4.7.1)\n",
            "Collecting uvicorn>=0.14.0 (from gradio)\n",
            "  Downloading uvicorn-0.23.2-py3-none-any.whl (59 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.5/59.5 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting websockets<12.0,>=10.0 (from gradio)\n",
            "  Downloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m16.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.5.0->gradio) (2023.6.0)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio) (4.19.0)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio) (0.12.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio) (3.12.2)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio) (4.66.1)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (1.1.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (0.11.0)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (4.42.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (1.4.4)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (3.1.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas<3.0,>=1.0->gradio) (2023.3)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4->gradio) (0.5.0)\n",
            "Requirement already satisfied: pydantic-core==2.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4->gradio) (2.6.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (3.2.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (2.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (2023.7.22)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio) (8.1.7)\n",
            "Collecting h11>=0.8 (from uvicorn>=0.14.0->gradio)\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting starlette<0.28.0,>=0.27.0 (from fastapi->gradio)\n",
            "  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.0/67.0 kB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpcore<0.18.0,>=0.15.0 (from httpx->gradio)\n",
            "  Downloading httpcore-0.17.3-py3-none-any.whl (74 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m74.5/74.5 kB\u001b[0m \u001b[31m11.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio) (1.3.0)\n",
            "Requirement already satisfied: anyio<5.0,>=3.0 in /usr/local/lib/python3.10/dist-packages (from httpcore<0.18.0,>=0.15.0->httpx->gradio) (3.7.1)\n",
            "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (23.1.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (2023.7.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.30.2)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.9.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio) (1.16.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5.0,>=3.0->httpcore<0.18.0,>=0.15.0->httpx->gradio) (1.1.3)\n",
            "Building wheels for collected packages: ffmpy\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.1-py3-none-any.whl size=5579 sha256=baf0e0fc40c2519812aeaa3b472b6f1f9c250ec0ce39ba364a66ff1515e517db\n",
            "  Stored in directory: /root/.cache/pip/wheels/01/a6/d1/1c0828c304a4283b2c1639a09ad86f83d7c487ef34c6b4a1bf\n",
            "Successfully built ffmpy\n",
            "Installing collected packages: pydub, ffmpy, websockets, semantic-version, python-multipart, orjson, h11, aiofiles, uvicorn, starlette, httpcore, httpx, fastapi, gradio-client, gradio\n",
            "Successfully installed aiofiles-23.2.1 fastapi-0.101.1 ffmpy-0.3.1 gradio-3.41.1 gradio-client-0.5.0 h11-0.14.0 httpcore-0.17.3 httpx-0.24.1 orjson-3.9.5 pydub-0.25.1 python-multipart-0.0.6 semantic-version-2.10.0 starlette-0.27.0 uvicorn-0.23.2 websockets-11.0.3\n",
            "Collecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: sentencepiece\n",
            "Successfully installed sentencepiece-0.1.99\n"
          ]
        }
      ],
      "source": [
        "# https://github.com/artidoro/qlora/blob/main/examples/guanaco_7B_demo_colab.ipynb modified\n",
        "\n",
        "# Install latest bitsandbytes & transformers, accelerate from source\n",
        "!pip install -q -U bitsandbytes\n",
        "!pip install -q -U git+https://github.com/huggingface/transformers.git\n",
        "!pip install -q -U git+https://github.com/huggingface/peft.git\n",
        "!pip install -q -U git+https://github.com/huggingface/accelerate.git\n",
        "# Other requirements for the demo\n",
        "!pip install gradio\n",
        "!pip install sentencepiece"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Installation and Setup\n",
        "The initial portion of the code involves installing various Python packages required for running the demo. These packages include bitsandbytes, transformers, accelerate, gradio, and sentencepiece. These libraries provide the necessary tools for model loading, text generation, and creating the graphical interface for user interaction."
      ],
      "metadata": {
        "id": "xzbrBDIM9Fl8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the model.\n",
        "# Note: It can take a while to download LLaMA and add the adapter modules.\n",
        "# You can also use the 13B model by loading in 4bits.\n",
        "\n",
        "import torch\n",
        "from peft import PeftModel\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer"
      ],
      "metadata": {
        "id": "7QfvjO_f7XDo"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model Loading\n",
        "\n",
        "After installing the required packages, the code proceeds to load the LLaMA-7B language model and adapter modules. The `AutoModelForCausalLM` class from the `transformers` library is used to load the pre-trained model. Additionally, the `PeftModel` class from the `peft` library is utilized to incorporate adapter modules into the model. Adapter modules allow fine-tuning of pre-trained models for specific tasks without affecting the original model parameters."
      ],
      "metadata": {
        "id": "nRpCL8Dm9N3N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_name = \"decapoda-research/llama-7b-hf\"\n",
        "adapters_name = 'timdettmers/guanaco-7b'"
      ],
      "metadata": {
        "id": "iFnk_KG57gqC"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "torch.cuda.empty_cache()\n",
        "print(f\"Starting to load the model {model_name} into memory\")\n",
        "\n",
        "m = AutoModelForCausalLM.from_pretrained(\n",
        "    model_name,\n",
        "    # load_in_4bit=True,\n",
        "    torch_dtype=torch.bfloat16,\n",
        "    device_map={\"\": 0}\n",
        ")\n",
        "torch.cuda.empty_cache()\n",
        "m = PeftModel.from_pretrained(m, adapters_name)\n",
        "torch.cuda.empty_cache()\n",
        "m = m.merge_and_unload()\n",
        "tok = LlamaTokenizer.from_pretrained(model_name)\n",
        "tok.bos_token_id = 1\n",
        "\n",
        "stop_token_ids = [0]\n",
        "\n",
        "print(f\"Successfully loaded the model {model_name} into memory\")\n",
        "torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "EPNUoJ4q7liu"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Chatbot Setup\n",
        "\n",
        "The code defines a series of functions and configurations to set up the chatbot's behavior within the Gradio interface. These functions handle message processing, conversation history management, and text generation using the loaded model.\n",
        "\n",
        "- The `convert_history_to_text` function converts the conversation history into a formatted text that includes both user and assistant messages.\n",
        "- The `user` function appends the user's message to the conversation history.\n",
        "- The `bot` function uses the loaded model to generate responses. It tokenizes the conversation history, sets up text generation parameters such as temperature and top-k sampling, and iteratively generates tokens while adhering to stopping criteria. The generated text is appended to the assistant's message in the history."
      ],
      "metadata": {
        "id": "dXXgTSOf9XKL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Setup the gradio Demo.\n",
        "import datetime\n",
        "import os\n",
        "from threading import Event, Thread\n",
        "from uuid import uuid4\n",
        "\n",
        "import gradio as gr\n",
        "import requests"
      ],
      "metadata": {
        "id": "Oww--XXh7q9a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "max_new_tokens = 1536\n",
        "start_message = \"\"\"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\"\"\""
      ],
      "metadata": {
        "id": "HSANsfGZ7twv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Gradio Interface\n",
        "\n",
        "The Gradio interface is used to create a user-friendly GUI for interacting with the chatbot. Users can input messages, adjust advanced options such as temperature and sampling techniques, and view the chatbot's responses in real-time.\n",
        "\n",
        "- The GUI displays a chat message box where users can type their messages.\n",
        "- The \"Submit\" button triggers user input processing and chatbot response generation.\n",
        "- Advanced options like temperature, top-p sampling, top-k sampling, and repetition penalty can be adjusted using sliders.\n",
        "- A disclaimer is included to highlight that the model's outputs may not always be factually accurate.\n",
        "- A privacy policy link is provided for user reference."
      ],
      "metadata": {
        "id": "c-wCLir99eDF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class StopOnTokens(StoppingCriteria):\n",
        "    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:\n",
        "        for stop_id in stop_token_ids:\n",
        "            if input_ids[0][-1] == stop_id:\n",
        "                return True\n",
        "        return False\n",
        "\n",
        "\n",
        "def convert_history_to_text(history):\n",
        "    text = start_message + \"\".join(\n",
        "        [\n",
        "            \"\".join(\n",
        "                [\n",
        "                    f\"### Human: {item[0]}\\n\",\n",
        "                    f\"### Assistant: {item[1]}\\n\",\n",
        "                ]\n",
        "            )\n",
        "            for item in history[:-1]\n",
        "        ]\n",
        "    )\n",
        "    text += \"\".join(\n",
        "        [\n",
        "            \"\".join(\n",
        "                [\n",
        "                    f\"### Human: {history[-1][0]}\\n\",\n",
        "                    f\"### Assistant: {history[-1][1]}\\n\",\n",
        "                ]\n",
        "            )\n",
        "        ]\n",
        "    )\n",
        "    return text\n",
        "\n",
        "def log_conversation(conversation_id, history, messages, generate_kwargs):\n",
        "    logging_url = os.getenv(\"LOGGING_URL\", None)\n",
        "    if logging_url is None:\n",
        "        return\n",
        "\n",
        "    timestamp = datetime.datetime.now().strftime(\"%Y-%m-%dT%H:%M:%S\")\n",
        "\n",
        "    data = {\n",
        "        \"conversation_id\": conversation_id,\n",
        "        \"timestamp\": timestamp,\n",
        "        \"history\": history,\n",
        "        \"messages\": messages,\n",
        "        \"generate_kwargs\": generate_kwargs,\n",
        "    }\n",
        "\n",
        "    try:\n",
        "        requests.post(logging_url, json=data)\n",
        "    except requests.exceptions.RequestException as e:\n",
        "        print(f\"Error logging conversation: {e}\")\n",
        "\n",
        "def user(message, history):\n",
        "    # Append the user's message to the conversation history\n",
        "    return \"\", history + [[message, \"\"]]\n",
        "\n",
        "def bot(history, temperature, top_p, top_k, repetition_penalty, conversation_id):\n",
        "    print(f\"history: {history}\")\n",
        "    # Initialize a StopOnTokens object\n",
        "    stop = StopOnTokens()\n",
        "\n",
        "    # Construct the input message string for the model by concatenating the current system message and conversation history\n",
        "    messages = convert_history_to_text(history)\n",
        "\n",
        "    # Tokenize the messages string\n",
        "    input_ids = tok(messages, return_tensors=\"pt\").input_ids\n",
        "    input_ids = input_ids.to(m.device)\n",
        "    streamer = TextIteratorStreamer(tok, timeout=10.0, skip_prompt=True, skip_special_tokens=True)\n",
        "    generate_kwargs = dict(\n",
        "        input_ids=input_ids,\n",
        "        max_new_tokens=max_new_tokens,\n",
        "        temperature=temperature,\n",
        "        do_sample=temperature > 0.0,\n",
        "        top_p=top_p,\n",
        "        top_k=top_k,\n",
        "        repetition_penalty=repetition_penalty,\n",
        "        streamer=streamer,\n",
        "        stopping_criteria=StoppingCriteriaList([stop]),\n",
        "    )\n",
        "\n",
        "    stream_complete = Event()\n",
        "\n",
        "    def generate_and_signal_complete():\n",
        "        m.generate(**generate_kwargs)\n",
        "        stream_complete.set()\n",
        "\n",
        "    def log_after_stream_complete():\n",
        "        stream_complete.wait()\n",
        "        log_conversation(\n",
        "            conversation_id,\n",
        "            history,\n",
        "            messages,\n",
        "            {\n",
        "                \"top_k\": top_k,\n",
        "                \"top_p\": top_p,\n",
        "                \"temperature\": temperature,\n",
        "                \"repetition_penalty\": repetition_penalty,\n",
        "            },\n",
        "        )\n",
        "\n",
        "    t1 = Thread(target=generate_and_signal_complete)\n",
        "    t1.start()\n",
        "\n",
        "    t2 = Thread(target=log_after_stream_complete)\n",
        "    t2.start()\n",
        "\n",
        "    # Initialize an empty string to store the generated text\n",
        "    partial_text = \"\"\n",
        "    for new_text in streamer:\n",
        "        partial_text += new_text\n",
        "        history[-1][1] = partial_text\n",
        "        yield history\n",
        "\n",
        "\n",
        "def get_uuid():\n",
        "    return str(uuid4())"
      ],
      "metadata": {
        "id": "fGNIp_QQ70Cv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with gr.Blocks(\n",
        "    theme=gr.themes.Soft(),\n",
        "    css=\".disclaimer {font-variant-caps: all-small-caps;}\",\n",
        ") as demo:\n",
        "    conversation_id = gr.State(get_uuid)\n",
        "    gr.Markdown(\n",
        "        \"\"\"Guanaco Demo\n",
        "\"\"\"\n",
        "    )\n",
        "    chatbot = gr.Chatbot().style(height=500)\n",
        "    with gr.Row():\n",
        "        with gr.Column():\n",
        "            msg = gr.Textbox(\n",
        "                label=\"Chat Message Box\",\n",
        "                placeholder=\"Chat Message Box\",\n",
        "                show_label=False,\n",
        "            ).style(container=False)\n",
        "        with gr.Column():\n",
        "            with gr.Row():\n",
        "                submit = gr.Button(\"Submit\")\n",
        "                stop = gr.Button(\"Stop\")\n",
        "                clear = gr.Button(\"Clear\")\n",
        "    with gr.Row():\n",
        "        with gr.Accordion(\"Advanced Options:\", open=False):\n",
        "            with gr.Row():\n",
        "                with gr.Column():\n",
        "                    with gr.Row():\n",
        "                        temperature = gr.Slider(\n",
        "                            label=\"Temperature\",\n",
        "                            value=0.7,\n",
        "                            minimum=0.0,\n",
        "                            maximum=1.0,\n",
        "                            step=0.1,\n",
        "                            interactive=True,\n",
        "                            info=\"Higher values produce more diverse outputs\",\n",
        "                        )\n",
        "                with gr.Column():\n",
        "                    with gr.Row():\n",
        "                        top_p = gr.Slider(\n",
        "                            label=\"Top-p (nucleus sampling)\",\n",
        "                            value=0.9,\n",
        "                            minimum=0.0,\n",
        "                            maximum=1,\n",
        "                            step=0.01,\n",
        "                            interactive=True,\n",
        "                            info=(\n",
        "                                \"Sample from the smallest possible set of tokens whose cumulative probability \"\n",
        "                                \"exceeds top_p. Set to 1 to disable and sample from all tokens.\"\n",
        "                            ),\n",
        "                        )\n",
        "                with gr.Column():\n",
        "                    with gr.Row():\n",
        "                        top_k = gr.Slider(\n",
        "                            label=\"Top-k\",\n",
        "                            value=0,\n",
        "                            minimum=0.0,\n",
        "                            maximum=200,\n",
        "                            step=1,\n",
        "                            interactive=True,\n",
        "                            info=\"Sample from a shortlist of top-k tokens — 0 to disable and sample from all tokens.\",\n",
        "                        )\n",
        "                with gr.Column():\n",
        "                    with gr.Row():\n",
        "                        repetition_penalty = gr.Slider(\n",
        "                            label=\"Repetition Penalty\",\n",
        "                            value=1.0,\n",
        "                            minimum=1.0,\n",
        "                            maximum=2.0,\n",
        "                            step=0.1,\n",
        "                            interactive=True,\n",
        "                            info=\"Penalize repetition — 1.0 to disable.\",\n",
        "                        )\n",
        "    with gr.Row():\n",
        "        gr.Markdown(\n",
        "            \"Disclaimer: The model can produce factually incorrect output, and should not be relied on to produce \"\n",
        "            \"factually accurate information. The model was trained on various public datasets; while great efforts \"\n",
        "            \"have been taken to clean the pretraining data, it is possible that this model could generate lewd, \"\n",
        "            \"biased, or otherwise offensive outputs.\",\n",
        "            elem_classes=[\"disclaimer\"],\n",
        "        )\n",
        "    with gr.Row():\n",
        "        gr.Markdown(\n",
        "            \"[Privacy policy](https://gist.github.com/samhavens/c29c68cdcd420a9aa0202d0839876dac)\",\n",
        "            elem_classes=[\"disclaimer\"],\n",
        "        )\n",
        "\n",
        "    submit_event = msg.submit(\n",
        "        fn=user,\n",
        "        inputs=[msg, chatbot],\n",
        "        outputs=[msg, chatbot],\n",
        "        queue=False,\n",
        "    ).then(\n",
        "        fn=bot,\n",
        "        inputs=[\n",
        "            chatbot,\n",
        "            temperature,\n",
        "            top_p,\n",
        "            top_k,\n",
        "            repetition_penalty,\n",
        "            conversation_id,\n",
        "        ],\n",
        "        outputs=chatbot,\n",
        "        queue=True,\n",
        "    )\n",
        "    submit_click_event = submit.click(\n",
        "        fn=user,\n",
        "        inputs=[msg, chatbot],\n",
        "        outputs=[msg, chatbot],\n",
        "        queue=False,\n",
        "    ).then(\n",
        "        fn=bot,\n",
        "        inputs=[\n",
        "            chatbot,\n",
        "            temperature,\n",
        "            top_p,\n",
        "            top_k,\n",
        "            repetition_penalty,\n",
        "            conversation_id,\n",
        "        ],\n",
        "        outputs=chatbot,\n",
        "        queue=True,\n",
        "    )\n",
        "    stop.click(\n",
        "        fn=None,\n",
        "        inputs=None,\n",
        "        outputs=None,\n",
        "        cancels=[submit_event, submit_click_event],\n",
        "        queue=False,\n",
        "    )\n",
        "    clear.click(lambda: None, None, chatbot, queue=False)\n",
        "\n",
        "demo.queue(max_size=128, concurrency_count=2)"
      ],
      "metadata": {
        "id": "ofQ671Nc8EZ8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Conclusion\n",
        "\n",
        "In conclusion, the provided code demonstrates the implementation of a chatbot named Guanaco, powered by the LLaMA-7B language model. The chatbot interacts with users through a user-friendly Gradio interface, generating responses based on input conversation history. By breaking down the code into various sections, we've explored how the model is loaded, the chatbot's behavior is defined, and the Gradio interface is set up for user interaction.\n",
        "\n",
        "It's important to note that this demo showcases the capabilities of the LLaMA-7B model and its interaction with users. However, as with any AI-generated content, users should be aware that the model's responses may not always be entirely accurate or appropriate, and they should exercise caution when using the chatbot for factual information or important decisions."
      ],
      "metadata": {
        "id": "mdxKTYGU9lFr"
      }
    }
  ]
}