gpt_llm_trainer_v2,_with_GPT_3_5_Fine_Tuning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mshumer/gpt-llm-trainer/blob/main/gpt_llm_trainer_v2%2C_with_GPT_3_5_Fine_Tuning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Describe your model -> fine-tuned GPT-3.5\n",
        "By Matt Shumer (https://twitter.com/mattshumer_)\n",
        "\n",
        "The goal of this notebook is to experiment with a new way to make it very easy to build a task-specific model for your use-case.\n",
        "\n",
        "First, use the best GPU available (go to Runtime -> change runtime type)\n",
        "\n",
        "To create your model, just go to the first code cell, and describe the model you want to build in the prompt. Be descriptive and clear.\n",
        "\n",
        "Select a temperature (high=creative, low=precise), and the number of training examples to generate to train the model. From there, just run all the cells.\n",
        "\n",
        "You can change the model you want to fine-tune by changing `model_name` in the `Define Hyperparameters` cell."
      ],
      "metadata": {
        "id": "wM8MRkf8Dr94"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Data generation step"
      ],
      "metadata": {
        "id": "Way3_PuPpIuE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Write your prompt here. Make it as descriptive as possible!\n",
        "\n",
        "Then, choose the temperature (between 0 and 1) to use when generating data. Lower values are great for precise tasks, like writing code, whereas larger values are better for creative tasks, like writing stories.\n",
        "\n",
        "Finally, choose how many examples you want to generate. The more you generate, a) the longer it takes and b) the more expensive data generation will be. But generally, more examples will lead to a higher-quality model. 100 is usually the minimum to start."
      ],
      "metadata": {
        "id": "lY-3DvlIpVSl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prompt = \"A model that takes in a puzzle-like reasoning-heavy question in English, and responds with a well-reasoned, step-by-step thought out response in Spanish.\"\n",
        "temperature = .4\n",
        "number_of_examples = 50"
      ],
      "metadata": {
        "id": "R7WKZyxtpUPS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run this to generate the dataset."
      ],
      "metadata": {
        "id": "1snNou5PrIci"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install openai tenacity"
      ],
      "metadata": {
        "id": "zuL2UaqlsmBD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import openai\n",
        "import random\n",
        "from tenacity import retry, stop_after_attempt, wait_exponential\n",
        "\n",
        "openai.api_key = \"YOUR API KEY HERE\"\n",
        "\n",
        "N_RETRIES = 3\n",
        "\n",
        "@retry(stop=stop_after_attempt(N_RETRIES), wait=wait_exponential(multiplier=1, min=4, max=70))\n",
        "def generate_example(prompt, prev_examples, temperature=.5):\n",
        "    messages=[\n",
        "        {\n",
        "            \"role\": \"system\",\n",
        "            \"content\": f\"You are generating data which will be used to train a machine learning model.\\n\\nYou will be given a high-level description of the model we want to train, and from that, you will generate data samples, each with a prompt/response pair.\\n\\nYou will do so in this format:\\n```\\nprompt\\n-----------\\n$prompt_goes_here\\n-----------\\n\\nresponse\\n-----------\\n$response_goes_here\\n-----------\\n```\\n\\nOnly one prompt/response pair should be generated per turn.\\n\\nFor each turn, make the example slightly more complex than the last, while ensuring diversity.\\n\\nMake sure your samples are unique and diverse, yet high-quality and complex enough to train a well-performing model.\\n\\nHere is the type of model we want to train:\\n`{prompt}`\"\n",
        "        }\n",
        "    ]\n",
        "\n",
        "    if len(prev_examples) > 0:\n",
        "        if len(prev_examples) > 8:\n",
        "            prev_examples = random.sample(prev_examples, 8)\n",
        "        for example in prev_examples:\n",
        "            messages.append({\n",
        "                \"role\": \"assistant\",\n",
        "                \"content\": example\n",
        "            })\n",
        "\n",
        "    response = openai.ChatCompletion.create(\n",
        "        model=\"gpt-4\",\n",
        "        messages=messages,\n",
        "        temperature=temperature,\n",
        "        max_tokens=1000,\n",
        "    )\n",
        "\n",
        "    return response.choices[0].message['content']\n",
        "\n",
        "# Generate examples\n",
        "prev_examples = []\n",
        "for i in range(number_of_examples):\n",
        "    print(f'Generating example {i}')\n",
        "    example = generate_example(prompt, prev_examples, temperature)\n",
        "    prev_examples.append(example)\n",
        "\n",
        "print(prev_examples)"
      ],
      "metadata": {
        "id": "Rdsd82ngpHCG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We also need to generate a system message."
      ],
      "metadata": {
        "id": "KC6iJzXjugJ-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def generate_system_message(prompt):\n",
        "\n",
        "    response = openai.ChatCompletion.create(\n",
        "        model=\"gpt-4\",\n",
        "        messages=[\n",
        "          {\n",
        "            \"role\": \"system\",\n",
        "            \"content\": \"You will be given a high-level description of the model we are training, and from that, you will generate a simple system prompt for that model to use. Remember, you are not generating the system message for data generation -- you are generating the system message to use for inference. A good format to follow is `Given $INPUT_DATA, you will $WHAT_THE_MODEL_SHOULD_DO.`.\\n\\nMake it as concise as possible. Include nothing but the system prompt in your response.\\n\\nFor example, never write: `\\\"$SYSTEM_PROMPT_HERE\\\"`.\\n\\nIt should be like: `$SYSTEM_PROMPT_HERE`.\"\n",
        "          },\n",
        "          {\n",
        "              \"role\": \"user\",\n",
        "              \"content\": prompt.strip(),\n",
        "          }\n",
        "        ],\n",
        "        temperature=temperature,\n",
        "        max_tokens=500,\n",
        "    )\n",
        "\n",
        "    return response.choices[0].message['content']\n",
        "\n",
        "system_message = generate_system_message(prompt)\n",
        "\n",
        "print(f'The system message is: `{system_message}`. Feel free to re-run this cell if you want a better result.')"
      ],
      "metadata": {
        "id": "xMcfhW6Guh2E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's put our examples into a dataframe and turn them into a final pair of datasets."
      ],
      "metadata": {
        "id": "G6BqZ-hjseBF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "import pandas as pd\n",
        "\n",
        "# Initialize lists to store prompts and responses\n",
        "prompts = []\n",
        "responses = []\n",
        "\n",
        "# Parse out prompts and responses from examples\n",
        "for example in prev_examples:\n",
        "  try:\n",
        "    split_example = example.split('-----------')\n",
        "    prompts.append(split_example[1].strip())\n",
        "    responses.append(split_example[3].strip())\n",
        "  except:\n",
        "    pass\n",
        "\n",
        "# Create a DataFrame\n",
        "df = pd.DataFrame({\n",
        "    'prompt': prompts,\n",
        "    'response': responses\n",
        "})\n",
        "\n",
        "# Remove duplicates\n",
        "df = df.drop_duplicates()\n",
        "\n",
        "print('There are ' + str(len(df)) + ' successfully-generated examples.')\n",
        "\n",
        "# Initialize list to store training examples\n",
        "training_examples = []\n",
        "\n",
        "# Create training examples in the format required for GPT-3.5 fine-tuning\n",
        "for index, row in df.iterrows():\n",
        "    training_example = {\n",
        "        \"messages\": [\n",
        "            {\"role\": \"system\", \"content\": system_message.strip()},\n",
        "            {\"role\": \"user\", \"content\": row['prompt']},\n",
        "            {\"role\": \"assistant\", \"content\": row['response']}\n",
        "        ]\n",
        "    }\n",
        "    training_examples.append(training_example)\n",
        "\n",
        "# Save training examples to a .jsonl file\n",
        "with open('training_examples.jsonl', 'w') as f:\n",
        "    for example in training_examples:\n",
        "        f.write(json.dumps(example) + '\\n')"
      ],
      "metadata": {
        "id": "7CEdkYeRsdmB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Upload the file to OpenAI"
      ],
      "metadata": {
        "id": "KWTY6qVgXD_T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "file_id = openai.File.create(\n",
        "  file=open(\"/content/training_examples.jsonl\", \"rb\"),\n",
        "  purpose='fine-tune'\n",
        ").id"
      ],
      "metadata": {
        "id": "4LjEUrI9XDgT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Train the model! You may need to wait a few minutes before running the next cell to allow for the file to process on OpenAI's servers."
      ],
      "metadata": {
        "id": "HmYRIq8dW9IR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "job = openai.FineTuningJob.create(training_file=file_id, model=\"gpt-3.5-turbo\")\n",
        "\n",
        "job_id = job.id"
      ],
      "metadata": {
        "id": "rdEyXmkoW80I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Now, just wait until the fine-tuning run is done, and you'll have a ready-to-use model!\n",
        "\n",
        "Run this cell every 20 minutes or so -- eventually, you'll see a message \"New fine-tuned model created: ft:gpt-3.5-turbo-0613:xxxxxxxxxxxx\"\n",
        "\n",
        "Once you see that message, you can go to the OpenAI Playground (or keep going to the next cells and use the API) to try the model!"
      ],
      "metadata": {
        "id": "XUSX5QzmZMTd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "openai.FineTuningJob.list_events(id=job_id, limit=10)"
      ],
      "metadata": {
        "id": "45DJZ7hHaBx0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Once your model is trained, run the next cell to grab the fine-tuned model name."
      ],
      "metadata": {
        "id": "91ihW2O27Phl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_name_pre_object = openai.FineTuningJob.retrieve(job_id)\n",
        "model_name = model_name_pre_object.fine_tuned_model\n",
        "print(model_name)"
      ],
      "metadata": {
        "id": "eWBRBPh8aEzH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Let's try it out!"
      ],
      "metadata": {
        "id": "2OmZLoBX7oQM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "response = openai.ChatCompletion.create(\n",
        "    model=model_name,\n",
        "    messages=[\n",
        "      {\n",
        "        \"role\": \"system\",\n",
        "        \"content\": system_message,\n",
        "      },\n",
        "      {\n",
        "          \"role\": \"user\",\n",
        "          \"content\": df['prompt'].sample().values[0],\n",
        "      }\n",
        "    ],\n",
        ")\n",
        "\n",
        "response.choices[0].message['content']"
      ],
      "metadata": {
        "id": "uxbrmzc5dMuC"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNN+zp3xSlZKggNbwTiLMxu",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}