gpt_prompt_engineer_Classification_Version.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mshumer/gpt-prompt-engineer/blob/main/gpt_prompt_engineer_Classification_Version.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "L0Ey7JZ5iLo1"
      },
      "source": [
        "# gpt-prompt-engineer -- Classification Version\n",
        "By Matt Shumer (https://twitter.com/mattshumer_)\n",
        "\n",
        "Github repo: https://github.com/mshumer/gpt-prompt-engineer\n",
        "\n",
        "Generate an optimal prompt for a given classification task that can be evaluated with 'true'/'false' outputs.\n",
        "\n",
        "You just need to describe the task clearly, and provide some test cases (for example, if we're classifying statements as 'happy' or not, a 'true' test case could be \"I had a great day!\", and a 'false' test case could be \"I am feeling gloomy.\").\n",
        "\n",
        "To generate a prompt:\n",
        "1. In the first cell, add in your OpenAI key.\n",
        "2. If you don't have GPT-4 access, change `CANDIDATE_MODEL='gpt-4'` in the second cell to `CANDIDATE_MODEL='gpt-3.5-turbo'`. If you do have access, skip this step.\n",
        "2. In the last cell, fill in the description of your task, as many test cases as you want (test cases are example prompts and their expected output), and the number of prompts to generate.\n",
        "3. Run all the cells! The AI will generate a number of candidate prompts, and test them all to find the best one!\n",
        "\n",
        "🪄🐝 To use [Weights & Biases logging](https://wandb.ai/site/prompts) to your LLM configs and the generated prompt outputs, just set `use_wandb = True`.\n",
        "\n",
        "🪄🔮 To use [Portkey](https://docs.portke.ai) for logging and tracing prompt chains and responses, just set `use_portkey = True`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install openai prettytable tqdm tenacity wandb -qq"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "UW3ztLRsolnk"
      },
      "outputs": [],
      "source": [
        "from prettytable import PrettyTable\n",
        "import time\n",
        "import wandb\n",
        "import openai\n",
        "from tenacity import retry, stop_after_attempt, wait_exponential\n",
        "\n",
        "openai.api_key = \"ADD YOUR KEY HERE\" # enter your OpenAI API key here\n",
        "\n",
        "use_wandb = True # set to True if you want to use wandb to log your config and results\n",
        "\n",
        "use_portkey = False #set to True if you want to use Portkey to log all the prompt chains and their responses Check https://portkey.ai/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {},
      "outputs": [],
      "source": [
        "candidate_gen_system_prompt = \"\"\"Your job is to generate system prompts for GPT-4, given a description of the use-case and some test cases.\n",
        "\n",
        "The prompts you will be generating will be for classifiers, with 'true' and 'false' being the only possible outputs.\n",
        "\n",
        "In your generated prompt, you should describe how the AI should behave in plain English. Include what it will see, and what it's allowed to output. Be creative in with prompts to get the best possible results. The AI knows it's an AI -- you don't need to tell it this.\n",
        "\n",
        "You will be graded based on the performance of your prompt... but don't cheat! You cannot include specifics about the test cases in your prompt. Any prompts with examples will be disqualified.\n",
        "\n",
        "Most importantly, output NOTHING but the prompt. Do not include anything else in your message.\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [],
      "source": [
        "CANDIDATE_MODEL = 'gpt-4'\n",
        "CANDIDATE_MODEL_TEMPERATURE = 0.9\n",
        "\n",
        "EVAL_MODEL = 'gpt-3.5-turbo'\n",
        "EVAL_MODEL_TEMPERATURE = 0\n",
        "EVAL_MODEL_MAX_TOKENS = 1\n",
        "\n",
        "NUMBER_OF_PROMPTS = 10 # this determines how many candidate prompts to generate... the higher, the more expensive\n",
        "\n",
        "N_RETRIES = 3  # number of times to retry a call to the ranking model if it fails\n",
        "\n",
        "WANDB_PROJECT_NAME = \"gpt-prompt-eng\" # used if use_wandb is True, Weights &| Biases project name\n",
        "WANDB_RUN_NAME = None # used if use_wandb is True, optionally set the Weights & Biases run name to identify this run\n",
        "\n",
        "PORTKEY_API = \"\" # used if use_portkey is True. Get api key here: https://app.portkey.ai/ (click on profile photo on top left)\n",
        "PORTKEY_TRACE = \"prompt_engineer_classification_test_run\" # used if use_portkey is True. Trace each run with a separate ID to differentiate prompt chains\n",
        "HEADERS = {} # don't change. headers will auto populate if use_portkey is true."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {},
      "outputs": [],
      "source": [
        "def start_wandb_run():\n",
        "  # start a new wandb run and log the config\n",
        "  wandb.init(\n",
        "    project=WANDB_PROJECT_NAME, \n",
        "    name=WANDB_RUN_NAME,\n",
        "    config={\n",
        "      \"candidate_gen_system_prompt\": candidate_gen_system_prompt, \n",
        "      \"candiate_model\": CANDIDATE_MODEL,\n",
        "      \"candidate_model_temperature\": CANDIDATE_MODEL_TEMPERATURE,\n",
        "      \"generation_model\": EVAL_MODEL,\n",
        "      \"generation_model_temperature\": EVAL_MODEL_TEMPERATURE,\n",
        "      \"generation_model_max_tokens\": EVAL_MODEL_MAX_TOKENS,\n",
        "      \"n_retries\": N_RETRIES,\n",
        "      \"number_of_prompts\": NUMBER_OF_PROMPTS\n",
        "      })\n",
        "  \n",
        "  return "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Optional logging to Weights & Biases to reocrd the configs, prompts and results\n",
        "if use_wandb:\n",
        "  start_wandb_run()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def start_portkey_run():\n",
        "  # define Portkey headers to start logging all prompts & their responses\n",
        "  openai.api_base=\"https://api.portkey.ai/v1/proxy\"\n",
        "  HEADERS = {\n",
        "    \"x-portkey-api-key\": PORTKEY_API, \n",
        "    \"x-portkey-mode\": \"proxy openai\",\n",
        "    \"x-portkey-trace-id\": PORTKEY_TRACE,\n",
        "    #\"x-portkey-retry-count\": 5 # perform automatic retries with exponential backoff if the OpenAI requests fails\n",
        "  } \n",
        "  return HEADERS"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Optional prompt & responses logging\n",
        "if use_portkey:\n",
        "    HEADERS=start_portkey_run()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "KTRFiBhSouz8"
      },
      "outputs": [],
      "source": [
        "# Get Score - retry up to N_RETRIES times, waiting exponentially between retries.\n",
        "@retry(stop=stop_after_attempt(N_RETRIES), wait=wait_exponential(multiplier=1, min=4, max=70))\n",
        "def generate_candidate_prompts(description, test_cases, number_of_prompts):\n",
        "  outputs = openai.ChatCompletion.create(\n",
        "      model=CANDIDATE_MODEL,\n",
        "      messages=[\n",
        "          {\"role\": \"system\", \"content\": candidate_gen_system_prompt},\n",
        "          {\"role\": \"user\", \"content\": f\"Here are some test cases:`{test_cases}`\\n\\nHere is the description of the use-case: `{description.strip()}`\\n\\nRespond with your prompt, and nothing else. Be creative.\"}\n",
        "          ],\n",
        "      temperature=CANDIDATE_MODEL_TEMPERATURE,\n",
        "      n=number_of_prompts,\n",
        "      headers=HEADERS)\n",
        "\n",
        "  prompts = []\n",
        "\n",
        "  for i in outputs.choices:\n",
        "    prompts.append(i.message.content)\n",
        "  return prompts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "id": "w4ltgxntszwK"
      },
      "outputs": [],
      "source": [
        "def test_candidate_prompts(test_cases, prompts):\n",
        "  prompt_results = {prompt: {'correct': 0, 'total': 0} for prompt in prompts}\n",
        "\n",
        "  # Initialize the table\n",
        "  table = PrettyTable()\n",
        "  table_field_names = [\"Prompt\", \"Expected\"] + [f\"Prompt {i+1}-{j+1}\" for j, prompt in enumerate(prompts) for i in range(prompts.count(prompt))]\n",
        "  table.field_names = table_field_names\n",
        "\n",
        "  # Wrap the text in the \"Prompt\" column\n",
        "  table.max_width[\"Prompt\"] = 100\n",
        "\n",
        "  if use_wandb:\n",
        "    wandb_table = wandb.Table(columns=table_field_names)\n",
        "    if wandb.run is None:\n",
        "      start_wandb_run()\n",
        "\n",
        "  for test_case in test_cases:\n",
        "      row = [test_case['prompt'], test_case['answer']]\n",
        "      for prompt in prompts:\n",
        "          x = openai.ChatCompletion.create(\n",
        "              model=EVAL_MODEL,\n",
        "              messages=[\n",
        "                  {\"role\": \"system\", \"content\": prompt},\n",
        "                  {\"role\": \"user\", \"content\": f\"{test_case['prompt']}\"}\n",
        "              ],\n",
        "              logit_bias={\n",
        "                  '1904': 100,  # 'true' token\n",
        "                  '3934': 100,  # 'false' token\n",
        "              },\n",
        "              max_tokens=EVAL_MODEL_MAX_TOKENS,\n",
        "              temperature=EVAL_MODEL_TEMPERATURE,\n",
        "              headers=HEADERS\n",
        "          ).choices[0].message.content\n",
        "\n",
        "\n",
        "          status = \"✅\" if x == test_case['answer'] else \"❌\"\n",
        "          row.append(status)\n",
        "\n",
        "          # Update model results\n",
        "          if x == test_case['answer']:\n",
        "              prompt_results[prompt]['correct'] += 1\n",
        "          prompt_results[prompt]['total'] += 1\n",
        "\n",
        "      table.add_row(row)\n",
        "      if use_wandb:\n",
        "        wandb_table.add_data(*row)\n",
        "\n",
        "  print(table)\n",
        "\n",
        "  # Calculate and print the percentage of correct answers and average time for each model\n",
        "  best_prompt = None\n",
        "  best_percentage = 0\n",
        "  if use_wandb:\n",
        "    prompts_results_table = wandb.Table(columns=[\"Prompt Number\", \"Prompt\", \"Percentage\", \"Correct\", \"Total\"])\n",
        "  \n",
        "  for i, prompt in enumerate(prompts):\n",
        "      correct = prompt_results[prompt]['correct']\n",
        "      total = prompt_results[prompt]['total']\n",
        "      percentage = (correct / total) * 100\n",
        "      print(f\"Prompt {i+1} got {percentage:.2f}% correct.\")\n",
        "      if use_wandb:\n",
        "         prompts_results_table.add_data(i, prompt, percentage, correct, total)\n",
        "      if percentage > best_percentage:\n",
        "          best_percentage = percentage\n",
        "          best_prompt = prompt\n",
        "\n",
        "  if use_wandb: # log the results to a Weights & Biases table and finsih the run\n",
        "    wandb.log({\"prompt_results\": prompts_results_table})\n",
        "    best_prompt_table = wandb.Table(columns=[\"Best Prompt\", \"Best Percentage\"])\n",
        "    best_prompt_table.add_data(best_prompt, best_percentage)\n",
        "    wandb.log({\"best_prompt\": best_prompt_table})\n",
        "    wandb.log({\"prompt_ratings\": wandb_table})\n",
        "    wandb.finish()\n",
        "\n",
        "  print(f\"The best prompt was '{best_prompt}' with a correctness of {best_percentage:.2f}%.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "id": "SBJEi1hkrT9T"
      },
      "outputs": [],
      "source": [
        "test_cases = [\n",
        "    {\n",
        "        'prompt': 'Find the best contact email on this site.',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'who is the current president?',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'order me a pizza',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'what are some ways a doctor could use an assistant?',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'write a speech on the danger of cults',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Make a reservation at The Accent for 9pm',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'organize my google drive',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Find the highest-rated Italian restaurant near me.',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Explain the theory of relativity.',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'What are the main differences between Python and Java programming languages?',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Translate the following English sentence to Spanish: \"The weather today is great.\"',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Create a new event on my calendar for tomorrow at 2 pm.',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Write a short story about a lonely cowboy.',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Design a logo for a startup.',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Compose a catchy jingle for a new soda brand.',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Calculate the square root of 1999.',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'What are the health benefits of yoga?',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'find me a source of meat that can be shipped to canada',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Find the best-selling book of all time.',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'What are the top 5 tourist attractions in Brazil?',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'List the main ingredients in a traditional lasagna recipe.',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'How does photosynthesis work in plants?',\n",
        "        'answer': 'true'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Write a Python program to reverse a string.',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Create a workout routine for a beginner.',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Edit my resume to highlight my project management skills.',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Draft an email to a client to discuss a new proposal.',\n",
        "        'answer': 'false'\n",
        "    },\n",
        "    {\n",
        "        'prompt': 'Plan a surprise birthday party for my best friend.',\n",
        "        'answer': 'false'\n",
        "    }]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {},
      "outputs": [],
      "source": [
        "description = \"Decide if a task is research-heavy.\" # describe the classification task clearly\n",
        "\n",
        "# If Weights & Biases is enabled, log the description and test cases too\n",
        "if use_wandb:\n",
        "    if wandb.run is None:\n",
        "      start_wandb_run()\n",
        "      wandb.config.update({\"description\": description, \n",
        "                          \"test_cases\": test_cases})\n",
        "\n",
        "candidate_prompts = generate_candidate_prompts(description, test_cases, NUMBER_OF_PROMPTS)\n",
        "test_candidate_prompts(test_cases, candidate_prompts)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "authorship_tag": "ABX9TyMvbQztC95mJY9x+Gc/uEm+",
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}