notebooks/part-1-openai.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L80oqpDfyTnx"
      },
      "source": [
        "ChatGPT is powered by `gpt-3.5-turbo` and `gpt-4`, OpenAI's most advanced models.\n",
        "\n",
        "You can build your own applications with `gpt-3.5-turbo` or `gpt-4` using the OpenAI API.\n",
        "\n",
        "Chat models take a series of messages as input, and return an AI-written message as output.\n",
        "\n",
        "This guide illustrates the chat format with a few example API calls."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Xe8AsWYyTny"
      },
      "source": [
        "## 1. Import the OpenAI library"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v3xcTFtiyTnz"
      },
      "outputs": [],
      "source": [
        "!pip install python-dotenv tiktoken tenacity openai -U --quiet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NnWuVLOzyTnz"
      },
      "outputs": [],
      "source": [
        "# import the OpenAI Python library for calling the OpenAI API\n",
        "import openai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3abbUOANyTn0"
      },
      "source": [
        "## 2. Load the API key\n",
        "In this course, we've provided some code that loads the OpenAI API key for you."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ll5HtkR7yTn0"
      },
      "outputs": [],
      "source": [
        "from dotenv import load_dotenv, find_dotenv\n",
        "import os\n",
        "_ = load_dotenv(find_dotenv()) # read local .env file\n",
        "\n",
        "# openai.api_key  = os.environ['OPENAI_API_KEY']\n",
        "openai.api_key = <OPENAI-API-KEY>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "id": "1nc6BdQ1yTn0"
      },
      "source": [
        "## 3. An example chat API call\n",
        "\n",
        "A chat API call has two required inputs:\n",
        "- `model`: the name of the model you want to use (e.g., `gpt-3.5-turbo`, `gpt-4`, `gpt-3.5-turbo-0613`, `gpt-3.5-turbo-16k-0613`)\n",
        "- `messages`: a list of message objects, where each object has two required fields:\n",
        "    - `role`: the role of the messenger (either `system`, `user`, or `assistant`)\n",
        "    - `content`: the content of the message (e.g., `Write me a beautiful poem`)\n",
        "\n",
        "Messages can also contain an optional `name` field, which give the messenger a name. E.g., `example-user`, `Alice`, `BlackbeardBot`. Names may not contain spaces.\n",
        "\n",
        "As of June 2023, you can also optionally submit a list of `functions` that tell GPT whether it can generate JSON to feed into a function. For details, see the [documentation](https://platform.openai.com/docs/guides/gpt/function-calling), [API reference](https://platform.openai.com/docs/api-reference/chat), or the Cookbook guide [How to call functions with chat models](How_to_call_functions_with_chat_models.ipynb).\n",
        "\n",
        "Typically, a conversation will start with a system message that tells the assistant how to behave, followed by alternating user and assistant messages, but you are not required to follow this format.\n",
        "\n",
        "Let's look at an example chat API calls to see how the chat format works in practice."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "VE5iEoPAyTn0"
      },
      "outputs": [],
      "source": [
        "# Example OpenAI Python library request\n",
        "MODEL = \"gpt-3.5-turbo\"\n",
        "response = openai.ChatCompletion.create(\n",
        "    model=MODEL,\n",
        "    messages=[\n",
        "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
        "        {\"role\": \"user\", \"content\": \"Knock knock.\"},\n",
        "        {\"role\": \"assistant\", \"content\": \"Who's there?\"},\n",
        "        {\"role\": \"user\", \"content\": \"Orange.\"},\n",
        "    ],\n",
        "    temperature=0,\n",
        ")\n",
        "\n",
        "response\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "gnlKwjvLyTn1"
      },
      "source": [
        "As you can see, the response object has a few fields:\n",
        "- `id`: the ID of the request\n",
        "- `object`: the type of object returned (e.g., `chat.completion`)\n",
        "- `created`: the timestamp of the request\n",
        "- `model`: the full name of the model used to generate the response\n",
        "- `usage`: the number of tokens used to generate the replies, counting prompt, completion, and total\n",
        "- `choices`: a list of completion objects (only one, unless you set `n` greater than 1)\n",
        "    - `message`: the message object generated by the model, with `role` and `content`\n",
        "    - `finish_reason`: the reason the model stopped generating text (either `stop`, or `length` if `max_tokens` limit was reached)\n",
        "    - `index`: the index of the completion in the list of choices"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "JXkb2KkfyTn1"
      },
      "source": [
        "Extract just the reply with:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "xFtwf_75yTn1"
      },
      "outputs": [],
      "source": [
        "response['choices'][0]['message']['content']\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "582tS5HTyTn2"
      },
      "source": [
        "Even non-conversation-based tasks can fit into the chat format, by placing the instruction in the first user message.\n",
        "\n",
        "For example, to ask the model to explain asynchronous programming in the style of the pirate Blackbeard, we can structure conversation as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "g1l8W6xCyTn2"
      },
      "outputs": [],
      "source": [
        "# example with a system message\n",
        "response = openai.ChatCompletion.create(\n",
        "    model=MODEL,\n",
        "    messages=[\n",
        "        {\"role\": \"system\", \"content\": \"You are a helpful assistant who replies only in the style of the pirate Blackbeard.\"},\n",
        "        {\"role\": \"user\", \"content\": \"Explain asynchronous programming \"},\n",
        "    ],\n",
        "    temperature=0,\n",
        ")\n",
        "\n",
        "print(response['choices'][0]['message']['content'])\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "VDWN-UiNyTn2"
      },
      "outputs": [],
      "source": [
        "# example without a system message\n",
        "response = openai.ChatCompletion.create(\n",
        "    model=MODEL,\n",
        "    messages=[\n",
        "        {\"role\": \"user\", \"content\": \"Explain asynchronous programming.\"},\n",
        "    ],\n",
        "    temperature=0,\n",
        ")\n",
        "\n",
        "print(response['choices'][0]['message']['content'])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "id": "-MsetkmkyTn2"
      },
      "source": [
        "## 4. Tips for instructing gpt-3.5-turbo-0301\n",
        "\n",
        "Best practices for instructing models may change from model version to model version. The advice that follows applies to `gpt-3.5-turbo-0301` and may not apply to future models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "3VriaI0ayTn2"
      },
      "source": [
        "### System messages\n",
        "\n",
        "The system message can be used to prime the assistant with different personalities or behaviors.\n",
        "\n",
        "Be aware that `gpt-3.5-turbo-0301` does not generally pay as much attention to the system message as `gpt-4-0314` or `gpt-3.5-turbo-0613`. Therefore, for `gpt-3.5-turbo-0301`, we recommend placing important instructions in the user message instead. Some developers have found success in continually moving the system message near the end of the conversation to keep the model's attention from drifting away as conversations get longer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "CVIv9PgEyTn2"
      },
      "outputs": [],
      "source": [
        "# An example of a system message that primes the assistant to explain concepts in great depth\n",
        "response = openai.ChatCompletion.create(\n",
        "    model=MODEL,\n",
        "    messages=[\n",
        "        {\"role\": \"system\", \"content\": \"You are a friendly and helpful teaching assistant. You explain concepts in great depth using simple terms, and you give examples to help people learn. At the end of each explanation, you ask a question to check for understanding\"},\n",
        "        {\"role\": \"user\", \"content\": \"Can you explain how fractions work?\"},\n",
        "    ],\n",
        "    temperature=0,\n",
        ")\n",
        "\n",
        "print(response[\"choices\"][0][\"message\"][\"content\"])\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "d1WbAEEXyTn2"
      },
      "outputs": [],
      "source": [
        "# An example of a system message that primes the assistant to give brief, to-the-point answers\n",
        "response = openai.ChatCompletion.create(\n",
        "    model=MODEL,\n",
        "    messages=[\n",
        "        {\"role\": \"system\", \"content\": \"You are a laconic assistant. You reply with brief, to-the-point answers with no elaboration.\"},\n",
        "        {\"role\": \"user\", \"content\": \"Can you explain how fractions work?\"},\n",
        "    ],\n",
        "    temperature=0,\n",
        ")\n",
        "\n",
        "print(response[\"choices\"][0][\"message\"][\"content\"])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "vvvcWVV9yTn2"
      },
      "source": [
        "### Few-shot prompting\n",
        "\n",
        "In some cases, it's easier to show the model what you want rather than tell the model what you want.\n",
        "\n",
        "One way to show the model what you want is with faked example messages.\n",
        "\n",
        "For example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "U0g2TiWayTn2"
      },
      "outputs": [],
      "source": [
        "# An example of a faked few-shot conversation to prime the model into translating business jargon to simpler speech\n",
        "response = openai.ChatCompletion.create(\n",
        "    model=MODEL,\n",
        "    messages=[\n",
        "        {\"role\": \"system\", \"content\": \"You are a helpful, pattern-following assistant.\"},\n",
        "        {\"role\": \"user\", \"content\": \"Help me translate the following corporate jargon into plain English.\"},\n",
        "        {\"role\": \"assistant\", \"content\": \"Sure, I'd be happy to!\"},\n",
        "        {\"role\": \"user\", \"content\": \"New synergies will help drive top-line growth.\"},\n",
        "        {\"role\": \"assistant\", \"content\": \"Things working well together will increase revenue.\"},\n",
        "        {\"role\": \"user\", \"content\": \"Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.\"},\n",
        "        {\"role\": \"assistant\", \"content\": \"Let's talk later when we're less busy about how to do better.\"},\n",
        "        {\"role\": \"user\", \"content\": \"This late pivot means we don't have time to boil the ocean for the client deliverable.\"},\n",
        "    ],\n",
        "    temperature=0,\n",
        ")\n",
        "\n",
        "print(response[\"choices\"][0][\"message\"][\"content\"])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "H1YFiBfIyTn2"
      },
      "source": [
        "To help clarify that the example messages are not part of a real conversation, and shouldn't be referred back to by the model, you can try setting the `name` field of `system` messages to `example_user` and `example_assistant`.\n",
        "\n",
        "Transforming the few-shot example above, we could write:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "nRtJhBuSyTn3"
      },
      "outputs": [],
      "source": [
        "# The business jargon translation example, but with example names for the example messages\n",
        "response = openai.ChatCompletion.create(\n",
        "    model=MODEL,\n",
        "    messages=[\n",
        "        {\"role\": \"system\", \"content\": \"You are a helpful, pattern-following assistant that translates corporate jargon into plain English.\"},\n",
        "        {\"role\": \"system\", \"name\":\"example_user\", \"content\": \"New synergies will help drive top-line growth.\"},\n",
        "        {\"role\": \"system\", \"name\": \"example_assistant\", \"content\": \"Things working well together will increase revenue.\"},\n",
        "        {\"role\": \"system\", \"name\":\"example_user\", \"content\": \"Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.\"},\n",
        "        {\"role\": \"system\", \"name\": \"example_assistant\", \"content\": \"Let's talk later when we're less busy about how to do better.\"},\n",
        "        {\"role\": \"user\", \"content\": \"This late pivot means we don't have time to boil the ocean for the client deliverable.\"},\n",
        "    ],\n",
        "    temperature=0,\n",
        ")\n",
        "\n",
        "print(response[\"choices\"][0][\"message\"][\"content\"])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "B0A4WMcQyTn3"
      },
      "source": [
        "Not every attempt at engineering conversations will succeed at first.\n",
        "\n",
        "If your first attempts fail, don't be afraid to experiment with different ways of priming or conditioning the model.\n",
        "\n",
        "As an example, one developer discovered an increase in accuracy when they inserted a user message that said \"Great job so far, these have been perfect\" to help condition the model into providing higher quality responses.\n",
        "\n",
        "For more ideas on how to lift the reliability of the models, consider reading our guide on [techniques to increase reliability](../techniques_to_improve_reliability.md). It was written for non-chat models, but many of its principles still apply."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "id": "-_zAFGnryTn3"
      },
      "source": [
        "## 5. Counting tokens\n",
        "\n",
        "When you submit your request, the API transforms the messages into a sequence of tokens.\n",
        "\n",
        "The number of tokens used affects:\n",
        "- the cost of the request\n",
        "- the time it takes to generate the response\n",
        "- when the reply gets cut off from hitting the maximum token limit (4,096 for `gpt-3.5-turbo` or 8,192 for `gpt-4`)\n",
        "\n",
        "You can use the following function to count the number of tokens that a list of messages will use.\n",
        "\n",
        "Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee.\n",
        "\n",
        "In particular, requests that use the optional functions input will consume extra tokens on top of the estimates calculated below.\n",
        "\n",
        "Read more about counting tokens in [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "Uai8KoJhyTn3"
      },
      "outputs": [],
      "source": [
        "import tiktoken\n",
        "\n",
        "\n",
        "def num_tokens_from_messages(messages, model=\"gpt-3.5-turbo-0613\"):\n",
        "    \"\"\"Return the number of tokens used by a list of messages.\"\"\"\n",
        "    try:\n",
        "        encoding = tiktoken.encoding_for_model(model)\n",
        "    except KeyError:\n",
        "        print(\"Warning: model not found. Using cl100k_base encoding.\")\n",
        "        encoding = tiktoken.get_encoding(\"cl100k_base\")\n",
        "    if model in {\n",
        "        \"gpt-3.5-turbo-0613\",\n",
        "        \"gpt-3.5-turbo-16k-0613\",\n",
        "        \"gpt-4-0314\",\n",
        "        \"gpt-4-32k-0314\",\n",
        "        \"gpt-4-0613\",\n",
        "        \"gpt-4-32k-0613\",\n",
        "        }:\n",
        "        tokens_per_message = 3\n",
        "        tokens_per_name = 1\n",
        "    elif model == \"gpt-3.5-turbo-0301\":\n",
        "        tokens_per_message = 4  # every message follows <|start|>{role/name}\\n{content}<|end|>\\n\n",
        "        tokens_per_name = -1  # if there's a name, the role is omitted\n",
        "    elif \"gpt-3.5-turbo\" in model:\n",
        "        print(\"Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.\")\n",
        "        return num_tokens_from_messages(messages, model=\"gpt-3.5-turbo-0613\")\n",
        "    elif \"gpt-4\" in model:\n",
        "        print(\"Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.\")\n",
        "        return num_tokens_from_messages(messages, model=\"gpt-4-0613\")\n",
        "    else:\n",
        "        raise NotImplementedError(\n",
        "            f\"\"\"num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens.\"\"\"\n",
        "        )\n",
        "    num_tokens = 0\n",
        "    for message in messages:\n",
        "        num_tokens += tokens_per_message\n",
        "        for key, value in message.items():\n",
        "            num_tokens += len(encoding.encode(value))\n",
        "            if key == \"name\":\n",
        "                num_tokens += tokens_per_name\n",
        "    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>\n",
        "    return num_tokens\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "vQVAEQMeyTn3"
      },
      "outputs": [],
      "source": [
        "# let's verify the function above matches the OpenAI API response\n",
        "\n",
        "import openai\n",
        "\n",
        "example_messages = [\n",
        "    {\n",
        "        \"role\": \"system\",\n",
        "        \"content\": \"You are a helpful, pattern-following assistant that translates corporate jargon into plain English.\",\n",
        "    },\n",
        "    {\n",
        "        \"role\": \"system\",\n",
        "        \"name\": \"example_user\",\n",
        "        \"content\": \"New synergies will help drive top-line growth.\",\n",
        "    },\n",
        "    {\n",
        "        \"role\": \"system\",\n",
        "        \"name\": \"example_assistant\",\n",
        "        \"content\": \"Things working well together will increase revenue.\",\n",
        "    },\n",
        "    {\n",
        "        \"role\": \"system\",\n",
        "        \"name\": \"example_user\",\n",
        "        \"content\": \"Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.\",\n",
        "    },\n",
        "    {\n",
        "        \"role\": \"system\",\n",
        "        \"name\": \"example_assistant\",\n",
        "        \"content\": \"Let's talk later when we're less busy about how to do better.\",\n",
        "    },\n",
        "    {\n",
        "        \"role\": \"user\",\n",
        "        \"content\": \"This late pivot means we don't have time to boil the ocean for the client deliverable.\",\n",
        "    },\n",
        "]\n",
        "\n",
        "for model in [\n",
        "    \"gpt-3.5-turbo-0301\",\n",
        "    \"gpt-3.5-turbo-0613\",\n",
        "    \"gpt-3.5-turbo\",\n",
        "    \"gpt-4-0314\",\n",
        "    \"gpt-4-0613\",\n",
        "    \"gpt-4\",\n",
        "    ]:\n",
        "    print(model)\n",
        "    # example token count from the function defined above\n",
        "    print(f\"{num_tokens_from_messages(example_messages, model)} prompt tokens counted by num_tokens_from_messages().\")\n",
        "    # example token count from the OpenAI API\n",
        "    response = openai.ChatCompletion.create(\n",
        "        model=model,\n",
        "        messages=example_messages,\n",
        "        temperature=0,\n",
        "        max_tokens=1,  # we're only counting input tokens here, so let's not waste tokens on the output\n",
        "    )\n",
        "    print(f'{response[\"usage\"][\"prompt_tokens\"]} prompt tokens counted by the OpenAI API.')\n",
        "    print()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "id": "WCJJR3zzyTn3"
      },
      "source": [
        "## 6. Handling Rate Limits\n",
        "\n",
        "When you call the OpenAI API repeatedly, you may encounter error messages that say `429: 'Too Many Requests'` or `RateLimitError`. These error messages come from exceeding the API's rate limits.\n",
        "\n",
        "This guide shares tips for avoiding and handling rate limit errors.\n",
        "\n",
        "To see an example script for throttling parallel requests to avoid rate limit errors, see [api_request_parallel_processor.py](api_request_parallel_processor.py).\n",
        "\n",
        "## Why rate limits exist\n",
        "\n",
        "Rate limits are a common practice for APIs, and they're put in place for a few different reasons.\n",
        "\n",
        "- First, they help protect against abuse or misuse of the API. For example, a malicious actor could flood the API with requests in an attempt to overload it or cause disruptions in service. By setting rate limits, OpenAI can prevent this kind of activity.\n",
        "- Second, rate limits help ensure that everyone has fair access to the API. If one person or organization makes an excessive number of requests, it could bog down the API for everyone else. By throttling the number of requests that a single user can make, OpenAI ensures that everyone has an opportunity to use the API without experiencing slowdowns.\n",
        "- Lastly, rate limits can help OpenAI manage the aggregate load on its infrastructure. If requests to the API increase dramatically, it could tax the servers and cause performance issues. By setting rate limits, OpenAI can help maintain a smooth and consistent experience for all users.\n",
        "\n",
        "Although hitting rate limits can be frustrating, rate limits exist to protect the reliable operation of the API for its users.\n",
        "\n",
        "## Default rate limits\n",
        "\n",
        "As of Jan 2023, the default rate limits are:\n",
        "\n",
        "<table>\n",
        "<thead>\n",
        "  <tr>\n",
        "    <th></th>\n",
        "    <th>Text Completion &amp; Embedding endpoints</th>\n",
        "    <th>Code &amp; Edit endpoints</th>\n",
        "  </tr>\n",
        "</thead>\n",
        "<tbody>\n",
        "  <tr>\n",
        "    <td>Free trial users</td>\n",
        "    <td>\n",
        "        <ul>\n",
        "            <li>20 requests / minute</li>\n",
        "            <li>150,000 tokens / minute</li>\n",
        "        </ul>\n",
        "    </td>\n",
        "    <td>\n",
        "        <ul>\n",
        "            <li>20 requests / minute</li>\n",
        "            <li>150,000 tokens / minute</li>\n",
        "        </ul>\n",
        "    </td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td>Pay-as-you-go users (in your first 48 hours)</td>\n",
        "    <td>\n",
        "        <ul>\n",
        "            <li>60 requests / minute</li>\n",
        "            <li>250,000 davinci tokens / minute (and proportionally more for cheaper models)</li>\n",
        "        </ul>\n",
        "    </td>\n",
        "    <td>\n",
        "        <ul>\n",
        "            <li>20 requests / minute</li>\n",
        "            <li>150,000 tokens / minute</li>\n",
        "        </ul>\n",
        "    </td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td>Pay-as-you-go users (after your first 48 hours)</td>\n",
        "    <td>\n",
        "        <ul>\n",
        "            <li>3,000 requests / minute</li>\n",
        "            <li>250,000 davinci tokens / minute (and proportionally more for cheaper models)</li>\n",
        "        </ul>\n",
        "    </td>\n",
        "    <td>\n",
        "        <ul>\n",
        "            <li>20 requests / minute</li>\n",
        "            <li>150,000 tokens / minute</li>\n",
        "        </ul>\n",
        "    </td>\n",
        "  </tr>\n",
        "</tbody>\n",
        "</table>\n",
        "\n",
        "For reference, 1,000 tokens is roughly a page of text.\n",
        "\n",
        "### Other rate limit resources\n",
        "\n",
        "Read more about OpenAI's rate limits in these other resources:\n",
        "\n",
        "- [Guide: Rate limits](https://beta.openai.com/docs/guides/rate-limits/overview)\n",
        "- [Help Center: Is API usage subject to any rate limits?](https://help.openai.com/en/articles/5955598-is-api-usage-subject-to-any-rate-limits)\n",
        "- [Help Center: How can I solve 429: 'Too Many Requests' errors?](https://help.openai.com/en/articles/5955604-how-can-i-solve-429-too-many-requests-errors)\n",
        "\n",
        "### Requesting a rate limit increase\n",
        "\n",
        "If you'd like your organization's rate limit increased, please fill out the following form:\n",
        "\n",
        "- [OpenAI Rate Limit Increase Request form](https://forms.gle/56ZrwXXoxAN1yt6i9)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "ke5p0PjvyTn3"
      },
      "source": [
        "### Example rate limit error\n",
        "\n",
        "A rate limit error will occur when API requests are sent too quickly. If using the OpenAI Python library, they will look something like:\n",
        "\n",
        "```\n",
        "RateLimitError: Rate limit reached for default-codex in organization org-{id} on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues or if you’d like to request an increase.\n",
        "```\n",
        "\n",
        "Below is example code for triggering a rate limit error."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "hen8ns4NyTn3",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# import openai  # for making OpenAI API requests\n",
        "\n",
        "# # request a bunch of completions in a loop\n",
        "# for _ in range(100):\n",
        "#     openai.Completion.create(\n",
        "#         model=\"code-cushman-001\",\n",
        "#         prompt=\"def magic_function():\\n\\t\",\n",
        "#         max_tokens=10,\n",
        "#     )\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "VdvtNGgqyTn4"
      },
      "source": [
        "### How to avoid rate limit errors\n",
        "\n",
        "#### Retrying with exponential backoff\n",
        "\n",
        "One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached.\n",
        "\n",
        "This approach has many benefits:\n",
        "\n",
        "- Automatic retries means you can recover from rate limit errors without crashes or missing data\n",
        "- Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail\n",
        "- Adding random jitter to the delay helps retries from all hitting at the same time\n",
        "\n",
        "Note that unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work.\n",
        "\n",
        "Below are a few example solutions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "h-_gpXKfyTn4"
      },
      "source": [
        "#### Using the Tenacity library\n",
        "\n",
        "[Tenacity](https://tenacity.readthedocs.io/en/latest/) is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.\n",
        "\n",
        "To add exponential backoff to your requests, you can use the `tenacity.retry` [decorator](https://peps.python.org/pep-0318/). The following example uses the `tenacity.wait_random_exponential` function to add random exponential backoff to a request.\n",
        "\n",
        "Note that the Tenacity library is a third-party tool, and OpenAI makes no guarantees about its reliability or security."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "NZxj1BkvyTn4"
      },
      "outputs": [],
      "source": [
        "import openai  # for OpenAI API calls\n",
        "from tenacity import (\n",
        "    retry,\n",
        "    stop_after_attempt,\n",
        "    wait_random_exponential,\n",
        ")  # for exponential backoff\n",
        "\n",
        "\n",
        "@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))\n",
        "def completion_with_backoff(**kwargs):\n",
        "    return openai.Completion.create(**kwargs)\n",
        "\n",
        "\n",
        "completion_with_backoff(model=\"text-davinci-002\", prompt=\"Once upon a time,\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "heading_collapsed": true,
        "id": "xeoproh0yTn4"
      },
      "source": [
        "## 7. How to stream completions\n",
        "\n",
        "By default, when you request a completion from the OpenAI, the entire completion is generated before being sent back in a single response.\n",
        "\n",
        "If you're generating long completions, waiting for the response can take many seconds.\n",
        "\n",
        "To get responses sooner, you can 'stream' the completion as it's being generated. This allows you to start printing or processing the beginning of the completion before the full completion is finished.\n",
        "\n",
        "To stream completions, set `stream=True` when calling the chat completions or completions endpoints. This will return an object that streams back the response as [data-only server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). Extract chunks from the `delta` field rather than the `message` field.\n",
        "\n",
        "### Downsides\n",
        "\n",
        "Note that using `stream=True` in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate. which has implications for [approved usage](https://beta.openai.com/docs/usage-guidelines).\n",
        "\n",
        "Another small drawback of streaming responses is that the response no longer includes the `usage` field to tell you how many tokens were consumed. After receiving and combining all of the responses, you can calculate this yourself using [`tiktoken`](How_to_count_tokens_with_tiktoken.ipynb).\n",
        "\n",
        "### Example code\n",
        "\n",
        "Below is shown:\n",
        "1. What a typical chat completion response looks like\n",
        "2. What a streaming chat completion response looks like\n",
        "3. How much time is saved by streaming a chat completion\n",
        "4. How to stream non-chat completions (used by older models like `text-davinci-003`)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "8tHi3dsTyTn4"
      },
      "outputs": [],
      "source": [
        "# imports\n",
        "import openai  # for OpenAI API calls\n",
        "import time  # for measuring time duration of API calls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "5_wTiD-9yTn4"
      },
      "source": [
        "### 1. What a typical chat completion response looks like\n",
        "\n",
        "With a typical ChatCompletions API call, the response is first computed and then returned all at once."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "6dP052HAyTn4"
      },
      "outputs": [],
      "source": [
        "# Example of an OpenAI ChatCompletion request\n",
        "# https://platform.openai.com/docs/guides/chat\n",
        "\n",
        "# record the time before the request is sent\n",
        "start_time = time.time()\n",
        "\n",
        "# send a ChatCompletion request to count to 100\n",
        "response = openai.ChatCompletion.create(\n",
        "    model='gpt-3.5-turbo',\n",
        "    messages=[\n",
        "        {'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ...'}\n",
        "    ],\n",
        "    temperature=0,\n",
        ")\n",
        "\n",
        "# calculate the time it took to receive the response\n",
        "response_time = time.time() - start_time\n",
        "\n",
        "# print the time delay and text received\n",
        "print(f\"Full response received {response_time:.2f} seconds after request\")\n",
        "print(f\"Full response received:\\n{response}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "dPZQgJRiyTn4"
      },
      "source": [
        "The reply can be extracted with `response['choices'][0]['message']`.\n",
        "\n",
        "The content of the reply can be extracted with `response['choices'][0]['message']['content']`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "Lu8Wy4O1yTn4"
      },
      "outputs": [],
      "source": [
        "reply = response['choices'][0]['message']\n",
        "print(f\"Extracted reply: \\n{reply}\")\n",
        "\n",
        "reply_content = response['choices'][0]['message']['content']\n",
        "print(f\"Extracted content: \\n{reply_content}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "L8xUDTF3yTn5"
      },
      "source": [
        "### 2. How to stream a chat completion\n",
        "\n",
        "With a streaming API call, the response is sent back incrementally in chunks via an [event stream](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). In Python, you can iterate over these events with a `for` loop.\n",
        "\n",
        "Let's see what it looks like:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "169gnOafyTn5"
      },
      "outputs": [],
      "source": [
        "# Example of an OpenAI ChatCompletion request with stream=True\n",
        "# https://platform.openai.com/docs/guides/chat\n",
        "\n",
        "# a ChatCompletion request\n",
        "response = openai.ChatCompletion.create(\n",
        "    model='gpt-3.5-turbo',\n",
        "    messages=[\n",
        "        {'role': 'user', 'content': \"What's 1+1? Answer in one word.\"}\n",
        "    ],\n",
        "    temperature=0,\n",
        "    stream=True  # this time, we set stream=True\n",
        ")\n",
        "\n",
        "for chunk in response:\n",
        "    print(chunk)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "CYg72YCOyTn5"
      },
      "source": [
        "As you can see above, streaming responses have a `delta` field rather than a `message` field. `delta` can hold things like:\n",
        "- a role token (e.g., `{\"role\": \"assistant\"}`)\n",
        "- a content token (e.g., `{\"content\": \"\\n\\n\"}`)\n",
        "- nothing (e.g., `{}`), when the stream is over"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "6EApiYcCyTn5"
      },
      "source": [
        "### 3. How much time is saved by streaming a chat completion\n",
        "\n",
        "Now let's ask `gpt-3.5-turbo` to count to 100 again, and see how long it takes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "ZwL2dBoJyTn5",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# Example of an OpenAI ChatCompletion request with stream=True\n",
        "# https://platform.openai.com/docs/guides/chat\n",
        "\n",
        "# record the time before the request is sent\n",
        "start_time = time.time()\n",
        "\n",
        "# send a ChatCompletion request to count to 100\n",
        "response = openai.ChatCompletion.create(\n",
        "    model='gpt-3.5-turbo',\n",
        "    messages=[\n",
        "        {'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ...'}\n",
        "    ],\n",
        "    temperature=0,\n",
        "    stream=True  # again, we set stream=True\n",
        ")\n",
        "\n",
        "# create variables to collect the stream of chunks\n",
        "collected_chunks = []\n",
        "collected_messages = []\n",
        "# iterate through the stream of events\n",
        "for chunk in response:\n",
        "    chunk_time = time.time() - start_time  # calculate the time delay of the chunk\n",
        "    collected_chunks.append(chunk)  # save the event response\n",
        "    chunk_message = chunk['choices'][0]['delta']  # extract the message\n",
        "    collected_messages.append(chunk_message)  # save the message\n",
        "    print(f\"Message received {chunk_time:.2f} seconds after request: {chunk_message}\")  # print the delay and text\n",
        "\n",
        "# print the time delay and text received\n",
        "print(f\"Full response received {chunk_time:.2f} seconds after request\")\n",
        "full_reply_content = ''.join([m.get('content', '') for m in collected_messages])\n",
        "print(f\"Full conversation received: {full_reply_content}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "5uWrDXCwyTn6"
      },
      "source": [
        "#### Time comparison\n",
        "\n",
        "In the example above, both requests took about 3 seconds to fully complete. Request times will vary depending on load and other stochastic factors.\n",
        "\n",
        "However, with the streaming request, we received the first token after 0.1 seconds, and subsequent tokens every ~0.01-0.02 seconds."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "bffR_uJNyTn6"
      },
      "source": [
        "### 4. How to stream non-chat completions (used by older models like `text-davinci-003`)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "MbvHSROVyTn6"
      },
      "source": [
        "#### A typical completion request\n",
        "\n",
        "With a typical Completions API call, the text is first computed and then returned all at once."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "ZG64Eks-yTn6"
      },
      "outputs": [],
      "source": [
        "# Example of an OpenAI Completion request\n",
        "# https://beta.openai.com/docs/api-reference/completions/create\n",
        "\n",
        "# record the time before the request is sent\n",
        "start_time = time.time()\n",
        "\n",
        "# send a Completion request to count to 100\n",
        "response = openai.Completion.create(\n",
        "    model='text-davinci-002',\n",
        "    prompt='1,2,3,',\n",
        "    max_tokens=193,\n",
        "    temperature=0,\n",
        ")\n",
        "\n",
        "# calculate the time it took to receive the response\n",
        "response_time = time.time() - start_time\n",
        "\n",
        "# extract the text from the response\n",
        "completion_text = response['choices'][0]['text']\n",
        "\n",
        "# print the time delay and text received\n",
        "print(f\"Full response received {response_time:.2f} seconds after request\")\n",
        "print(f\"Full text received: {completion_text}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "lO0dZrdHyTn6"
      },
      "source": [
        "#### A streaming completion request\n",
        "\n",
        "With a streaming Completions API call, the text is sent back via a series of events. In Python, you can iterate over these events with a `for` loop."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "hidden": true,
        "id": "AFC-Xtb_yTn6",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# Example of an OpenAI Completion request, using the stream=True option\n",
        "# https://beta.openai.com/docs/api-reference/completions/create\n",
        "\n",
        "# record the time before the request is sent\n",
        "start_time = time.time()\n",
        "\n",
        "# send a Completion request to count to 100\n",
        "response = openai.Completion.create(\n",
        "    model='text-davinci-002',\n",
        "    prompt='1,2,3,',\n",
        "    max_tokens=193,\n",
        "    temperature=0,\n",
        "    stream=True,  # this time, we set stream=True\n",
        ")\n",
        "\n",
        "# create variables to collect the stream of events\n",
        "collected_events = []\n",
        "completion_text = ''\n",
        "# iterate through the stream of events\n",
        "for event in response:\n",
        "    event_time = time.time() - start_time  # calculate the time delay of the event\n",
        "    collected_events.append(event)  # save the event response\n",
        "    event_text = event['choices'][0]['text']  # extract the text\n",
        "    completion_text += event_text  # append the text\n",
        "    print(f\"Text received: {event_text} ({event_time:.2f} seconds after request)\")  # print the delay and text\n",
        "\n",
        "# print the time delay and text received\n",
        "print(f\"Full response received {event_time:.2f} seconds after request\")\n",
        "print(f\"Full text received: {completion_text}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "tr2fBnlPyTn6"
      },
      "source": [
        "#### Time comparison\n",
        "\n",
        "In the example above, both requests took about 3 seconds to fully complete. Request times will vary depending on load and other stochastic factors.\n",
        "\n",
        "However, with the streaming request, we received the first token after 0.18 seconds, and subsequent tokens every ~0.01-0.02 seconds."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "hidden": true,
        "id": "_AfZXKWLyTn6"
      },
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "vscode": {
      "interpreter": {
        "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}