05-Adding_Memory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "01a8b5c0-87cb-4302-8e3c-dc809d0039fb",
   "metadata": {},
   "source": [
    "# Understanding Memory in LLMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f73380-6395-4e9f-9c83-3f47a5d7e292",
   "metadata": {},
   "source": [
    "In the previous Notebooks, we successfully explored how OpenAI models can enhance the results from Azure AI Search queries. \n",
    "\n",
    "However, we have yet to discover how to engage in a conversation with the LLM. With [Bing Chat](http://chat.bing.com/), for example, this is possible, as it can understand and reference the previous responses.\n",
    "\n",
    "There is a common misconception that LLMs (Large Language Models) have memory. This is not true. While they possess knowledge, they do not retain information from previous questions asked to them.\n",
    "\n",
    "In this Notebook, our goal is to illustrate how we can effectively \"endow the LLM with memory\" by employing prompts and context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "733c782e-204c-47d0-8dae-c9df7091ab23",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "from langchain_community.chat_message_histories import ChatMessageHistory, CosmosDBChatMessageHistory\n",
    "from langchain_core.chat_history import BaseChatMessageHistory\n",
    "from langchain_core.runnables import ConfigurableFieldSpec\n",
    "from langchain_core.runnables.history import RunnableWithMessageHistory\n",
    "from langchain_openai import AzureChatOpenAI\n",
    "from langchain_openai import AzureOpenAIEmbeddings\n",
    "from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from operator import itemgetter\n",
    "from typing import List\n",
    "\n",
    "from IPython.display import Markdown, HTML, display  \n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "\n",
    "#custom libraries that we will use later in the app\n",
    "from common.utils import CustomAzureSearchRetriever, get_answer\n",
    "from common.prompts import DOCSEARCH_PROMPT\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "import logging\n",
    "\n",
    "# Get the root logger\n",
    "logger = logging.getLogger()\n",
    "# Set the logging level to a higher level to ignore INFO messages\n",
    "logger.setLevel(logging.WARNING)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6bc63c55-a57d-49a7-b6c7-0f18bca8199e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc72b22-11c2-4df0-91b8-033d01829663",
   "metadata": {},
   "source": [
    "### Let's start with the basics\n",
    "Let's use a very simple example to see if the GPT model of Azure OpenAI have memory. We again will be using langchain to simplify our code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3eef5dc9-8b80-4085-980c-865fa41fa1f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "QUESTION = \"Tell me some use cases for reinforcement learning\"\n",
    "FOLLOW_UP_QUESTION = \"What was my prior question?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a00181d5-bd76-4ce4-a256-75ac5b58c60f",
   "metadata": {},
   "outputs": [],
   "source": [
    "COMPLETION_TOKENS = 1000\n",
    "# Create an OpenAI instance\n",
    "llm = AzureChatOpenAI(deployment_name=os.environ[\"GPT35_DEPLOYMENT_NAME\"], \n",
    "                      temperature=0.5, max_tokens=COMPLETION_TOKENS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9502d0f1-fddf-40d1-95d2-a1461dcc498a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We create a very simple prompt template, just the question as is:\n",
    "output_parser = StrOutputParser()\n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    (\"system\", \"You are an assistant that give thorough responses to users.\"),\n",
    "    (\"user\", \"{input}\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c5c9903e-15c7-4e05-87a1-58e5a7917ba2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning has a wide range of applications across various fields. Some common use cases include:\n",
       "\n",
       "1. **Game playing**: Reinforcement learning has been successfully used to train agents to play complex games like Chess, Go, and video games. For example, AlphaGo, developed by DeepMind, used reinforcement learning to become the world champion in the game of Go.\n",
       "\n",
       "2. **Robotics**: Reinforcement learning is used to train robots to perform complex tasks such as grasping objects, navigating environments, and manipulating objects. This enables robots to learn from experience and improve their performance over time.\n",
       "\n",
       "3. **Autonomous vehicles**: Reinforcement learning is used to train autonomous vehicles to make decisions in real-time while navigating through traffic, following traffic rules, and avoiding collisions. This technology is crucial for the development of self-driving cars.\n",
       "\n",
       "4. **Recommendation systems**: Reinforcement learning can be used to optimize recommendation systems by learning user preferences and providing personalized recommendations for products, movies, music, and more.\n",
       "\n",
       "5. **Finance**: Reinforcement learning is used in algorithmic trading to make decisions on buying, selling, and holding financial assets based on market conditions. It can also be used for risk management and portfolio optimization.\n",
       "\n",
       "6. **Healthcare**: Reinforcement learning can be used to optimize treatment plans for patients by learning from patient data and adjusting treatment strategies to maximize patient outcomes.\n",
       "\n",
       "7. **Resource management**: Reinforcement learning can be used to optimize resource allocation in various industries such as energy, logistics, and manufacturing. It can help in scheduling tasks, managing inventory, and improving efficiency.\n",
       "\n",
       "8. **Natural language processing**: Reinforcement learning can be used to train chatbots and virtual assistants to interact with users in a more natural and intelligent way. It can also be used for language translation and text summarization.\n",
       "\n",
       "These are just a few examples of the many use cases for reinforcement learning. It is a versatile and powerful technique that can be applied to a wide range of problems in different domains."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's see what the GPT model responds\n",
    "chain = prompt | llm | output_parser\n",
    "response_to_initial_question = chain.invoke({\"input\": QUESTION})\n",
    "display(Markdown(response_to_initial_question))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "99acaf3c-ce68-4b87-b24a-6065b15ff9a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was \"What was my prior question?\""
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Now let's ask a follow up question\n",
    "printmd(chain.invoke({\"input\": FOLLOW_UP_QUESTION}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3e1c143-c95f-4566-a8b4-af8c42f08dd2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "As you can see, it doesn't remember what it just responded, sometimes it responds based only on the system prompt, or just randomly. This proof that the LLM does NOT have memory and that we need to give the memory as a a conversation history as part of the prompt, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0946ce71-6285-432e-b011-9c2dc1ba7b8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_prompt = ChatPromptTemplate.from_template(\n",
    "\"\"\"\n",
    "    {history}\n",
    "    Human: {question}\n",
    "    AI:\n",
    "\"\"\"\n",
    ")\n",
    "chain = hist_prompt | llm | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6d088e51-e5eb-4143-b87d-b2be429eb864",
   "metadata": {},
   "outputs": [],
   "source": [
    "Conversation_history = \"\"\"\n",
    "Human: {question}\n",
    "AI: {response}\n",
    "\"\"\".format(question=QUESTION, response=response_to_initial_question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d99e34ad-5539-44dd-b080-3ad05efd2f01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was \"Tell me some use cases for reinforcement learning\"."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain.invoke({\"history\":Conversation_history, \"question\": FOLLOW_UP_QUESTION}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "045e5af6-55d6-4353-b3f6-3275c95db00a",
   "metadata": {},
   "source": [
    "**Bingo!**, so we now know how to create a chatbot using LLMs, we just need to keep the state/history of the conversation and pass it as context every time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eafd1694-0077-4aa8-bd01-e9f763ce36a3",
   "metadata": {},
   "source": [
    "## Now that we understand the concept of memory via adding history as a context, let's go back to our GPT Smart Search engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9787ffb6-2b11-4b03-92fc-9443cd1f2ab9",
   "metadata": {},
   "source": [
    "From Langchain website:\n",
    "    \n",
    "A memory system needs to support two basic actions: reading and writing. Recall that every chain defines some core execution logic that expects certain inputs. Some of these inputs come directly from the user, but some of these inputs can come from memory. A chain will interact with its memory system twice in a given run.\n",
    "\n",
    "    AFTER receiving the initial user inputs but BEFORE executing the core logic, a chain will READ from its memory system and augment the user inputs.\n",
    "    AFTER executing the core logic but BEFORE returning the answer, a chain will WRITE the inputs and outputs of the current run to memory, so that they can be referred to in future runs.\n",
    "    \n",
    "So this process adds delays to the response, but it is a necessary delay :)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f36e8f14-e566-4ae9-a7d4-6dee7f469dad",
   "metadata": {},
   "source": [
    "![image](https://python.langchain.com/assets/images/memory_diagram-0627c68230aa438f9b5419064d63cbbc.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ef9f459b-e8b8-40b9-a94d-80c079968594",
   "metadata": {},
   "outputs": [],
   "source": [
    "index1_name = \"cogsrch-index-files\"\n",
    "index2_name = \"cogsrch-index-csv\"\n",
    "index3_name = \"cogsrch-index-books\"\n",
    "indexes = [index1_name, index2_name, index3_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b01852c2-6192-496c-adff-4270f9380469",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize our custom retriever \n",
    "retriever = CustomAzureSearchRetriever(indexes=indexes, topK=10, reranker_threshold=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "633937e8-18e6-43f2-b4d5-fc36157a4d97",
   "metadata": {},
   "source": [
    "If you check closely in prompts.py, there is an optional variable in the `DOCSEARCH_PROMPT` called `history`. Now it is the time to use it. It is basically a place holder were we will inject the conversation in the prompt so the LLM is aware of it before it answers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "035fa6e6-226c-400f-a504-30255385f43b",
   "metadata": {},
   "source": [
    "**Now let's add memory to it:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3c8c9381-08d0-4808-9ab1-78156ca1be6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "store = {} # Our first memory will be a dictionary in memory\n",
    "\n",
    "# We have to define a custom function that takes a session_id and looks somewhere\n",
    "# (in this case in a dictionary in memory) for the conversation\n",
    "def get_session_history(session_id: str) -> BaseChatMessageHistory:\n",
    "    if session_id not in store:\n",
    "        store[session_id] = ChatMessageHistory()\n",
    "    return store[session_id]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "48ff51e1-2b1e-4c67-965d-1c2e2f55e005",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We use our original chain with the retriever but removing the StrOutputParser\n",
    "chain = (\n",
    "    {\n",
    "        \"context\": itemgetter(\"question\") | retriever, \n",
    "        \"question\": itemgetter(\"question\"),\n",
    "        \"history\": itemgetter(\"history\")\n",
    "    }\n",
    "    | DOCSEARCH_PROMPT\n",
    "    | llm\n",
    ")\n",
    "\n",
    "## Then we pass the above chain to another chain that adds memory to it\n",
    "\n",
    "output_parser = StrOutputParser()\n",
    "\n",
    "chain_with_history = RunnableWithMessageHistory(\n",
    "    chain,\n",
    "    get_session_history,\n",
    "    input_messages_key=\"question\",\n",
    "    history_messages_key=\"history\",\n",
    ") | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0e582915-243f-42cb-bb1e-c35a20ee0b9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is where we configure the session id\n",
    "config={\"configurable\": {\"session_id\": \"abc123\"}}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ff493b1-b133-4880-a040-e80f7460e7af",
   "metadata": {},
   "source": [
    "Notice below, that we are adding a `history` variable in the call. This variable will hold the chat historywithin the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d91a7ff4-6148-459d-917c-37302805dd09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning (RL) has various applications across different domains. Here are some specific use cases for reinforcement learning:\n",
       "\n",
       "1. **Routing Algorithms**: Reinforcement learning algorithms can be used for routing in networks. These algorithms learn and adapt routing table entries based on feedback from the environment to optimize network traffic flow and minimize delays<sup><a href=\"https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9905/9905015v1.pdf\" target=\"_blank\">[1]</a></sup>.\n",
       "\n",
       "2. **Personalized Music Recommendation**: RL can be applied to personalized music recommendation systems. By simulating the interaction process of listeners and continuously updating the recommendation model based on their preferences, RL can provide better song sequences that match listeners' preferences<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/\" target=\"_blank\">[2]</a></sup>.\n",
       "\n",
       "3. **Epidemic Modeling**: RL can be used in microscopic epidemic modeling to consider the consequences of individual decisions on the spread of diseases. By minimizing agents' cost functions and solving for optimal decisions, RL can predict disease spread and identify the need for external interventions<sup><a href=\"https://arxiv.org/pdf/2004.12959v1.pdf\" target=\"_blank\">[3]</a></sup>.\n",
       "\n",
       "4. **Adaptive Control**: Adaptive control systems can benefit from RL algorithms to improve decision-making processes based on experience. These systems are concerned with improving sequences of decisions in dynamic systems, and RL can enhance these algorithms<sup><a href=\"https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf\" target=\"_blank\">[4]</a></sup>.\n",
       "\n",
       "5. **Multi-Agent Systems**: RL can be applied in multi-agent systems to enable agents to learn and adapt their behaviors based on interactions with the environment and other agents. This can lead to more efficient and collaborative decision-making processes<sup><a href=\"https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0306/0306091v1.pdf\" target=\"_blank\">[5]</a></sup>.\n",
       "\n",
       "These examples demonstrate the versatility and effectiveness of reinforcement learning in various real-world applications."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke({\"question\": QUESTION}, config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "25dfc233-450f-4671-8f1c-0b446e46f048",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was about \"Tell me some use cases for reinforcement learning.\""
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke({\"question\": FOLLOW_UP_QUESTION},config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c67073c2-9a82-4e44-a9e2-48fe868c1634",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "You're welcome! If you have any more questions in the future, feel free to ask. Goodbye!"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke({\"question\": \"Thank you! Good bye\"},config=config))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87405173",
   "metadata": {},
   "source": [
    "## Using CosmosDB as persistent memory\n",
    "\n",
    "In previous cell we have added local RAM memory to our chatbot. However, it is not persistent, it gets deleted once the app user's session is terminated. It is necessary then to use a Database for persistent storage of each of the bot user conversations, not only for Analytics and Auditing, but also if we wish to provide recommendations in the future. \n",
    "\n",
    "Here we will store the conversation history into CosmosDB for future auditing purpose.\n",
    "We will use a class in LangChain use CosmosDBChatMessageHistory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d87cc7c6-5ef1-4492-b133-9f63a392e223",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the function to retrieve the conversation\n",
    "\n",
    "def get_session_history(session_id: str, user_id: str) -> CosmosDBChatMessageHistory:\n",
    "    cosmos = CosmosDBChatMessageHistory(\n",
    "        cosmos_endpoint=os.environ['AZURE_COSMOSDB_ENDPOINT'],\n",
    "        cosmos_database=os.environ['AZURE_COSMOSDB_NAME'],\n",
    "        cosmos_container=os.environ['AZURE_COSMOSDB_CONTAINER_NAME'],\n",
    "        connection_string=os.environ['AZURE_COMOSDB_CONNECTION_STRING'],\n",
    "        session_id=session_id,\n",
    "        user_id=user_id\n",
    "        )\n",
    "\n",
    "    # prepare the cosmosdb instance\n",
    "    cosmos.prepare_cosmos()\n",
    "    return cosmos\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "94f4179b-c1c7-49da-9c80-a42c275ed4d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain_with_history = RunnableWithMessageHistory(\n",
    "    chain,\n",
    "    get_session_history,\n",
    "    input_messages_key=\"question\",\n",
    "    history_messages_key=\"history\",\n",
    "    history_factory_config=[\n",
    "        ConfigurableFieldSpec(\n",
    "            id=\"user_id\",\n",
    "            annotation=str,\n",
    "            name=\"User ID\",\n",
    "            description=\"Unique identifier for the user.\",\n",
    "            default=\"\",\n",
    "            is_shared=True,\n",
    "        ),\n",
    "        ConfigurableFieldSpec(\n",
    "            id=\"session_id\",\n",
    "            annotation=str,\n",
    "            name=\"Session ID\",\n",
    "            description=\"Unique identifier for the conversation.\",\n",
    "            default=\"\",\n",
    "            is_shared=True,\n",
    "        ),\n",
    "    ],\n",
    ") | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8cf1f1f0-6e46-4136-9f33-4e46617b7d4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is where we configure the session id and user id\n",
    "random_session_id = \"session\"+ str(random.randint(1, 1000))\n",
    "ramdom_user_id = \"user\"+ str(random.randint(1, 1000))\n",
    "\n",
    "config={\"configurable\": {\"session_id\": random_session_id, \"user_id\": ramdom_user_id}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "0b20c00c-4098-4970-84e5-f71ea7615c65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'configurable': {'session_id': 'session996', 'user_id': 'user213'}}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7e3c32f4-f883-4045-91f9-ca317c2d01fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning (RL) has various applications across different domains. Some notable use cases for reinforcement learning include:\n",
       "\n",
       "1. **Routing Algorithms**: Reinforcement learning algorithms are increasingly used in routing applications. For example, ant-based algorithms and stigmergetic routing frameworks leverage reinforcement learning for learning routing table entries based on network conditions and performance metrics<sup><a href=\"https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0207/0207073v1.pdf\" target=\"_blank\">[1]</a></sup>.\n",
       "\n",
       "2. **Personalized Recommendation Systems**: Reinforcement learning can be applied to personalized recommendation systems, such as music recommendation algorithms. By simulating the interaction process with users and continuously updating models based on preferences, reinforcement learning can enhance the accuracy of song sequence recommendations<sup><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206183/\" target=\"_blank\">[2]</a></sup>.\n",
       "\n",
       "3. **Multi-Agent Systems**: Reinforcement learning is used in multi-agent systems for solving complex problems. In routing, for instance, RL algorithms help routers make optimal forwarding decisions based on reinforcement signals from the environment<sup><a href=\"https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9905/9905015v1.pdf\" target=\"_blank\">[3]</a></sup>.\n",
       "\n",
       "4. **Game Playing**: RL has been extensively applied in game playing scenarios. For example, in chess, reinforcement learning can be used to learn strategies by analyzing gameplay data and making decisions based on rewards and outcomes<sup><a href=\"https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0004/0004001v1.pdf\" target=\"_blank\">[4]</a></sup>.\n",
       "\n",
       "5. **Adaptive Control**: Adaptive control systems leverage reinforcement learning algorithms to improve decision-making processes based on experience. These systems are used in dynamic environments where actions and states are vectors, and the system dynamics are smooth, allowing for adaptive learning and control<sup><a href=\"https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf\" target=\"_blank\">[5]</a></sup>.\n",
       "\n",
       "These examples showcase the versatility and effectiveness of reinforcement learning in solving a wide range of problems across different fields."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke({\"question\": QUESTION}, config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7e29643b-a531-4117-8e85-9c88a625cf02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was about the various use cases of reinforcement learning."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke({\"question\": FOLLOW_UP_QUESTION},config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "50146f05-5ef6-484f-a8ec-9631643054f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "We discussed various use cases of reinforcement learning and then you requested a one-line summary of our conversation."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"Can you tell me a one line summary of our conversation?\"},\n",
    "    config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "8bc02369-904c-4063-93e1-fff24fe6a3ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error code: 400 - {'error': {'message': \"This model's maximum context length is 16385 tokens. However, you requested 17216 tokens (15716 in the messages, 1500 in the completion). Please reduce the length of the messages or completion.\", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"Thank you very much!\"},\n",
    "    config=config))\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "87d60faa-1446-4c07-8970-0f9712c33b2f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "I provided a one-line summary in response to your request for a concise overview of our conversation."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"I do have one more question, why did you give me a one line summary?\"},\n",
    "    config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "cfe748aa-6116-4a7a-97e6-f1c680dd23ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "I provided a one-line summary as you specifically asked for a one-line summary of our conversation."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"why not 2?\"},\n",
    "    config=config))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdc5ac98",
   "metadata": {},
   "source": [
    "#### Let's check our Azure CosmosDB to see the whole conversation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e30694-ae2a-47bb-a5c7-db51ecdbba1e",
   "metadata": {},
   "source": [
    "![CosmosDB Memory](./images/cosmos-chathistory.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6789cada-23a3-451a-a91a-0906ceb0bd14",
   "metadata": {},
   "source": [
    "# Summary\n",
    "##### Adding memory to our application allows the user to have a conversation, however this feature is not something that comes with the LLM, but instead, memory is something that we must provide to the LLM in form of context of the question.\n",
    "\n",
    "We added persitent memory using CosmosDB.\n",
    "\n",
    "We also can notice that the current chain that we are using is smart, but not that much. Although we have given memory to it, it searches for similar docs everytime, regardless of the input. This doesn't seem efficient, but regardless, we are very close to finish our first RAG-talk to your data Bot.\n",
    "\n",
    "\n",
    "## <u>Important Note</u>:<br>\n",
    "As we proceed, while all the code will remain compatible with GPT-3.5 models (1106 or newer), we highly recommend transitioning to GPT-4. Here's why:\n",
    "\n",
    "**GPT-3.5-Turbo** can be likened to a 7-year-old child. You can provide it with concise instructions, but it struggles sometimes to follow them accurately (not too reliable). Additionally, its limited \"memory\" (token context) can make sustained conversations challenging. Its response are also simple not deep.\n",
    "\n",
    "**GPT-4-Turbo** exhibits the capabilities of a 10-12-year-old child. It possesses enhanced reasoning skills, consistently adheres to instructions and its answers are beter. It has extended memory retention (larger context size) for instructions, and it excels at following them. Its responses are deep and thorough.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c629ebf4-aced-45b7-a6a2-315810d37d48",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "We know now how to do a Smart Search Engine that can power a chatbot!! great!\n",
    "\n",
    "In the next notebook 6, we are going to build our first RAG bot. In order to do this we will introduce the concept of Agents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f53a8f7a-5e28-4d5f-9a33-0a3be0536b0f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}