-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RPG GAME]FunctionAgent Version #236
Open
Southpika
wants to merge
18
commits into
PaddlePaddle:develop
Choose a base branch
from
Southpika:app
base: develop
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 10 commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
5b49622
init rpg game
Southpika 26a5871
temp save
Southpika 61c9488
temp save
Southpika 8aa739b
add function agent rpg
Southpika c36871e
fix bug
Southpika ab95841
Merge remote-tracking branch 'upstream/develop' into app
Southpika bb2a21d
renew readme and fix link
Southpika e539854
reformat
Southpika b67a176
reformat
Southpika 9f66a15
fix des
Southpika 9a8c1db
fix des
Southpika 62ce209
Merge branch 'develop' into app
shiyutang 814117b
Merge remote-tracking branch 'upstream/develop' into app
Southpika d40b86c
Merge remote-tracking branch 'origin/app' into app
Southpika f76c74f
Merge remote-tracking branch 'upstream/develop' into app
Southpika dd0218f
add tool choice
Southpika 160f0f9
Merge remote-tracking branch 'upstream/develop' into app
Southpika 6a82fa8
update to newest file
Southpika File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
import argparse | ||
import os | ||
|
||
from erniebot_agent.agents import FunctionAgent | ||
from erniebot_agent.chat_models import ERNIEBot | ||
from erniebot_agent.tools import RemoteToolkit | ||
|
||
|
||
def parse_args(): | ||
parser = argparse.ArgumentParser(prog="erniebot-RPG") | ||
parser.add_argument("--access-token", type=str, default=None, help="Access token to use.") | ||
parser.add_argument("--model", type=str, default="ernie-3.5", help="Model name") | ||
return parser.parse_args() | ||
|
||
|
||
if __name__ == "__main__": | ||
os.environ["EB_AGENT_LOGGING_LEVEL"] = "info" | ||
args = parse_args() | ||
|
||
if os.getenv("EB_AGENT_ACCESS_TOKEN") is None and args.access_token is None: | ||
raise RuntimeError( | ||
"Please set EB_AGENT_ACCESS_TOKEN in environment variables" | ||
"or parse it in command line by --access-token." | ||
) | ||
|
||
if args.access_token is not None: | ||
access_token = args.access_token | ||
elif os.getenv("EB_AGENT_ACCESS_TOKEN") is not None: | ||
access_token = os.getenv("EB_AGENT_ACCESS_TOKEN") | ||
|
||
llm = ERNIEBot( | ||
model=args.model, api_type="aistudio", access_token=access_token, enable_multi_step_tool_call=True | ||
) | ||
tool = RemoteToolkit.from_aistudio("texttospeech").get_tools()[0] | ||
agent = FunctionAgent( | ||
llm=llm, | ||
tools=[tool], | ||
) | ||
agent.launch_gradio_demo() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,84 @@ | ||
# RPGGameAgent | ||
|
||
## 介绍 | ||
|
||
RPGGameAgent是一个基于Agent完成的文字类角色扮演的游戏,用户可以通过指定游戏脚本以及角色与Agent进行交互游戏,Agent将通过互动式的方式为玩家提供一个基于游戏剧情的在线RPG游戏体验。在互动过程中,Agent主要采用文心模型生成场景以及游戏发展的选择,并结合文生图工具生成场景图片,以丰富游戏体验。 | ||
|
||
目前该Agent提供两种方式进行 | ||
|
||
* 基于FunctionAgent,通过instruction来进行触发工具(暂不稳定,待tool choice上线)。 | ||
* 基于Prompt通过ToolFormat语句运行工具实现Agent。 | ||
|
||
## 如何开始 | ||
|
||
通过bash运行:通过执行脚本启动RPGGameAgent,并指定模型、剧情和访问令牌等参数。 | ||
|
||
```bash | ||
export EB_AGENT_LOGGING_LEVEL='info' | ||
python rpg_game_agent.py --access-token YOUR_ACCESS_TOKEN --game 射雕英雄传 --model ernie-3.5 | ||
``` | ||
|
||
## 通过FunctionAgent+Instruction实现 | ||
通过Instruction指示通过FunctionAgent如何调用工具,实现GameAgent的结果生成,先通过ChatStory工具生成最终的互动结果,然后调用ImageGenerateTool工具根据场景生成图片,最终截取ChatStory工具生成的结果作为最终输出。 | ||
|
||
### 关键步骤 | ||
1. 工具准备: | ||
需要一个ChatStory工具用来生成故事情节以及一个ImageGenerateTool工具生成场景图片 | ||
2. 通过以下instruction指示Agent来达到触发 | ||
|
||
```markdown | ||
你是《{SCRIPT}》沉浸式图文RPG场景助手,能够生成图文剧情。\ | ||
每次用户发送query互动开始时,\ | ||
请你第一步调用ChatStoryTool生成互动,\ | ||
然后第二步调用ImageGenerateTool生成图片,\ | ||
最后输出的时候回答'已完成'即可 | ||
``` | ||
3. 直接调用Agent,获得包括<场景描述>、<场景图片>和<选择>的互动结果。 | ||
|
||
## 通过ToolFormat实现手动编排Agent | ||
|
||
除了基于FunctionCall的FunctionAgent实现以外,同时我们也支持ToolFormat:以手动编排的方式通过Prompt激活Agent。 | ||
|
||
即:通过事先定义Agent想要操作的步骤,然后通过指定的tool识别范式来运行tool。 | ||
|
||
### 关键步骤 | ||
|
||
1. Planning:通过Prompt事先指定相应的指令作为Plan,在Plan中具体指定哪一步要做什么以及工具调用。 | ||
|
||
```python | ||
INSTRUCTION = """你的指令是为我提供一个基于《{SCRIPT}》剧情的在线RPG游戏体验。\ | ||
在这个游戏中,玩家将扮演《{SCRIPT}》剧情关键角色,你可以自行决定玩家的角色。\ | ||
游戏情景将基于《{SCRIPT}》剧情。这个游戏的玩法是互动式的,并遵循以下特定格式: | ||
|
||
<场景描述>:根据玩家的选择,故事情节将按照《{SCRIPT}》剧情的线索发展。你将描述角色所处的环境和情况。场景描述不少于50字。 | ||
|
||
<场景图片>:对于每个场景,你将创造一个概括该情况的图像。在这个步骤你需要调用画图工具ImageGenerationTool并按json格式输出相应调用详情。\ | ||
ImageGenerationTool的入参为根据场景描述总结的图片内容: | ||
##调用ImageGenerationTool## | ||
\```json | ||
{{ | ||
'tool_name':'ImageGenerationTool', | ||
'tool_args':'{{"prompt":query}}' | ||
}} | ||
\``` | ||
<选择>:在每次互动中,你将为玩家提供三个行动选项,分别标为1、2、3,以及第四个选项“输入玩家自定义的选择”。故事情节将根据玩家选择的行动进展。 | ||
如果一个选择不是直接来自《{SCRIPT}》剧情,你将创造性地适应故事,最终引导它回归原始情节。 | ||
|
||
整个故事将围绕《{SCRIPT}》丰富而复杂的世界展开。每次互动必须包括<场景描述>、<场景图片>和<选择>。所有内容将以中文呈现。 | ||
你的重点将仅仅放在提供场景描述,场景图片和选择上,不包含其他游戏指导。场景尽量不要重复,要丰富一些。 | ||
|
||
当我说游戏开始的时候,开始游戏。每次只要输出【一组】互动,【不要自己生成互动】。""" | ||
``` | ||
|
||
2. Execute Tool:通过在流式输出的过程中遇到ToolFormat的部分(在此例子中为 \```json\```),开始异步执行相应的工具。 | ||
|
||
需要在INSTRUCTION中讲述清楚具体的相关参数,格式为json,tool_name为调用工具名称,tool_args为工具所有入参(json)。 | ||
|
||
``` | ||
{ | ||
'tool_name':'ImageGenerationTool', | ||
'tool_args':'{{"prompt":query}}' | ||
} | ||
``` | ||
|
||
3. 等待工具执行完成,获得包括<场景描述>、<场景图片>和<选择>的互动结果,以及文生图工具生成结果。 |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
from typing import Type | ||
|
||
from pydantic import Field | ||
|
||
from erniebot_agent.agents import Agent | ||
from erniebot_agent.tools.base import Tool, ToolParameterView | ||
|
||
|
||
class ChatStoryToolInputView(ToolParameterView): | ||
query: str = Field(description="用户的指令") | ||
|
||
|
||
class ChatStoryToolOutputView(ToolParameterView): | ||
return_story: str = Field(description="生成的包括<场景描述>、<场景图片>和<选择>的互动内容") | ||
|
||
|
||
class ChatStoryTool(Tool): | ||
description: str = "结合用户的选择、{GAME}背景故事以及玩家角色,按要求生成接下来的故事情节" | ||
input_type: Type[ToolParameterView] = ChatStoryToolInputView | ||
ouptut_type: Type[ToolParameterView] = ChatStoryToolOutputView | ||
|
||
def __init__(self, agent, game: str) -> None: | ||
super().__init__() | ||
self.agent: Agent = agent | ||
self.description = self.description.format(GAME=game) | ||
|
||
async def __call__(self, query: str) -> str: | ||
response = await self.agent.run(query) | ||
return {"return_story": response.text} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
from typing import Type | ||
|
||
from pydantic import Field | ||
|
||
from erniebot_agent.file import GlobalFileManagerHandler | ||
from erniebot_agent.tools.base import Tool, ToolParameterView | ||
|
||
|
||
class ImageGenerateToolInputView(ToolParameterView): | ||
query: str = Field(description="当前场景图片的描述") | ||
|
||
|
||
class ImageGenerateToolOutputView(ToolParameterView): | ||
output_image: str = Field(description="返回的图片文件,格式为file-xxxx, 不包括<file></file>") | ||
|
||
|
||
class ImageGenerateTool(Tool): | ||
description: str = "根据用户的选择以及当前的互动内容,按照场景图片部分的内容生成图片。" | ||
input_type: Type[ToolParameterView] = ImageGenerateToolInputView | ||
otuput_story: Type[ToolParameterView] = ImageGenerateToolOutputView | ||
|
||
async def __call__(self, query: str) -> str: | ||
# output_dir = query | ||
file_manager = await GlobalFileManagerHandler().get() | ||
# self.file_manager.create_file_from_bytes() | ||
file = await file_manager.create_file_from_path("/Users/tanzhehao/Desktop/git.png") # for mimic | ||
return {"output_image": file.id} |
212 changes: 212 additions & 0 deletions
212
erniebot-agent/applications/rpg_game/rpg_game_function.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,212 @@ | ||
import argparse | ||
import base64 | ||
import os | ||
import tempfile | ||
from typing import Any | ||
|
||
from chat_story_tool import ChatStoryTool | ||
from img_gen_tool import ImageGenerateTool | ||
|
||
from erniebot_agent.agents import FunctionAgent | ||
from erniebot_agent.agents.base import BaseAgent | ||
from erniebot_agent.chat_models.erniebot import ERNIEBot | ||
from erniebot_agent.memory import SlidingWindowMemory | ||
from erniebot_agent.memory.messages import AIMessage, SystemMessage | ||
from erniebot_agent.utils.common import get_file_type | ||
from erniebot_agent.utils.html_format import IMAGE_HTML | ||
|
||
os.environ["EB_AGENT_LOGGING_LEVEL"] = "info" | ||
|
||
INSTRUCTION = """你的指令是为我提供一个基于《{SCRIPT}》剧情的在线RPG游戏体验。在这个游戏中,玩家将扮演《{SCRIPT}》剧情关键角色,游戏情景将基于《{SCRIPT}》剧情。\ | ||
这个游戏的玩法是互动式的,并遵循以下特定格式: | ||
|
||
<场景描述>:根据玩家的选择,故事情节将按照《{SCRIPT}》剧情的线索发展。你将描述角色所处的环境和情况,不得少于三句话。 | ||
|
||
<场景图片>:对于每个场景,你将创造一个概括该场景情况的图像。 | ||
|
||
<选择>:在每次互动中,你将为玩家提供三个行动选项,分别标为1、2、3,以及第四个选项“输入玩家自定义的选择”。故事情节将根据玩家选择的行动进展。\ | ||
如果一个选择不是直接来自《{SCRIPT}》剧情,你将创造性地适应故事,最终引导它回归原始情节。 | ||
|
||
整个故事将围绕《{SCRIPT}》丰富而复杂的世界展开。每次互动必须包括<场景描述>、<场景图片>和<选择>。所有内容将以中文呈现。\ | ||
你的重点将仅仅放在提供场景描述,场景图片和选择上,不包含其他游戏指导。场景尽量不要重复,要丰富一些。 | ||
|
||
当我说游戏开始的时候,开始游戏。每次只要输出一组互动,不要自己生成互动。""" | ||
|
||
|
||
def parse_args(): | ||
parser = argparse.ArgumentParser(prog="erniebot-RPG") | ||
parser.add_argument("--access-token", type=str, default=None, help="Access token to use.") | ||
parser.add_argument("--game", type=str, default="射雕英雄传", help="Story name") | ||
parser.add_argument("--model", type=str, default="ernie-3.5", help="Model name") | ||
return parser.parse_args() | ||
|
||
|
||
args = parse_args() | ||
|
||
|
||
class GameAgent(FunctionAgent): | ||
def launch_gradio_demo(self: BaseAgent, **launch_kwargs: Any): | ||
try: | ||
import gradio as gr # type: ignore | ||
except ImportError: | ||
raise ImportError( | ||
"Could not import gradio, which is required for `launch_gradio_demo()`." | ||
" Please run `pip install erniebot-agent[gradio]` to install the optional dependencies." | ||
) from None | ||
|
||
raw_messages = [] | ||
|
||
def _messages_to_dicts(messages): | ||
return [message.to_dict() for message in messages] | ||
|
||
def _pre_chat(text, history): | ||
history.append([text, None]) | ||
return history, gr.update(value="", interactive=False), gr.update(interactive=False) | ||
|
||
async def _chat(history): | ||
prompt = history[-1][0] | ||
response = await self.run(prompt) | ||
self.memory.msg_manager.messages[-1] = AIMessage( | ||
eval(response.chat_history[2].content)["return_story"] | ||
) | ||
raw_messages.extend(response.chat_history) | ||
if len(response.chat_history) >= 3: | ||
output_result = eval(response.chat_history[2].content)["return_story"] | ||
else: | ||
output_result = response.text | ||
if response.steps and response.steps[-1].output_files: | ||
# If there is a file output in the last round, then we need to show it. | ||
output_file = response.steps[-1].output_files[-1] | ||
file_content = await output_file.read_contents() | ||
if get_file_type(output_file.filename) == "image": | ||
# If it is a image, we can display it in the same chat page. | ||
base64_encoded = base64.b64encode(file_content).decode("utf-8") | ||
output_result = eval(response.chat_history[2].content)[ | ||
"return_story" | ||
] + IMAGE_HTML.format(BASE64_ENCODED=base64_encoded) | ||
history[-1][1] = output_result | ||
return ( | ||
history, | ||
_messages_to_dicts(raw_messages), | ||
_messages_to_dicts(self.memory.get_messages()), | ||
) | ||
|
||
def _post_chat(): | ||
return gr.update(interactive=True), gr.update(interactive=True) | ||
|
||
def _clear(): | ||
raw_messages.clear() | ||
self.reset_memory() | ||
return None, None, None, None | ||
|
||
with gr.Blocks( | ||
title="ERNIE Bot Agent Demo", theme=gr.themes.Soft(spacing_size="sm", text_size="md") | ||
) as demo: | ||
with gr.Column(): | ||
with gr.Tab(label="Chat"): | ||
chatbot = gr.Chatbot( | ||
label="Chat history", | ||
latex_delimiters=[ | ||
{"left": "$$", "right": "$$", "display": True}, | ||
{"left": "$", "right": "$", "display": False}, | ||
], | ||
bubble_full_width=False, | ||
height=700, | ||
) | ||
|
||
with gr.Row(): | ||
prompt_textbox = gr.Textbox( | ||
label="Prompt", placeholder="Write a prompt here...", scale=15 | ||
) | ||
submit_button = gr.Button("Submit", min_width=150) | ||
clear_button = gr.Button("Clear", min_width=100) | ||
|
||
with gr.Accordion("Tools", open=False): | ||
attached_tools = self.get_tools() | ||
tool_descriptions = [tool.function_call_schema() for tool in attached_tools] | ||
gr.JSON(value=tool_descriptions) | ||
with gr.Accordion("Raw messages", open=False): | ||
all_messages_json = gr.JSON(label="All messages") | ||
agent_memory_json = gr.JSON(label="Messges in memory") | ||
|
||
prompt_textbox.submit( | ||
_pre_chat, | ||
inputs=[prompt_textbox, chatbot], | ||
outputs=[chatbot, prompt_textbox, submit_button], | ||
).then( | ||
_chat, | ||
inputs=[chatbot], | ||
outputs=[ | ||
chatbot, | ||
all_messages_json, | ||
agent_memory_json, | ||
], | ||
).then( | ||
_post_chat, outputs=[prompt_textbox, submit_button] | ||
) | ||
submit_button.click( | ||
_pre_chat, | ||
inputs=[prompt_textbox, chatbot], | ||
outputs=[chatbot, prompt_textbox, submit_button], | ||
).then( | ||
_chat, | ||
inputs=[chatbot], | ||
outputs=[ | ||
chatbot, | ||
all_messages_json, | ||
agent_memory_json, | ||
], | ||
).then( | ||
_post_chat, outputs=[prompt_textbox, submit_button] | ||
) | ||
clear_button.click( | ||
_clear, | ||
outputs=[ | ||
chatbot, | ||
prompt_textbox, | ||
all_messages_json, | ||
agent_memory_json, | ||
], | ||
) | ||
with tempfile.TemporaryDirectory() as td: | ||
if "allowed_paths" in launch_kwargs: | ||
if not isinstance(launch_kwargs["allowed_paths"], list): | ||
raise TypeError("`allowed_paths` must be a list") | ||
allowed_paths = launch_kwargs["allowed_paths"] + [td] | ||
launch_kwargs.pop("allowed_paths") | ||
else: | ||
allowed_paths = [td] | ||
demo.launch(allowed_paths=allowed_paths, **launch_kwargs) | ||
|
||
|
||
def creates_story_tool(): | ||
memory = SlidingWindowMemory(max_round=2) | ||
llm = ERNIEBot(model=args.model, api_type="aistudio") | ||
agent = FunctionAgent( | ||
llm=llm, tools=[], system_message=SystemMessage(INSTRUCTION.format(SCRIPT=args.game)), memory=memory | ||
) | ||
tool = ChatStoryTool(agent, game=args.game) | ||
return tool | ||
|
||
|
||
def main(): | ||
img_tool = ImageGenerateTool() | ||
story_tool = creates_story_tool() | ||
SYSTEM_MESSAGE = "你是《{SCRIPT}》沉浸式图文RPG场景助手,能够生成图文剧情。\ | ||
每次用户发送query或者输入数字开始互动时,\ | ||
请你先调用ChatStoryTool生成互动,然后调用ImageGenerateTool生成图片,\ | ||
最后输出的时候回答'已完成'即可。" | ||
|
||
llm = ERNIEBot(model=args.model, api_type="aistudio", enable_multi_step_tool_call=True) | ||
memory = SlidingWindowMemory(max_round=2) | ||
agent = GameAgent( | ||
llm=llm, | ||
tools=[story_tool, img_tool], | ||
memory=memory, | ||
system_message=SystemMessage(SYSTEM_MESSAGE.format(SCRIPT=args.game)), | ||
) | ||
agent.launch_gradio_demo() | ||
|
||
|
||
if __name__ == "__main__": | ||
main() |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
因具体_chat函数部分有修改,因此不能直接继承launch_gradio_demo方法