feat(CodeActAgent): Support Agent-User Interaction during Task Execution and the Full Integration of CodeActAgent

[xingyaoww]

Dependency:

• feat(sandbox): Candidate Implementation of Sandbox Plugin to Support Jupyter #1255
• Add SWE-agent tools as sandbox plugins #1305
• fix(sandbox): mount_dir for ssh_box #1333
• Default for sandbox to use the host network #1334
• Get RUN_AS_DEVIN working with app sandbox #1426
• Revert ssh box implemetation, fix multi-line command issues and add unit tests #1460

For general OpenDevin components:

• Add IPythonRunCellAction that allows the agent to use Jupyter plugin introduced in feat(sandbox): Candidate Implementation of Sandbox Plugin to Support Jupyter #1255 for task solving.
• AgentTalkAction that allows the agent to communicate with the user (either to communicate results OR ask for clarification)
• Agent-User Interaction: Now allow agent's wait for user's response before proceed to task completion.
• Added __str__ method to some of the actions to get the printed in a readable way
• Add thought attribute to most Action (if applicable)

For CodeActAgent:

• Allow CodeActAgent to do two things for environmental interactions: (1) use ipython AND (2) use bash
• Tweak the prompts of CodeActAgent to ensure its generality.
• The ability to use SWE-Agent's tool for task solving (still need more extensive testing).

For integration tests:

• Added integration tests for CodeActAgent.
• Support STDIN mock for "user responses" in integration test

---

The best thing about CodeActAgent is that we have existing data and open models (e.g., CodeActAgent, CodeQwen) trained on that data, which could means we can have a usable local model that can run on laptop.

Below demo is completed with CodeQwen, here's my config.toml:

LLM_MODEL="ollama/codeqwen"
LLM_API_KEY="ollama"
LLM_BASE_URL="http://localhost:11434"
LLM_EMBEDDING_MODEL="local"
WORKSPACE_BASE="./workspace"

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 71.24183% with 44 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (main@62e4fb4). Click here to learn what that means.

Files	Patch %	Lines
opendevin/action/bash.py	62.16%	14 Missing ⚠️
opendevin/controller/agent_controller.py	62.50%	12 Missing ⚠️
opendevin/server/agent/agent.py	0.00%	8 Missing ⚠️
agenthub/codeact_agent/codeact_agent.py	76.92%	6 Missing ⚠️
opendevin/action/agent.py	85.71%	2 Missing ⚠️
opendevin/observation/run.py	80.00%	2 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1290   +/-   ##
=======================================
  Coverage        ?   60.66%           
=======================================
  Files           ?       84           
  Lines           ?     3605           
  Branches        ?        0           
=======================================
  Hits            ?     2187           
  Misses          ?     1418           
  Partials        ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rbren]

There's currently a status indicator on the frontend, which shows agent status--so it's there somewhere!

[rbren]

@xingyaoww I implemented the user_message action here: https://github.com/OpenDevin/OpenDevin/commits/rb/codeact/

But I can't find a way to contribute it to your PR...

Edit: this is as far as I got

https://github.com/xingyaoww/OpenDevin/pull/1/files

[rbren]

@xingyaoww once my PR is merged into yours, this LGTM!

[xingyaoww]

@rbren merged! this should be ready to go!

[rbren]

🎉 glad to have this in!

[xingyaoww]

@rbren Thanks!!!

Next steps:

• Frontend!
  • Implement Jupyter Frontend #1363
  • Switch to Jupyter Tab when a python execution request comes in (once Implement Jupyter Frontend #1363 is merged)
  • frontend: Add [continue] and [finish] button for user to click when agent is awaiting user input #1487
• Evaluation: get SWE-Bench score for CodeActAgent!