[feat] Integrate BrowserGym #1452

frankxu2004 · 2024-04-29T15:16:37Z

Currently the browser is implemented with raw playwright. Ideally we can reuse existing work BrowserGym to enable more interactions with a browser other than just goto url. This first PR aims at replacing the current browser action with BrowserGym integration. In the future we can easily add more actions supported by BrowserGym to enable agent interaction with the web.

In the proposed setup, we need to start a new backend server browser_api_server.py as an interface for all browser-related interactions. In the future, all browser-related actions will just be HTTP API calls to this server.

I would love suggestions from core maintainers on how to best handle this server's lifecycle with the main backend server.

Right now, to run it, run poetry run python opendevin/browser_server/browser_api_server.py along side with the main backend server. Maybe we could add this line to Makefile?

cc @neubig @xingyaoww

Fixes #1384

…tegrate_browsergym

neubig · 2024-04-29T15:25:12Z

Awesome thanks a bunch @frankxu2004 ! We'll take a look.

frankxu2004 · 2024-04-29T16:13:02Z

An example interaction:

==============
STEP 0

12:01:15 - PLAN
browse www.google.com
12:01:17 - ACTION
BrowseURLAction(url='www.google.com', action='browse')
12:01:18 - OBSERVATION
Error: Protocol error (Page.navigate): Cannot navigate to invalid URL
=========================== logs ===========================
navigating to "/projects/ogma3/fangzhex/OpenDevinwww.google.com", waiting until "load"
============================================================


==============
STEP 1

12:01:18 - PLAN
browse www.google.com
12:01:20 - ACTION
AgentThinkAction(thought='I need to double check the URL format before browsing again.', action='think')


==============
STEP 2

12:01:20 - PLAN
browse www.google.com
12:01:21 - ACTION
BrowseURLAction(url='https://www.google.com', action='browse')
12:01:24 - OBSERVATION
[ About ](https://about.google/?fg=1&utm_source=google-
US&utm_medium=referral&utm_campaign=hp-header) [ Store
](https://store.google.com/US?utm_source=hp_header&utm_medium=google_ooo&utm_campaign=GS100042&hl=en-
US)

[ Gmail ](https://mail.google.com/mail/&ogbl)

[ Images ](https://www.google.com/imghp?hl=en&ogbl)

[ ](https://www.google.com/intl/en/about/products)

[ Sign in
](https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=GAZAmgQ)

![Google](/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png)

Choose what you’re giving feedback on

* * * *

See more

Delete

* * Delete 

* Report inappropriate predictions 

I'm Feeling Curious

I'm Feeling Hungry

I'm Feeling Adventurous

I'm Feeling Playful

I'm Feeling Stellar

I'm Feeling Doodley

I'm Feeling Trendy

I'm Feeling Artistic

I'm Feeling Funny

[ Advertising ](https://www.google.com/intl/en_us/ads/?subid=ww-ww-et-g-awa-a-
g_hpafoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpafooter&fg=1)
[ Business ](https://www.google.com/services/?subid=ww-ww-et-g-awa-a-
g_hpbfoot1_1!o2&utm_source=google.com&utm_medium=referral&utm_campaign=google_hpbfooter&fg=1)
[ How Search works ](https://google.com/search/howsearchworks/?fg=1)

[
![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABUAAAAYCAMAAAAiV0Z6AAAAPFBMVEVLoEN0wU6CzFKCzFKCzFKCzFKCzFJSo0MSczNDmkCCzFJPoUMTczNdr0gmgziCzFITczMTczMTczMTczPh00jOAAAAFHRSTlPF/+bIsms8Ad///hX+//5/tXw7aMEAx10AAACaSURBVHgBbc4HDoRQCATQ33tbvf9dF9QxaCT9UQaltLHOh/golXKhMs5Xqa0xU1lyoa2fXFyQOsDG38qsLy4TaV+sFislovyhPzLJJrBu6eQOtpW0LjbJkzTuTDLRVNKa3uxJI+VdiRqXSeu6GW+Qxi29eLIi8H7EsYrT42BD+mQtNO5JMjRuC4lSY8V4hsLX0egGijvUSEP9AbylEsOkeCgWAAAAAElFTkSuQmCC)
Our third decade of climate action: join us
](https://sustainability.google/?utm_source=googlehpfooter&utm_medium=housepromos&utm_campaign=bottom-
footer&utm_content=)

[ Privacy ](https://policies.google.com/privacy?hl=en&fg=1) [ Terms
](https://policies.google.com/terms?hl=en&fg=1)

Settings

[ Search settings ](https://www.google.com/preferences?hl=en&fg=1)

[ Advanced search ](/advanced_search?hl=en&fg=1)

[ Your data in Search
](/history/privacyadvisor/search/unauth?utm_source=googlemenu&fg=1&cctld=com)

[ Search history ](/history/optout?hl=en&fg=1)

[ Search help
](https://support.google.com/websearch/?p=ws_results_help&hl=en&fg=1)

Send feedback

Dark theme: Off

Google apps




==============
STEP 3

12:01:24 - PLAN
browse www.google.com
12:01:27 - ACTION
AgentFinishAction(outputs={}, action='finish')
12:01:27 - opendevin:INFO: agent_controller.py:226
INFO
AgentFinishAction(outputs={}, action='finish')
12:01:27 - opendevin:INFO: agent_controller.py:107 - Task finished by agent

neubig

Thanks @frankxu2004 ! First a few high-level questions:

how is browser_server run? Does the user need to run it before starting OpenDevin? I didn't see any place where the server is started.
and actually do we need to stand up a flask server or is there an alternative way to communicate with BrowserGym? The reason why I ask is because having a separate server seems to add some complexity, and it'd be nice if we didn't have to make things too complex.

rbren · 2024-04-29T18:24:08Z

Agreed--I'm worried about the extra overhead here to run a separate server. Also curious what's going on under the hood with gymnasium--is it running a docker container?

frankxu2004 · 2024-04-29T21:23:01Z

Thanks for the comments!

@neubig

Right now, the server would be run along side with the main backend on host system by poetry run python opendevin/browser_server/browser_api_server.py. It can be moved inside sandbox docker for better security but we need to figure out how to communicate between main backend and the sandboxed server.
The main issue with this is that, in BrowserGym, Playwright is used with sync API, and as a result cannot be used inside an asyncio loop. The current main backend fastapi server is based on asyncio so putting browsergym instantiation code in the backend will result in playwright._impl._api_types.Error: It looks like you are using Playwright Sync API inside the asyncio loop.

Potential solution is to refactor BrowserGym using async playwright API, but as BrowserGym is quite complex this route might be requiring a lot of work. Another option is to explore ways of having a sync thread for BrowserGym, but I might need a bit more help.

@rbren

going on under the hood with gymnasium--is it running a docker container?
Underlying code for BrowserGym is just creating a Playwright sync_api context with Chromium, and maintain that context and expose high-level APIs for both useful observations for agents to ingest, and provide an execution platform where generated actions could execute on.

Hope it helps! Right now I use a single-threaded Flask server just for this sync vs async incompatibility.

neubig · 2024-04-30T02:58:28Z

Hey @frankxu2004 , could you try this? https://pypi.org/project/nest-asyncio/

frankxu2004 · 2024-04-30T03:36:12Z

Hey @frankxu2004 , could you try this? https://pypi.org/project/nest-asyncio/

Thanks for the suggestion. I tried applying this patch in the main backend but it seems like the asyncio server we used uvicorn cannot be patched

ValueError: Can't patch loop of type <class 'uvloop.Loop'>

From the package readme: Only event loops from asyncio can be patched; Loops from other projects, such as uvloop or quamash, generally can’t be patched.

Right now I am investigating replacing the separate Flask server with multiprocessing.Process and multiprocessing.Queue, hopefully eliminating the need for HTTP server. This way we can more easily handle the BrowserEnv's life cycle with AgentController

xingyaoww · 2024-04-30T03:45:45Z

I take a look and it seems it is challenging to make async & sync live well with each other. Starting the browser in a separate thread/process and communicate with it via Queue could work so that we don't need to start this as a separate server. Look forward to the implementation though - pls let me know if you need any help!!

…tegrate_browsergym

frankxu2004 · 2024-04-30T06:19:57Z

Updated the implementation cc @neubig @rbren @xingyaoww

No flask server needed. A process is created when AgentController is initialized. Question: how to handle graceful shutdown of the Process created? I am not super familiar with the codebase, but it seems that atexit is not used for AgentController.

xingyaoww

Thanks for the PR! This is great efforts and i can see it could saves us tremendous time from implementing each web action & setting up evaluation for web browsing.

opendevin/action/browse.py

opendevin/browser_server/browser_process.py

neubig

I agree, this is exciting, thanks @frankxu2004 !

I'll open up follow-up issues for:

Expanding our browser action space
Implementing a BrowserAgent that only does browsing
Evaluate it against WebArena using BrowserGym to make sure it works

opendevin/action/browse.py

opendevin/browser_server/browser_process.py

rbren

This is coming along great!

rbren · 2024-04-30T17:03:59Z

opendevin/action/browse.py

                )
+
+            text_content = html2text.html2text(flatten_dom_to_str(obs['dom_object']))


Might be good to make this conversion optional--I could see some agents wanting the raw HTML. Also, it's not necessarily an HTML URL! Maybe we add a format param which could be one of html, text, or screenshot?

Previous impl. of this action is using the text content of the playwright page as content, so now I am keeping this conversion as default. I agree we should add configurable observation here, but up to the agent implementation. As a result, I modified the BrowserObservation towards a more complete one, having all kinds of observation types if the agent wants to use them.

…tegrate_browsergym

… handling of browseurlaction

frankxu2004 · 2024-05-01T13:17:45Z

Updated html and text handling. Default to using text as content for this PR, to maintain the same observation as previous versions

codecov-commenter · 2024-05-01T13:24:42Z

Codecov Report

Attention: Patch coverage is 53.06122% with 46 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (main@0d77f49). Click here to learn what that means.

Files	Patch %	Lines
opendevin/browser/browser_env.py	43.42%	43 Missing ⚠️
opendevin/action/browse.py	0.00%	3 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1452   +/-   ##
=======================================
  Coverage        ?   60.88%           
=======================================
  Files           ?       85           
  Lines           ?     3710           
  Branches        ?        0           
=======================================
  Hits            ?     2259           
  Misses          ?     1451           
  Partials        ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

frankxu2004 · 2024-05-01T13:55:37Z

It seems like it's failing on MacOS but passing on Linux, but interestingly failing at the sandbox connection part. Any ideas why this might be? cc @xingyaoww
https://github.com/OpenDevin/OpenDevin/actions/runs/8909840159/job/24468062273

xingyaoww · 2024-05-01T15:38:12Z

Got this weird error while running it on the frontend (make run) with MonologueAgent - any idea what might cause it?

…tegrate_browsergym

…in into integrate_browsergym

…tegrate_browsergym

…in into integrate_browsergym

…tegrate_browsergym

xingyaoww

LGTM! Amazing work!

I've tested on both Linux and my local MacOS environment and can confirm it work as intended.

gasse · 2024-05-09T20:02:40Z

pyproject.toml

@@ -22,7 +22,8 @@ uvicorn = "*"
 types-toml = "*"
 numpy = "*"
 json-repair = "*"
-playwright = "*"
+browsergym = "*" # integrate browsergym as the browsing interface


As far as I understand you only need a dependency on browsergym-core.

browsergym is a meta-package that includes additional benchmarks with extra dependencies, browsergym-webarena, browsergym-workarena and browsergym-miniwob.

@frankxu2004

Hi @gasse Thanks for the comment! We are also interested in evaluating agents in the OpenDevin platform on web agent benchmarks such as WebArena, WorkArena and Miniwob++, so we might just leave it as-is for now!

frankxu2004 added 2 commits April 29, 2024 10:54

add a single-threaded server serving browsergym

c0b69a0

Merge branch 'main' of https://github.com/OpenDevin/OpenDevin into in…

ea134d1

…tegrate_browsergym

neubig requested review from neubig and xingyaoww April 29, 2024 15:24

update poetry

3ce6d07

frankxu2004 marked this pull request as draft April 29, 2024 15:31

update browser page content

22db1de

frankxu2004 marked this pull request as ready for review April 29, 2024 16:15

neubig reviewed Apr 29, 2024

View reviewed changes

add import to make sure browsergym environments are registered properly

afaf393

frankxu2004 added 2 commits April 30, 2024 02:08

remove flask server, use multiprocess impl and Pipe

32b1858

Merge branch 'main' of https://github.com/OpenDevin/OpenDevin into in…

37db390

…tegrate_browsergym

fix

4e9181f

xingyaoww reviewed Apr 30, 2024

View reviewed changes

opendevin/action/browse.py Outdated Show resolved Hide resolved

opendevin/action/browse.py Show resolved Hide resolved

opendevin/browser_server/browser_process.py Outdated Show resolved Hide resolved

opendevin/browser_server/browser_process.py Outdated Show resolved Hide resolved

neubig reviewed Apr 30, 2024

View reviewed changes

opendevin/action/browse.py Outdated Show resolved Hide resolved

opendevin/browser_server/browser_process.py Outdated Show resolved Hide resolved

opendevin/browser_server/browser_process.py Outdated Show resolved Hide resolved

This was referenced Apr 30, 2024

[Browsing]: Make a richer set of web browsing actions #1469

Closed

[Browsing]: Evaluate browsing agents using WebArena #1471

Open

rbren reviewed Apr 30, 2024

View reviewed changes

frankxu2004 added 4 commits April 30, 2024 20:51

Merge branch 'main' of https://github.com/OpenDevin/OpenDevin into in…

e0391d4

…tegrate_browsergym

refactor BrowserEnv

278750b

Merge branch 'main' of https://github.com/OpenDevin/OpenDevin into in…

249a73b

…tegrate_browsergym

update browser action and obs to include more complete info

58fb3bf

frankxu2004 added 3 commits May 1, 2024 06:57

add better html to text conversion

24ff14b

Merge branch 'main' of https://github.com/OpenDevin/OpenDevin into in…

459fe35

…tegrate_browsergym

update for better text conversion to maintain parity with the current…

415a6b9

… handling of browseurlaction

frankxu2004 requested a review from rbren May 1, 2024 13:17

update

18214e4

Merge branch 'main' into integrate_browsergym

caa9522

frankxu2004 and others added 14 commits May 1, 2024 12:11

Merge branch 'main' of https://github.com/OpenDevin/OpenDevin into in…

333fb8e

…tegrate_browsergym

Merge branch 'integrate_browsergym' of github.com:frankxu2004/OpenDev…

97de944

…in into integrate_browsergym

Merge branch 'main' of https://github.com/OpenDevin/OpenDevin into in…

7453649

…tegrate_browsergym

update poetry

163c3b7

update multiprocessing mp

74b3757

fix multiprocessing

5bbf22b

update

7a68405

Merge branch 'main' into integrate_browsergym

2515060

Merge branch 'main' into integrate_browsergym

08d19e0

Merge branch 'main' of https://github.com/OpenDevin/OpenDevin into in…

df3490d

…tegrate_browsergym

Merge branch 'integrate_browsergym' of github.com:frankxu2004/OpenDev…

fbddb8d

…in into integrate_browsergym

Merge branch 'main' into integrate_browsergym

e76ce24

Merge branch 'main' of https://github.com/OpenDevin/OpenDevin into in…

0d0ca9e

…tegrate_browsergym

update github workflow

bde562a

xingyaoww approved these changes May 2, 2024

View reviewed changes

xingyaoww merged commit 836864f into OpenDevin:main May 2, 2024
22 checks passed

frankxu2004 deleted the integrate_browsergym branch May 3, 2024 02:43

neubig mentioned this pull request May 3, 2024

[Bug]: In MicroAgents, prompt should not include browser screenshots #1545

Closed

2 tasks

gasse reviewed May 9, 2024

View reviewed changes

xingyaoww mentioned this pull request May 21, 2024

Add: a mechanism for tracking contributions to the paper #1917

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat] Integrate BrowserGym #1452

[feat] Integrate BrowserGym #1452

frankxu2004 commented Apr 29, 2024 •

edited by neubig

neubig commented Apr 29, 2024

frankxu2004 commented Apr 29, 2024

neubig left a comment

rbren commented Apr 29, 2024

frankxu2004 commented Apr 29, 2024

neubig commented Apr 30, 2024

frankxu2004 commented Apr 30, 2024 •

edited

xingyaoww commented Apr 30, 2024 •

edited

frankxu2004 commented Apr 30, 2024

xingyaoww left a comment

neubig left a comment

rbren left a comment

rbren Apr 30, 2024

frankxu2004 May 1, 2024

frankxu2004 commented May 1, 2024

codecov-commenter commented May 1, 2024 •

edited

frankxu2004 commented May 1, 2024 •

edited

xingyaoww commented May 1, 2024 •

edited

xingyaoww left a comment

gasse May 9, 2024

gasse May 9, 2024

frankxu2004 May 16, 2024

		)

		text_content = html2text.html2text(flatten_dom_to_str(obs['dom_object']))

[feat] Integrate BrowserGym #1452

[feat] Integrate BrowserGym #1452

Conversation

frankxu2004 commented Apr 29, 2024 • edited by neubig

neubig commented Apr 29, 2024

frankxu2004 commented Apr 29, 2024

neubig left a comment

Choose a reason for hiding this comment

rbren commented Apr 29, 2024

frankxu2004 commented Apr 29, 2024

neubig commented Apr 30, 2024

frankxu2004 commented Apr 30, 2024 • edited

xingyaoww commented Apr 30, 2024 • edited

frankxu2004 commented Apr 30, 2024

xingyaoww left a comment

Choose a reason for hiding this comment

neubig left a comment

Choose a reason for hiding this comment

rbren left a comment

Choose a reason for hiding this comment

rbren Apr 30, 2024

Choose a reason for hiding this comment

frankxu2004 May 1, 2024

Choose a reason for hiding this comment

frankxu2004 commented May 1, 2024

codecov-commenter commented May 1, 2024 • edited

Codecov Report

frankxu2004 commented May 1, 2024 • edited

xingyaoww commented May 1, 2024 • edited

xingyaoww left a comment

Choose a reason for hiding this comment

gasse May 9, 2024

Choose a reason for hiding this comment

gasse May 9, 2024

Choose a reason for hiding this comment

frankxu2004 May 16, 2024

Choose a reason for hiding this comment

frankxu2004 commented Apr 29, 2024 •

edited by neubig

frankxu2004 commented Apr 30, 2024 •

edited

xingyaoww commented Apr 30, 2024 •

edited

codecov-commenter commented May 1, 2024 •

edited

frankxu2004 commented May 1, 2024 •

edited

xingyaoww commented May 1, 2024 •

edited