Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bot test driver script #1033

Closed
wants to merge 8 commits into from
Closed

Bot test driver script #1033

wants to merge 8 commits into from

Conversation

kazimuth
Copy link
Contributor

Description of Changes

To allow running bot tests with a comment on a PR, like the other benchmarks.

Small PR but it took a lot of finagling. There might still be an extant tracy issue...

Expected complexity level and risk

complexity 2, risk 0

@kazimuth
Copy link
Contributor Author

kazimuth commented Apr 1, 2024

Man, I can't figure out why tracy isn't correctly producing a log. Sometimes it works, sometimes it doesn't. Really it doesn't seem set up for automated deployments like this. I think right now the issue is it's ignoring a SIGINT, which is supposed to shut it down and write out the log, but possibly there's also another issue related to spacetime server restarts. I'll try rearranging the script to let it shut down on its own...


target/release/spacetime publish -c bitcraft --wasm-file "$BOTS_DIR/bitcraft_spacetimedb_with_wasm_opt.wasm"

python3 "$BOTS_DIR/deploy_world.py" -H http://127.0.0.1:3000 -m bitcraft -f "$BOTS_DIR/Spacetime128x128.snapshot" >"$OUT_DIR/deploy_world.log" 2>&1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might be okay with running a smaller world, at least in the typical case? not sure

Copy link
Contributor Author

@kazimuth kazimuth Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you send me a smaller world I can swap it out. But, it actually uploads in about 15 minutes so I don't think it's a huge problem.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah fair enough. I guess there's also a question about how to maintain the world snapshot? I believe it's essentially just a replay log of queries, so it'll change with some PRs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

John is providing snapshots and other support stuff for bot tests on gdrive periodically, but it wants to be in source control or something soon, maybe with git lfs.

@bfops bfops added release-any To be landed in any release window no runtime change This change does not affect the final binaries labels Apr 1, 2024
@kazimuth
Copy link
Contributor Author

kazimuth commented Apr 1, 2024

It looks like the issue is that spacetime is failing to reboot. Currently, the script:

  1. runs spacetimedb without tracy to upload the world
  2. uploads the world
  3. shuts down spacetimedb
  4. starts spacetimedb with tracy
  5. runs tracy
  6. runs a bot test
  7. shuts down spacetimedb and tracy
  8. zips up all the logs and tracy trace and uploads them

the problem right now seems to be step 4. Sometimes spacetimedb is failing to restart after being shutdown via a SIGINT (CTRL-C in bash.) The error is the following:

Error: database error: IO error: could not acquire lock on "/home/ubuntu/actions-runner-linux-x64-2.314.1/_work/SpacetimeDB/SpacetimeDB/crates/bench/bottest/.spacetime/control_node/control_db/db": Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }

Not totally sure what to do next here. I could try and avoid restarting spacetimedb and just run tracy the whole time the world is uploading, but then the tracy logs are going to get huge.

@joshua-spacetime joshua-spacetime linked an issue Apr 25, 2024 that may be closed by this pull request
@kazimuth kazimuth deleted the jgilles/bottest2 branch May 31, 2024 17:51
@cloutiertyler
Copy link
Contributor

Closing in favor of #1255

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no runtime change This change does not affect the final binaries release-any To be landed in any release window
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Run bot workload in CI
3 participants