WASM For Core Game Loop & Mods

[zicklag]

This is a proposal for modifying the way we currently run core game logic for Jumpy, made with Bevy. We’re exploring a new approach after encountering some limitations in our JS/TS-based scripting system.

For clarification, this would be done after the MVP release we are working on right now.

Current Challenges

Right now, the core game simulation logic runs in a separate bevy Schedule that is passed to bevy_ggrs to run, rollback, and restore the game state during a match. There are a couple challenges with the current design:

Snapshot / Restore

It's a little difficult to make sure that we perfectly snapshot the whole game logic state. For every game component we create that needs to be snapshot and restored, we have to register it, and the bevy_ggrs plugin has to query all of those components, copy them to a snapshot, and be able to restore the state.

• This works decently well, but still requires being careful about what you do, for instance, you have to avoid using Bevy events or anything else that requires Local<T> state system parameters.
• Performance is something else that needs to be improved.
  • It should be possible to improve the current situation in bevy_ggrs by not using Reflect and possibly cloning in parallel.

Determinism

Ensuring determinism is a difficult, because, often for performance reasons, many things are not deterministic by default, such as query iteration order, or entity allocation.

• This requires creating custom ids or sort components that we can use to make sure we iterate in a specific order, etc. This is can be somewhat tedious.
• It's easy to miss something that might cause a determinism issue, and that can cause network de-sync issues that are hard to reproduce and may lead to difficult to find bugs that are hard to know if they exist before releasing.

Mod Script Performance

Recently we discovered that the performance of our scripting system built on bevy_mod_js_scripting wasn't fast enough to handle the requirement of being able to run 8 simulation frames per 16ms update.

• While we might be able to improve on this, there is a certain level of unavoidable cost to inter-op with JS.
• This is technically acceptable because mods are still fast enough to use for local play.

Proposal

My proposal is that we run the core game logic inside a WASM module.

I think would create a custom, extremely minimal ECS that restricts all components to types implementing Pod and TypeUuid ( Plain ol' Data ) and #[repr(C)]. This makes it easy for scripts to inter-op with and define different component types.

The ECS world would be put in it's own, shared WASM memory. The core game logic, compiled to WASM, would be given a reference to this ECS world memory so it could modify the world and run the game simulation.

Each mod would also be a separate WASM module, that would be given a turn, every simulation step, to modify the ECS world however it wanted, by efficiently borrowing the same WASM memory containing the ECS world. This means that mods could change anything in the core game, and they could be implemented in any language that compiles to WASM. And mods would run just as fast as the normal game logic with no inefficient JS type conversions of any sort.

We would provide out-of-the-box support for using both Rust for the game core/mods and AssemblyScript, which is a TypeScript subset. Since the AssemblyScript compiler can compile itself to WASM, we can easily embed it inside of Jumpy so that you don't have to install any NodeJS tooling or anything to write AssemblyScript mods.

---

This addresses all of the above challenges:

Snapshot / Restore

• The entire WASM linear memory can be copied directly, creating a faithful representation of the core game loop's state at that point in time.
• This should also be very fast: we are copying a contiguous slice of bytes with no processing of any sort.

Determinism

• Since all of the core game logic is trapped inside of WASM, it should all execute deterministically, essentially automatically, if we don't expose any bindings to non-deterministic functions.

Cons of Proposal

There a few cons of the proposal:

Writing a Simple ECS

We'd want to write our own little ECS for this, just to allow us to the ECS design model which I believe has shown great utility for games. This doesn't need to be complicated, or even crazy-optimized or sophisticated. Just enough to get us going.

This minimalist approach would actually have benefits for scripting I believe, by making it dead simple to expose all core game functionality to mods with our simplified data model.

No Multi-Threading The Core Game Loop

This is actually only a temporary limitation. It seems that multi-threaded WASM is even supported in Firefox now, so while it requires extra work to get working in Rust, multi-threaded WASM is becoming a real possibility I think.

Lack of Access to Bevy Directly in Game Loop

Part of our issue is determinism, and too much access to raw Bevy in the core game simulation, but lack of access to Bevy directly also could be inconvenient in some cases, and mean a little extra juggling for some things.

Actually, though, this, to me, seems more like a healthy separation of concerns than a real problem.

Pro's of the Proposal

Other than solving the above challenges:

Potential For Increased FishFolk Project Standardization ( FishBones )

This could be, I think, a step in the direction of our idea for a more standardized FishBones meta-engine that we share between different FishFolk games. It's not a full realization, but more like a step in the direction of making our game logic more self-contained, and allowing us to get for free, things like networking, menuing, input-bindings, localization, etc.

Request For Comments

I wanted to get some feedback on the idea from people who might have input or be effected by this direction:

• @erlend-sh: For your project management perspective on this
• @Piturnah: Because you're working on Bomby and might be effected by this if we started moving in the standardized FishBones engine direction for some FishFolk Rust games.
• @odecay: Same as above, and for your otherwise technical feedback.
• @gschup: Just in case you're interested, and because this is remotely related to GGRS.

[erlend-sh]

In what way does this architecture diverge from Gamercade by @RobDavenport? There appears to be some similarities, so it’d be good to make the differences explicit.

@afonsolage who worked on wabi might also find this interesting.

[zicklag]

Great question.

Compared to Gamercade, the biggest difference is that Gamercade is specifically built for neo-retro style games and presenting a stable API for interacting with the console, in the spirit of a fantasy console.

To the contrary, what we are doing here would not intentionally limit things like graphics or audio capabilities, but would start off providing whatever we happened to need to get our games working.

The idea would be to build the API around our minimal ECS, similar to how Bevy works, which is also an API style difference.

As times goes on it would probably stabilize as we found the common set of features needed for our games, and may even end up with an editor interface for things like maps, spritesheets, etc, so may end up being useful in many situations where Gamercade would be.

It's not clear exactly how it will develop, because I'm still experimenting with how to put things together for good maintainability, modability, performance, etc. and it would be somewhat of an experiment, not unlike the current state of the game. :)

So it is a little difficult to tell how Gamercade and Fishbones might diverge, or converge on ways to solve similar problems, but I think we have different goals at present.

[zicklag]

Oh, and Gamercade doesn't support mods, so that is the other big difference.

[zicklag]

@afonsolage who worked on afonsolage/wabi#1 might also find this interesting.

I read your wabi description, @afonsolage's, which is a great breakdown by-the-way!

The biggest difference between the approach I outline here and the one taken by wabi is that I'm not integrating directly with the Bevy ECS. There are two reasons for this:

• Avoiding the complication of trying to get the bevy component types and systems to integrate with WASM.
• Avoiding non-deterministic execution of the core game logic, by preventing it from accessing things like Bevy resource, events, local state, etc.

Instead we'll try to create our own, ultra-minimal ECS in it's own WASM memory that we can pass to the main game WASM instance, and the mod WASM instances. ( I'm pretty sure this is possible, but I need to do more investigation. )

This means that we context switch exactly once per mod, and never have to make calls back to Bevy during the execution.

Instead, when it's time to render the game, the Bevy app will borrow the minimal ECS world right out of the WASM memory, without copying. And then it will render the sprites that it finds in the minimal ECS world.

---

I'll have to do a little experimenting to make sure I can share the ECS memory across WASM modules. I'll also have to check to see how to architect the minimal ECS so it can be interacted with from other languages.

I'm thinking of making it extremely simple, all contained in one contiguous memory space, and allocating all of the component storages within itself in a way that makes it easy to know how to get to everything. Maybe meaning that it'd essentially be easy to implement the ECS in any language, instead of having to do bindings.

I need to do a little more research to find out what's possible and what that could look like, but I think there's an option that will work. Even if it meant we only support Rust and AssemblyScript for now, and other languages aren't necessarily easy to integrate.

[erlend-sh]

Instead we'll try to create our own, ultra-minimal ECS in it's own WASM memory that we can pass to the main game WASM instance, and the mod WASM instances.

So the main game would still be using Bevy ECS, yeh?

[zicklag]

No, the core game logic would also use the minimal ECS, not just mods. The FishBones "engine" would use the Bevy ECS, and that's where all the rendering, audio, etc, would be happening.

So we'd still use Bevy plugins for audio, rendering, UI, etc., and we make a simplified ECS interface for the core game logic to control the rendering, sounds, etc.

I think that this is appropriate, as long as our minimal ECS is still nice enough to use, and it can be much simpler because we don't need advanced scheduling. Everything runs in order in a single thread.

[erlend-sh]

Oh I see. We’re effectively drawing a line between our Engine ECS (Bevy) and Game ECS (custom). Cool!

[erlend-sh]

I should add that I’m currently thinking of this as post-v1.0 work more so than post-MVP, although we’re not ruling that out entirely.

This new architecture is very exciting, but there’s also very little prior art for it, meaning there are unknown unknowns we have not yet grappled with.

I’m strongly in favor of prototyping it however, whether it’s as a micro-fork of Jumpy (see hoppy) or the new direction for Bomby.

[zicklag]

This new architecture is very exciting, but there’s also very little prior art for it, meaning there are unknown unknowns we have not yet grappled with.

I definitely agree that there's a lot to figure out, and not a lot of reference. There are definitely a few points I'd have to experiment with before I figure out how they work.

I’m strongly in favor of prototyping it however, whether it’s as a micro-fork of Jumpy (see hoppy) or the new direction for Bomby.

Yeah, I think that's a good idea.

Good news for jumpy is that the core logic itself wouldn't change with the new architecture. It would be similar to the straight-forward rewrite from the TypeScript scripts to Rust. All the logic stays, but the details on the ECS interface changes slightly.

Since bomby is even simpler than Jumpy, it might be a good game to prototype with. And since I already have a good idea of what Jumpy needs as a game, having exposure to a different game might be good to broaden my instinct for what use-cases we'd want to cover.

[thunderseethe]

Super interesting idea, I've built some small prototypes of a similar fashion.

An issue I hit early on is without multi-memory you can't share instances of memory between modules. Leaving you to copy mod -> host -> another mod which is very slow. Hosts support the multimem proposal no problem, however no language generates wasm using this proposal and it's nontrivial how to do so.

One possibility in rust is to use a custom allocator with inline wasm that makes use of mulimem on your behalf in a pretty hardcoded fasion. Have you thought about how to get around these superfluous copies in the design of your wasm ECS?

[zicklag]

I put a small demo together in a repository here.

That's great!

I probably should have mentioned this earlier, as it might have saved you some time, but...

There's a comment on an issue I had opened to ask about shared memory in WASM with wasmtime and they said that importing the memory could cause corruption if the WASM modules are not aware of the fact that they are sharing memory.

One of the issues could be the allocator, for instance. Imagine multiple WASM modules all allocating into the same heap. We'd have to have a way to make sure two modules didn't allocate the same memory location. But they said that there's more to it than a custom allocator:

Changing the allocator is not enough. The stack and the data segment must also not overlap. I'm not sure if lld (the linker used by rustc for wasm) supports this. Emscripten does support it afaik when linking as dynamic libraries, but afaik lld doesn't support dynamic libraries on wasm.

So apparently we still need to build the WASM modules as dynamically linked modules. While the the comment linked above said that they didn't think that LLD supports dynamically linked WASM modules, I think it might actually be experimentally working, considering the documentation here, and even if LLD doesn't support it, we could build with Emscripten, which does support it ( though we have to figure out how to make emscripten do it, I don't know where the docs on that are ).

Then we can actually avoid the import memory, I believe, and use a dynamic linking strategy, like in this comment ( skip to the last codeblock in the post, to get a quick overview of the idea ).

If we got that working, the final obstacle is the allocator, and that is actually quite simple, we can just do what we do with C programs. We can have a dynamically linked WASM module exposed with functions for allocating and de-allocating memory, and all the linked WASM modules would call those functions for allocating and deallocating memory.

In Rust code we can then set a custom global allocator that would use those functions instead of the default Rust allocator.

For AssemblyScript, I don't think they have a way to override the allocator, and we'd have to make sure they could support the emscripten dynamic linking format to avoid problems with the stack and data segments, but that might not be complicated and maybe we could contribute that or fork it to get the features we need for that.

[zicklag]

I just had another weird idea. Maybe it'd be possible to take WASM object files instead of WASM modules, and then statically link the object files to each-other at runtime.

So instead of dynamic linking, we're just-in-time static linking. 🤔

I have a feeling that there's something major that would make that not work, and that might be worse than trying to figure out dynamic linking. I don't really know if it makes sense or not.

[tristanpemble]

thanks, this got me further.

So apparently we still need to build the WASM modules as dynamically linked modules. While the the comment linked above said that they didn't think that LLD supports dynamically linked WASM modules, I think it might actually be experimentally working, considering the documentation here, and even if LLD doesn't support it, we could build with Emscripten, which does support it ( though we have to figure out how to make emscripten do it, I don't know where the docs on that are ).

I tried for a while to get wasm-ld to generate a dynamically linked module to no avail. Emscripten was easy enough by just setting cargo:rustc-link-arg=-sSIDE_MODULE=2 in build.rs:

the problem is, I can't find a good way to run the Emscripten module with Wasmtime or Wasmer. Wasmtime does not support the Emscripten ABI, and Wasmer did at one point, but it has fallen out of date and doesn't work with the version of Emscripten used by Rust (3.x).. not sure where to go from here, unfortunately. I think to use an Emscripten built WASM module we'd end up having to implement or stub these functions.

the JIT linking idea is interesting, and some Googling found this repo which could be relevant, though it is beyond my level of understanding.

[zicklag]

How are you printing those metrics in your screenshot?

I think I got LLVM to generate a dynamic library by adding this to build.rs:

build.rs:

fn main() {
    println!("cargo:rustc-link-arg=--shared");
    println!("cargo:rustc-link-arg=--experimental-pic");
}

But I have tested anything else yet.

Edit: Yeah, wasm-dis shows the custom dylink.0 section it's supposed to have.

[tristanpemble]

How are you printing those metrics in your screenshot?

you can use wasm-objdump -x ./module.wasm from wabt

I think I got LLVM to generate a dynamic library by adding this to build.rs:

very cool! this works for me too, but unfortunately I quickly ran into this when doing anything more complicated (e.g. println!() or vec![].join())..

[zicklag]

Posting a comment on Discord and my response

TheRealZerve#1134
i took a quick glance over the wasm jumpy proposal. i think its a good idea to try but the proper architecture is extremely important, and you would need to really understand what is needed for determinism in gameplay and what can be done on separate threads later. ive found that using an ECS for its parallelism benefits, coupled with a scripting solution, are actually not a great solution. unless you severely limit what the scripting world has access to, you end up with a single "sync point" where it loses many of the benefits.

The gamercade approach works because the entire game state is handled inside of wasm memory, and for the most part everything outside of wasm-land is stateless.

It might be a good time to determine what level of modding you want to allow for the game, since more flexibility for modders will cause more restrictions

i can elaborate on this a bit later if some of these points arent clear

---

Hi and thanks for chiming in!

I'll share a little bit more how I'm thinking it will work and it'd be great if you could tell me if that sounds reasonable from your perspective.

The gamercade approach works because the entire game state is handled inside of wasm memory, and for the most part everything outside of wasm-land is stateless.

That's the idea. Because we are essentially forced into this model because of the need to rollback and restore anyway, this doesn't add much more of a restriction on the way we are currently doing things.

The idea is that the WASM world will create entities with sprite components on them, and the Bevy ECS world on the outside will read from those and create Bevy sprites for rendering.

The only thing that the WASM world will be able to get from the outside world is asset information, which doesn't change except in a non-network-session scenario with hot-reload enabled, and the player inputs.

In this case the level of modding we are going for is: mods can do anything that the core WASM game can do. There's nothing really different from the main WASM game and mods. They both have free reign over the WASM linear memory.

The trick to accomplishing this will mainly be creating an ECS that can be interacted with from different languages that compile to WASM.

The exact strategy we'll use for that is unclear right now, as I need to see how simple we can possibly make an ECS. The trick might be making it so simple that we could feasibly implement the ECS in multiple languages. The other option might be to try and do some WASM linking to link WASM functions for interacting with the ECS.

Linking would be preferable if we can make it work, since we don't have to implement the ECS multiple times. This wasmtime example might help.

ive found that using an ECS for its parallelism benefits, coupled with a scripting solution, are actually not a great solution. unless you severely limit what the scripting world has access to, you end up with a single "sync point" where it loses many of the benefits.

I think with the design I'm imagining we avoid that issue, because the WASM world isn't really allowed to query anything from the outside world. Here's a design diagram:

So we break the workflow into very well defined inputs and outputs, and we avoid expensive context switching by not even allowing the WASM mods to interact with anything outside the linear memory, making it truly data-oriented.

[zicklag]

Thoughts on an Implementation Plan

So I think we might be able to take a somewhat iterative adoption of this architecture to allow us to experiment, while still not doing a major code freeze or spending time on things that may not be an immediate high-priority ( like modding ).

Here's a rough thought: for each of the steps below, we should be able to make another Jumpy release. We don't need to get all of these done at the same time. That way we can still work on making the game more playable and add more features ( in theory ) while still making progress on moving to an improved architecture.

We can be pretty confident that we'll uncover things along the way and it won't play out exactly like this, but that's another good reason to attempt to be able to make releases in between, so we can show steady progress, even if there are unexpected changes to the plan.

Step 1: Create Minimal ECS For our Game Logic ( discussion )

Here we make a minimal, simple ECS that we put all of our game logic into. We make this ECS world implement Clone so we can trivially snapshot and restore it. This should help our world snapshot and restore performance, too, which might allow us to bring our simulation updates per second back up to 60, improving the game's visual FPS.

ℹ️ Note: If we still need to leave the simulation updates per second at 45 for performance reasons, we can implement interpolation for the sprites to make it look like it's running at 60 fps even though the simulation is slower. That's more complicated though, so if we can just make it more performant that'd be great.

This will be mean migrating the game logic to the new ECS, but it should be relatively straight forward, we're mostly just going from Bevy ECS to our minimal ECS. Also, it's good to do this early, so we can add items without having to rewrite them again later.

This should also make sure we get rid of 98% of any chance for any determinism bugs, which, when using the Bevy ECS, are very easy to accidentally introduce.

This gives us a lot of the advantages of the full design, other than modding, and allows us to keep working on features without having to migrate everything again.

Timing for Step 1

This is the step that I feel might be good to do right after the MVP, if it doesn't turn into a can of worms that we just don't want to deal with right now. But I think, with the MVP being nearly done, we need to discuss ( or I just need a little context on ) our next plans before coming to any conclusions.

The biggest unknown of Step 1 is the design for the minimal ECS. I have to do a little experimenting. I'm hoping to make it so simple that it's easy, but I'm not sure if that's going to work out like I'm imagining, without a little bit of testing.

Step 2: WASM

For step 2, we move our minimal ECS and game logic into a separately compiled WASM module.

This can make it super easy to add hot reload for all the game logic just by re-loading the WASM module. This would also probably be when we test out loading and running WASM modules in the browser ( something Wasmer already has an example for, which is promising ).

At this point, we take a big step in the direction of modding because all the game logic is now an asset ( the WASM module ). So not only do you get hot reloading, but people can fork the core game logic, without modifying the engine, and that means not having to re-install the game, and being able to load un-trusted Jumpy game forks without worry.

It's not selective modding where you can add an item without forking the game core, but it's a great step.

Step 3: Full Modding

In this step we work out how to have multiple WASM modules are coordinate on the same game, so that mods can add individual items that can be selectively plugged into the normal game. The idea will be that mods can be written in at least Rust and hopefully AssemblyScript, and maybe with the future possibility for all kinds of languages such as Python, JavaScript, etc.

This is the only step with huge unknowns and lack of prior art, so we can still get great features offered in Step 1 and 2 without having to worry about this just yet. Having Step 1 and 2 in place, though, makes it a much easier migration to the full modding phase.

[zicklag]

@makspll I was going to put this in Discord, but then it got too big. :D

---

Just to catch you up with how Jumpy is developing in the scripting realm: I'm currently in the middle of migrating the core game loop to our own micro ECS.

This helps us tackle the fact that Bevy's API is complicated and difficult to bind to scripts ( though we bind a lot of it bevy_mod_js_scripting, though without the best performance ).

By putting the core game in a much simpler ECS, I think we can make it much more feasible to actually bind the entire ECS API to scripts in a performant way, through a C API if necessary.

It doesn't give you unlimited power over Bevy, but it does give you unlimited power over the core game, running in the Micro ECS.

It looks like you're doing some great work on providing a clean way to bind scripts to Bevy, which should be totally compatible with our new model, but instead of triggering Bevy functions, etc. from scripts, you would be triggering functions in our micro ECS.

I think the micro ECS can make Jumpy way more approachable to different kinds of scripting APIs, including yours, because of the simple ECS API.

Once we get it working, we could even expose a C API that you could access from dynamic libraries ( .dll, .so, .dylib ) files compiled from Rust, C, or C++ if you needed native mods

Once I get the micro ECS migration finished, my attempt at adding scripting will be put the game and mods in dynamically linked WASM modules, so that they can all share the same WASM linear memory and avoid any FFI cost. That limits scripting languages to those that run in WASM, but Lua/Python/Rhai might still be options.

[erlend-sh]

Great comment by @philpax:
https://www.reddit.com/r/rust/comments/119bpd2/comment/j9ltxxy/

Hah, my work on https://www.reddit.com/r/rust/comments/118wlda/introducing_ambient_01_a_runtime_for_building/ me a minor domain expert in this :p

So this is absolutely possible, but there are several pain points you'll run into.

How can I share these types across the WASM-boundary? wasm_bindgen wouldn’t really help in this situation, as it is only (mainly?) focused on Javascript interop, correct?

Yep, you're right. For this, there are a few options. The ones most relevant to you are fp-bindgen, which targets Wasmer, and wit-bindgen, which targets wasmtime.

We use (an old version of) wit-bindgen, and have found it to work well at the actual work of translating types. It is very much in-development software, so there's a lot of rough edges when it comes to the actual experience of using it (error reporting when you get the syntax right, limited support for representing types, etc).

wit-bindgen's primary focus is supporting the Component Model, which is a standard model for WASM programs to interact and communicate with each other. We have not yet updated to use the CM, which is why we're using an old version, but it should simplify sharing of types across modules and throughout an API.

Note that you will likely need to write custom binding code on either side as the objects generated by your binding generator are unlikely to have all the functionality you'll want (derive-traits, etc). We essentially just use them as data transfer objects across the boundary.

I can't comment on the fp-bindgen experience, but my understanding is that it makes the Rust host/Rust guest experience as good as it's going to get. We didn't go with it as we're using wasmtime and because we wanted to support non-Rust guests.

Is there a way to pass a region of memory (such as the slices) from Rust to WASM? I don’t want to do any allocation within WASM, and would highly prefer to manage memory on the Rust side.

Yes and no. You have direct access to the VM's memory and can copy into it, but you would need to negotiate with the guest as to the memory to copy into (which will likely require a guest allocation).

There's no way to map host memory directly into the VM; this is due to the sandboxing that WebAssembly requires for its execution model.

I’ve chosen wasmer because it looked like a promising runtime, are there any other WASM-runtimes that might be more appropriate for this usecase?

As mentioned, we use wasmtime. It works well, is standards-compliant, is a product of the Bytecode Alliance (who are driving WASM and its surrounding ecosystem forward), and I've found the team behind it to be easy-going.

I can't comment on wasmer, but I've heard not-so-great rumours about the company and how they operate. (A quick search suggests this post). I'd steer clear for that reason, but that may not matter for your application.

And slightly subjective, but how bad of an idea is it to use rustc to dynamically generate code in this fashion?

It's... fine: there's no problems with the compiler being used this way. The main problem is that compiling Rust is no easy feat, which can lead to performance hitches when you're compiling.

Aside from that, it's generally pretty good. The developer experience can be a little challenging when you straddle two targets, but that can be managed with careful use of workspaces and overrides.

---

The overall devex on the guest side can be made to look and work quite well (native glam types, etc) but it'll require a fair bit of work.

Depending on the problem you're trying to solve, you may be better served by using a different guest language or a different language VM entirely (Rhai, Mun, and potentially Futhark come to mind)

[afonsolage]

I'm a bit late for the party, but if I understood correctly, we are talking about having multiple ECSs world, so each mod will have it's own independent world, there will be the main Bevy ECS World and a Minimal ECS World for each loaded and running WASM Mod. It's a interesting approach.

Have we considered using Bevy ECS multiple words? (I'm pretty sure yes), since bevy_render does kinda like that, in order to have better performance and parallelism in the rendering, it has it's own independent world. The main communication happens using extraction phases

I made a sketch on how I'm thinking it, after reading all thoughts:

IMO the hardest part is still how to interact between host and guest types. Some folks, like Ambient, made the whole engine thinking in running on wasm, and even so, there is an intermediare type, which still requires a translation, this is due how the memory are handled in native vs wasm environment (IIRC).

My goal with wabi is to enabled modders to create their own components and this be shared across mods, which currently is only possible using some kind of dynamic component.

If jumpy has a more restricted use cases and doesn't allow modders to create their own components, but instead, just use existing components, then this could be achievable just implementing this middle-man ECS, which would translate 1:1 to existing Bevy ECS components.

[zicklag]

there will be the main Bevy ECS World and a Minimal ECS World for each loaded and running WASM Mod. It's a interesting approach.

That's actually not what I was thinking, but that is an interesting thought.

Have we considered using Bevy ECS multiple words?

Yep. The only reasons we weren't using Bevy ECS was snapshoting, determinism, and the difficulty of making a scripting API to it.

Though admittedly, forcing the game core to run in WASM might address snapshotting and determinism. 🤔 It might prove important, that bones_ecs can snapshot and run deterministically without being run in WASM, but I'm not sure yet.

---

What I was thinking was having the entire minimal ECS world inside of a WASM linear memory, and just having one minimal ECS world.

Then all the other WASM mods and the WASM game core would interact with the same ECS world, in the same WASM memory.

There'd be zero copying when different WASM mods run their systems, they would all just interact with the same world.

There would be only one big copy performed per-frame, and that would be the extraction of the rendering components such as sprites, etc. from WASM linear memory, into the Bevy world for rendering.

That would happen similar to how we already do it here in bones_bevy_renderer.

Even though we have to make a copy for the sprite data during the extract phase, because Bevy will only render things in it's world, during the extract phase we can read the component data straight out of the WASM memory without copying, if we find a way to make sure we modify any component data containing pointers.

Because any pointers in components will be relative to the WASM memory, when the host wants to read them out of the WASM memory, it will have to have some way to re-map the pointers to it's memory own memory address, relative to the WASM linear memory address.

I haven't looked into that enough yet. And then there's the challenge of linking, which I still have to look into, as well.

Overall, it looks like the approach I'm thinking of is closest to the one settled on by @troyedwardsjr here, just replacing Flecs with our own bones_ecs.

It seems, if it weren't for the fact that Flecs is written in C and can't seamlessly compile to WASM when integrated with Rust, it actually could have been a good option for us.

[troyedwardsjr]

@zicklag It can compile fairly seamlessly to WASM and work in the browser with the Emscripten target, possibly also the WASI target. If you do ever change your mind on the Flecs part, I think it would be very useful to combine our efforts either majorly or in some small way so there is less parallel work in the open source space. I think with all of the efforts, thinking, and work we all collectively put into this problem, it'd be a shame if we don't work together.

I know you have been working with scripting + Rust ECS for a while now, as I've been following and chasing your rabbit hole of Github issues for a long time on various different Rust ECS projects / Bevy when I did my initial research into this idea a year or so ago, so I'm sure you have a lot of ideas.

However, I can tell you that the wasm32-unknown-unknown target, when it comes to multi-threading, dynamic linking, hot-reload, etc. is not supported or works poorly, and even if it comes with support for wit-bindgen when it's stable and that will be the "official" solution for communicating between WASM modules, as I mentioned in my dev log the serialization overhead and synchronizing data across different memory spaces is simply not suitable for games in terms of performance. So what is there really to gain from this target? Winit? Wasm-Bindgen which does a lot of data copying?

I think Emscripten should stop being a tier 2 target until we have all of these features and C bindings support in the tier 1 wasm target. I think if we combine our efforts as a community, we could create the Emscripten Rust ecosystem in order to provide us with the solutions that we need right now. Also, Godot Rust uses Emscripten as well, so it is being relied upon by a large project already for a performant game dev use case: https://godot-rust.github.io/book/gdnative/export/html5.html

[zicklag]

So what is there really to gain from this target? Winit? Wasm-Bindgen which does a lot of data copying?

That's a good question. Honestly I haven't deeply investigated what would happen if we tried to use the Emscripten target instead. WASM bindgen and winit would be the biggest motivators, combined with wanting to make it eaiser for users to compile mods.

I'd have to investigate what it takes to get Bevy working with emscripten. If winit doesn't work, we may have to create a new windowing plugin for Bevy.

We might need emscripten anyway, though, just to get any kind of WASM linking working with Rust because of overlapping stack/data segments in modules compiled with the normal Rust target. So we might not have a choice but to got with emscripten, which might make Flecs a good option,though I'd be hesitant to do another major ECS migration. At the same time, migrating to scripting might not be much harder than migrating to Flecs.

I've definietly got a lot of thinking/investigating/experimenting to do on the subject, so I can't say anything for sure yet, but I've got an open mind and I'll probably be looking closely at the work you've been doing. It'd definitely be great collaborate.

Once I get a chance to spend some time on this I'll be looking more carefully at an exact way forward.

[zicklag]

After our migration to Bones ECS and, more recently, our migration to the Bones Framework, we've essentially covered our major use-cases ( networking has yet to be re-implemented, but it's all possible within the new framework ).

WASM didn't pan out directly because of the lack of linking standards and a lack of motivation to switch to using Emscripten for WASM to alleviate that ( which may be a viable option later ).

We've satisfied our objectives by using the Bones framework, and even integrated Lua scripting, without directly migrating the whole core game to run in WASM, wrapping up this discussion! Thank you for everybody who was involved!