External Renderer

[odin47]

Description

Hello,

I am working on creating an external renderer to support multiple file formats(like OBJ) and displaying multiple models in a single scene using "registerRenderer" method. How do I achieve this and is there any example code regarding this.

Model Viewer Version
2.1.1

[elalish]

For this, I'd recommend you fork the repo, since three.js already has support for all this. Then you can just include their various loaders and add as many objects to our ModelScene as you want. It's nicer developing in TS anyway.

[elalish]

Yeah, that's fair. If you're willing to share your work, I could give you some help and you could add the result as a PR?

[pjdixit]

Hi Elaih,

Thank you for getting back! I was able to get it to work partially after some more fiddling. However, in AR mode I am not sure how to use my WebGL canvas to render instead of the model src attribute passed to the <model-viewer>.

I was able to get it to work to some extent in a Vue App. I will try to strip down the code to bare minimum in the next 2-3 days and send it your way.

[pjdixit]

To clarify further. here is the current live app - https://www.cadflux.com/arapp/arviewer. It uses default renderer without custom RendererInterface.

I have implemented our own non-three.js rendering engine that renders to WebGL canvas like here- https://www.cadflux.com/app/gallery.

I am trying to figure out how to use my WebGL canvas for AR through custom RendererInterface instead of default one.

[elalish]

Yes, that should be doable. I don't know of many people exercising this feature, so there may be bugs. A stripped down version should help us identify where any problems may be. Likewise, if there's anything that turns out be unclear in our docs on the subject, PRs are always welcome to improve those as well.

[pjdixit]

Its very encouraging to know it is feasible. I will send you the stripped down version soon to investigate. Thank you very much!

[pjdixit]

Hi @elalish I have attached a stripped down version of my attempt to use external renderer.

I have used a canvas slot which has canvas element with WebGL context that I would like to see in AR mode instead of the src file specified as attribute to <model-viewer> element.

I have also implemented the RendererInterface class CustomARRenderer using registerRenderer function and load, render, and resize function gets called as expected.

However, it is not clear from documentation how to specify that I would like to see my WebGL canvas element instead of GLB file in the AR mode.

This is what I see in regular mode (expected)

This what I see in AR mode ( not expected, how can I get by WebGl canvas to render here instead of the GLB file?

Source code:
ar_external_renderer.zip

Live Version - https://www.cadflux.com/Testing/ar_external_renderer/index.html

I will appreciate any guidance to achieve this. Thank you!

[elalish]

Oh, so it's working! It looks like you're just waiting for a feature we haven't implemented yet: using the external renderer for the WebXR AR session. Honestly I don't think it should be too hard - want to try a PR? I think you'll just need to copy a bit of logic from Renderer.ts to ARRenderer.ts, or maybe pack it into a function. I mostly haven't gotten around to it because building an external renderer for testing it is more work than the actual feature.

[ConfidantCommunications]

That's exactly what I am attempting to achieve. I can modify the default scene using registerEffectComposer (not using an external renderer) but it doesn't display the modified scene in AR mode.

[elalish]

Did you read my above comment? The external renderer just hasn't been hooked up to AR yet in MV. A PR would be most welcome!

[ConfidantCommunications]

Yes, thanks. I guess I am not understanding the steps it takes internally when it switches to AR mode. I can see that the constructor for ARRenderer intakes a renderer, which I assume is the default renderer. I don't want to use an external renderer if I don't need to do so. Are you saying that the only way to have ARRenderer use a runtime-modified scene is to use an external renderer? I'm willing to contribute code modifications but I want to make sure I'm going down the right path.

[ConfidantCommunications]

As it is, I can add new geometry to the default scene, but it doesn't get included when I switch to AR mode.

[elalish]

Oh - I misunderstood "That's exactly what I am attempting to achieve" - helps to be specific. It sounds like you don't want an external renderer, but instead want our effect composer to work in AR. That's yet another yet-to-be-implemented feature, which a PR would be great for. For that, I believe you just need to bring this logic to here. Just keep in mind that the browser is compositing our transparent render onto the camera feed, so your effects still can't read the camera pixels.

[ConfidantCommunications]

Thanks, I shall delve into this. Stay tuned!