You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am working with ARCore and a Sceneform fork (using Filament 1.21) to render an AR scene.
The problem-
I am trying to apply some post-processing effects (at every frame) on the camera preview, based on results from a custom ML model. From my understanding, this requires me to upload new data to a Texture very frequently, and I'm not sure of the most optimal approach for this.
My proposed workflow is as follows-
Use ARCore's acquireCameraFrame() method to receive camera frames in YUV format.
Convert the YUV image to RGB bitmap.
Feed the RGB bitmap into a custom ML model (for example, Mediapipe's ImageSegmenter model), which then outputs a bytebuffer (for example, a category mask where each pixel value represents a category)
Upload the (latest) bytebuffer as a Sampler2D Texture to the camera material's fragment shader, and mix it with the external/camera texture to achieve the required effects.
My question-
Is this a good overall approach for the given problem, considering it is initiated at every frame?
How do I efficiently convert the bytebuffer into a Sampler2D Texture for my fragment shader?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I am working with ARCore and a Sceneform fork (using Filament 1.21) to render an AR scene.
The problem-
I am trying to apply some post-processing effects (at every frame) on the camera preview, based on results from a custom ML model. From my understanding, this requires me to upload new data to a Texture very frequently, and I'm not sure of the most optimal approach for this.
My proposed workflow is as follows-
My question-
Beta Was this translation helpful? Give feedback.
All reactions