Show passthrough on excessive dropped frames (or if anchor lookup fails)

[shinyquagsire23]

If frame drops get too bad, the head snchor API returns a nil value after too much time has passed, and the image becomes head-mounted. We should consider dropping the immersive opacity (if possible) in this case, to prevent nausea.

If we cannot drop the opacity, we should bother Apple about it, because the command center thingy does it just fine.

[zarik5]

What alvr_client_openxr does is recording the history of sent timestamps + poses in a HashMap, and use the pose matching with the timestamp from the video frame. In this way you don't have to query the pose again. This is also important because from the time you poll the pose the first time to the time you draw the image, the pose might be interpolated/extrapolated slighly differently.

[shinyquagsire23]

I grepped around in CompositorServices, and also tried setting the alpha to 0.5/0.0 in the fragment shader, and it genuinely seems like there's no app-side control over the opacity. So I guess I'll go ahead and submit Feedback for that because it'd also be useful for ie, a flat game streaming app that allows keyboard peeping.

[shinyquagsire23]

@zarik5 Yeah I recorded the timestamps+poses initially, but until I do some Unsafe Swift, the anchor pose is get-only and has no setter. I might actually try that now just to see if it's even possible to set it, or if the compositor re-fetches the timestamp somehow.

[zarik5]

Ah damn, this is the same issue with WebXR IIRC. This is pretty limiting, because no setter means it will not support apps with multi-frame pipeline depth (like VR streaming) (unless late-latching is implemented, but is fundamentally incompatible with VR streaming). The solution is to reimplement reprojection/ATW. This would also be useful for PhoneVR.

[panthurrr]

What we do have is access to the spatial environment during full immersion; maybe possible to enable a spatially accurate floor onto the ground if we cannot dismiss the immersive view and that object will not be affected by any lag.

Currently this is a volumetric window which means it can be moved so not a great ground replacement but it's at least a spatial anchor that is stable.