Client-side MetalFX upscaling for image quality enhancement. #97
Conversation
| } | ||
|
|
||
|
|
||
| func convertRGBA16FloatToRGBA8Unorm(device: MTLDevice, commandQueue: MTLCommandQueue, sourceTexture: MTLTexture) -> MTLTexture? { |
There was a problem hiding this comment.
Is there a reason we can't keep the textures in RGB16? The drawable framebuffer is RGBA16Float so ideally we'd like to keep it so that HDR still works
There was a problem hiding this comment.
Actually wait, is this only for the PNG? If that's the case, then it's fine.
|
|
||
| // Simple YUV to RGB conversion | ||
| float3 rgb; | ||
| rgb.r = y + 1.402 * uv.y; |
There was a problem hiding this comment.
This should use the same YUV conversion matrix as the YUV fragment shaders, otherwise it will have different colors between HDR (BT.2020) and SDR (BT.709)
There was a problem hiding this comment.
May or may not want to also do NonlinearToLinearRGB and EncodingNonlinearToLinearRGB here too? Not sure if the MetalFX upscaler cares about the images being a bit over-bright.
|
|
||
| float3 rgb_uncorrect = in_tex_rgb.sample(colorSampler, sampleCoord).rgb; | ||
|
|
||
| const float DIV12 = 1. / 12.92; |
There was a problem hiding this comment.
Needs rebasing to use NonlinearToLinearRGB and EncodingNonlinearToLinearRGB now
| desc.outputHeight = Int(Float(inputHeight)*scaling) | ||
| desc.colorTextureFormat = pixelFormat | ||
| desc.outputTextureFormat = pixelFormat | ||
| desc.colorProcessingMode = .perceptual |
There was a problem hiding this comment.
If the YUV conversion shader is changed to use NonlinearToLinearRGB and EncodingNonlinearToLinearRGB, change this to .hdr or .linear
Basically these code is generated by ChatGPT, I’ll carefully check them. |
This enables MetalFX for ALVR-visionos.
With a toggle to control if use MetalFX and upscaling factor selection, and also debug option to save raw image and upscaled image.
To use MetalFX, we need to convert YUV420 to RGB first.
UI
Comparsions:
Top: raw
Bottom: Upscaled
Known issue:
The R and B channel of saved image is opposite, but it's ok for debugging seeing these images.
Honestly, due to the bandwidth limitation of apple, metalfx does not change a lot currently.