Masking with live cam? #677
-
|
I'm coming from DFL (xseg) and Rope (occluder, clip) and am wondering if it's possible to use any of these masking techniques on live video feed with roop-unleashed? I can't see in the roop-unleashed ui anywhere to even manage masking? |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 4 replies
-
|
You're in for a surprise, just click on the advanced masking button: DFL XSeg is fully supported, as is CLIP and Images can be masked by drawing a mask yourself. However back to your original question, no in live-cam mode no masking is included. It never occured to me that this would be needed? And if you're using roop unleashed in a browser then you're already using the gradio version 😃 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, would it be possible to implement basic masking in live? Something similar to rope's occlusion where it just masks whatever it thinks is in front of the face?
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: C0untFloyd ***@***.***>
Sent: Sunday, May 12, 2024 7:57:31 PM
To: C0untFloyd/roop-unleashed ***@***.***>
Cc: frosty3907 ***@***.***>; Author ***@***.***>
Subject: Re: [C0untFloyd/roop-unleashed] Masking with live cam? (Discussion #677)
You're in for a surprise, just click on the advanced masking button:
grafik.png (view on web)<https://github.com/C0untFloyd/roop-unleashed/assets/131583554/c4cfcf54-809a-409f-95fa-af2d04234127>
DFL XSeg is fully supported, as is CLIP and Images can be masked by drawing a mask yourself. However back to your original question, no in live-cam mode no masking is included. It never occured to me that this would be needed? And if you're using roop unleashed in a browser then you're already using the gradio version 😃
—
Reply to this email directly, view it on GitHub<#677 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIXUU55U4YW6DNEMLACRN7LZB44IXAVCNFSM6AAAAABHQLGOVGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMBUGA3TM>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, todays release always uses DFL XSeg and it seems to work quite well. |
Beta Was this translation helpful? Give feedback.
-
|
The mask_xseg processing in virtualcam had a major effect on frame rate on my M1Pro Mac, going from 3.4 fps to 2.4fps. I added a checkbox to enable/disable it in my private branch. And somewhat off-topic: I have also added an fps printout every 5 seconds, a checkbox option to mirror/flip the preview output and a slider to crop the input camera image (effectively zoom the camera input, but also lower the resolution). If there's interest, I could contribute my changes, but they are relatively simple. I haven't tested with the actual OBS feed though. The requirements.txt specifies onnxruntime-silicon==1.17.1, but this doesn't exist, because silicon was wrapped into onnxruntime 1.17, so the specific check for arm64 can be removed. |
Beta Was this translation helpful? Give feedback.
-
|
Amazing work! Thank you!
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: C0untFloyd ***@***.***>
Sent: Monday, May 13, 2024 6:17:13 AM
To: C0untFloyd/roop-unleashed ***@***.***>
Cc: frosty3907 ***@***.***>; Author ***@***.***>
Subject: Re: [C0untFloyd/roop-unleashed] Masking with live cam? (Discussion #677)
Yes, todays release always uses DFL XSeg and it seems to work quite well.
—
Reply to this email directly, view it on GitHub<#677 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIXUU5ZHPMFOWWZOQ4NTOTTZB7E4TAVCNFSM6AAAAABHQLGOVGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMJQGA4TE>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I like the idea of zooming the feed to reduce artifacts due to partial face detection on edges.
Also would love it if the 'drop frames with no face detected' option worked for live cam!
…________________________________
From: Juri Munkki ***@***.***>
Sent: Monday, 13 May 2024 10:42 PM
To: C0untFloyd/roop-unleashed ***@***.***>
Cc: frosty3907 ***@***.***>; Author ***@***.***>
Subject: Re: [C0untFloyd/roop-unleashed] Masking with live cam? (Discussion #677)
The mask_xseg processing in virtualcam had a major effect on frame rate on my M1Pro Mac, going from 3.4 fps to 2.4fps. I added a checkbox to enable/disable it in my private branch.
And somewhat off-topic:
I have also added an fps printout every 5 seconds, a checkbox option to mirror/flip the preview output and a slider to crop the input camera image (effectively zoom the camera input, but also lower the resolution). If there's interest, I could contribute my changes, but they are relatively simple. I haven't tested with the actual OBS feed though.
The requirements.txt specifies onnxruntime-silicon==1.17.1, but this doesn't exist, because silicon was wrapped into onnxruntime 1.17, so the specific check for arm64 can be removed.
—
Reply to this email directly, view it on GitHub<#677 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIXUU54J4R3GAFZ7HA66WHDZCCYLRAVCNFSM6AAAAABHQLGOVGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMJZGYYTG>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
It's really just a crop right now. I originally scaled the webcam image after cropping, but I didn't see any real advantages and you're just processing scaled up pixels. So for instance 1280x720 zoomed 2X will result in a 640x360 image output. I don't know how OBS handles that. I took a quick peek at the frame processing code and you have to go a few function calls deep to allow the virtual cam to determine that no face was found. The process manager is really for the video & image file processing, so one would have to be careful not to break anything else. Ideally, I think the virtual cam would do as much as possible in RAM and not generate files for every frame that it processes, so that could be another good thing to tackle while working on that. It might not really affect frame rates that much these days, but it still seems inefficient, if it can be avoided. |
Beta Was this translation helpful? Give feedback.
-
|
Just a real quick reply regarding files: the livecam code is abusing the preview processing meaning no files are written or created. It's already just memory processing. And yeah, the UI could be extended with some handy utility stuff. |
Beta Was this translation helpful? Give feedback.
-
|
That's good to know. I'm getting a new directory and an image.webp in the temp folder for every processed frame, but that's gradio then and there's probably no way around that except to stop using gradio for the preview. Disabling the preview in gradio doesn't seem to affect the frame rate in any way. |
Beta Was this translation helpful? Give feedback.
Yes, todays release always uses DFL XSeg and it seems to work quite well.