ReplayKit and multicam .audioMic issue

[levs42]

Describe the bug

There's some interference between stream.isMultiCamSessionEnabled = true and ReplayKit. Somehow enabling multi cam session changes audio format of .audioMic audio in SampleHandler. Without multicam:

inputFormat:<AVAudioFormat 0x3012d1ae0: 1 ch, 48000 Hz, Int16>

With multicam:

inputFormat:<AVAudioFormat 0x300ebdb80: 3 ch, 48000 Hz, Int16, interleaved>

Cannot hear audio on the stream. I tried to use channel map for the main track but it doesn't work. Audio buffer has data.

To Reproduce

1. Run iOS Example and keep it open
2. Start screen sharing

Expected behavior

To head audio on the stream

Version

main branch, 52a34fd

Smartphone info.

iPhone 15 Pro Max, iOS 15.5

Additional context

No response

Screenshots

No response

Relevant log output

No response

[shogo4405]

• Is it correct to understand that ReplayKit is the one used from the Control Center?
• The reason is that it becomes 3 channels. 😭

[leo150]

• Is it correct to understand that ReplayKit is the one used from the Control Center?

Yes, from the Control Center

• The reason is that it becomes 3 channels. 😭

Indeed, it's unfortunate! I believe audio mixer still should handle it if the proper channel mapping is provided, but somehow it doesn't. Will take a look too, just wanted to report the issue.

[shogo4405]

Currently, channels passing through this line fail to convert and are treated as muted audio.
https://github.com/shogo4405/HaishinKit.swift/blob/main/Sources/Util/AVAudioUtil.swift#L45-L47

Modify it as follows:
main...feature/replaykit-multicam-issue#diff-7d19719e55e081126ea8a413c2b66323cb0a7881098ef295294650da226b0922R37-R38

rtmpStream.audioMixerSettings = .init(sampleRate: 0, channels: 2)
rtmpStream.audioMixerSettings.tracks[0]?.downmix = true

In an environment with iOS 17.5 + iPhone 15 Pro Max, you can hear the mixed audio.
I am struggling with whether kAudioChannelLayoutTag_DVD_2 is the appropriate value.

[levs42]

While trying kAudioChannelLayoutTag_DVD_2 the right ear sounded significantly more quiet on the stream while listening it through headphones. I also tried without downmix with no difference; I tried kAudioChannelLayoutTag_AAC_3_0 and it worked without downmix and sounded fine on the stream. The layout is a bit different: Center, Left, Right instead of Left, Right, Center surround for DVD2. Maybe "surround" channel is treated differently by CoreAudio; I'm not too familiar and there's barely any doc. AAC_3_0 worked with 1 output channel too.

Edit:
CoreAudioTypes.h sheds some light how the types correspond between each other.

[shogo4405]

I changed it to MPEG_3_0_B because when I look at it later, I might mistakenly think "Why is it AAC when it's PCM audio?"

kAudioChannelLayoutTag_AAC_3_0 = kAudioChannelLayoutTag_MPEG_3_0_B, // C L R

It was ReplayKit audio, CLR was appropriate. However, on some devices, it might come as LRC.
I thought it might be a good idea to allow the library user to set the LayoutTag in the future.

Thank you for the feedback.

[levs42]

It's also worth updating output channels count in the screencast example, otherwise setupAudioNodes throws an error.