Expose audio sample buffers for Android #89
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
cloudwebrtc pushed a commit that referenced this pull request Sep 14, 2023
* Initial draft * Working impl * doc and cleanup * doc update
cloudwebrtc added a commit that referenced this pull request Sep 20, 2023
* Improve e2ee, add setSharedKey to KeyProvider. * update. * reset has_valid_key after RatchetKey. * update. * clone key_handler from share_key for each participant. * add RatchetSharedKey and ExportSharedKey. * make KeyProvider::SetSharedKey only valid when KeyProviderOptions::shared_key == true. * remove unused enum. * Fix memory leak when creating audio CMSampleBuffer #86 * add scalabilityMode for AV1. * fix bug for scalability-mode. * add scalability-mode support for VP9. * add failure tolerance for framecryptor. * add failureTolerance for android/objc. * fix: make H264's unencrypted_bytes consistent with js-sdk. * wip: ScalabilityModes for android. * update. * wip. * wip. * wip. * wip. * Fix camera rotation (#92) Use UIInterfaceOrientation instead of UIDeviceOrientation for RTCVideoFrame's rotation * done. * fix * update. * Expose audio sample buffers for Android (#89) * Initial draft * Working impl * doc and cleanup * doc update * add SetSifTrailer. * fix h264 freeze. --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: davidliu <davidliu@deviange.net>
cloudwebrtc added a commit that referenced this pull request May 21, 2024
allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) Allow custom audio processing by exposing AudioProcessingModule (#85) Expose audio sample buffers for Android (#89) feat: add external audio processor for android. (#103) android: make audio output attributes modifiable (#118) Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net>
cloudwebrtc added a commit that referenced this pull request May 21, 2024
allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) Allow custom audio processing by exposing AudioProcessingModule (#85) Expose audio sample buffers for Android (#89) feat: add external audio processor for android. (#103) android: make audio output attributes modifiable (#118) Fix external audio processor sample rate calculation (#108) Expose remote audio sample buffers on RTCAudioTrack (#84) Fix memory leak when creating audio CMSampleBuffer #86 Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net>
Merged
cloudwebrtc added a commit that referenced this pull request Jun 12, 2024
Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization 7454824 * allow listen-only mode in AudioUnit, adjust when category changes (#2) * release mic when category changes (#5) * Change defaults to iOS defaults (#7) * Sync audio session config (#8) * feat: support bypass voice processing for iOS. (#15) * Remove MacBookPro audio pan right code (#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (#29) * feat: add audio device changes detect for windows. (#41) * fix Linux compile (#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) * Stop recording on mute (turn off mic indicator) (#55) * Cherry pick audio selection from m97 release (#35) * [Mac] Allow audio device selection (#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) * Allow custom audio processing by exposing AudioProcessingModule (#85) * Expose audio sample buffers for Android (#89) * feat: add external audio processor for android. (#103) * android: make audio output attributes modifiable (#118) * Fix external audio processor sample rate calculation (#108) * Expose remote audio sample buffers on RTCAudioTrack (#84) * Fix memory leak when creating audio CMSampleBuffer #86 ## 3. Simulcast/SVC support for iOS/Android. b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. 9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. 841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com>
santhoshvai pushed a commit to GetStream/webrtc that referenced this pull request Nov 20, 2024
Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md webrtc-sdk/webrtc@b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization webrtc-sdk/webrtc@7454824 * allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk/webrtc#2) * release mic when category changes (webrtc-sdk/webrtc#5) * Change defaults to iOS defaults (webrtc-sdk/webrtc#7) * Sync audio session config (webrtc-sdk/webrtc#8) * feat: support bypass voice processing for iOS. (webrtc-sdk/webrtc#15) * Remove MacBookPro audio pan right code (webrtc-sdk/webrtc#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk/webrtc#29) * feat: add audio device changes detect for windows. (webrtc-sdk/webrtc#41) * fix Linux compile (webrtc-sdk/webrtc#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk/webrtc#52) * Stop recording on mute (turn off mic indicator) (webrtc-sdk/webrtc#55) * Cherry pick audio selection from m97 release (webrtc-sdk/webrtc#35) * [Mac] Allow audio device selection (webrtc-sdk/webrtc#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk/webrtc#80) * Allow custom audio processing by exposing AudioProcessingModule (webrtc-sdk/webrtc#85) * Expose audio sample buffers for Android (webrtc-sdk/webrtc#89) * feat: add external audio processor for android. (webrtc-sdk/webrtc#103) * android: make audio output attributes modifiable (webrtc-sdk/webrtc#118) * Fix external audio processor sample rate calculation (webrtc-sdk/webrtc#108) * Expose remote audio sample buffers on RTCAudioTrack (webrtc-sdk/webrtc#84) * Fix memory leak when creating audio CMSampleBuffer webrtc-sdk/webrtc#86 ## 3. Simulcast/SVC support for iOS/Android. webrtc-sdk/webrtc@b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. webrtc-sdk/webrtc@9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements webrtc-sdk/webrtc@a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. webrtc-sdk/webrtc@841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. webrtc-sdk/webrtc@fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. webrtc-sdk/webrtc@eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com> # Conflicts: # README.md # media/engine/webrtc_video_engine.cc # media/engine/webrtc_video_engine.h # modules/audio_device/audio_device_impl.cc # sdk/BUILD.gn # sdk/android/BUILD.gn # sdk/android/api/org/webrtc/RtpParameters.java # sdk/android/api/org/webrtc/SimulcastVideoEncoder.java # sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java # sdk/android/api/org/webrtc/VideoCodecInfo.java # sdk/android/src/jni/pc/rtp_parameters.cc # sdk/android/src/jni/simulcast_video_encoder.cc # sdk/android/src/jni/simulcast_video_encoder.h # sdk/android/src/jni/video_codec_info.cc # sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm # sdk/objc/api/peerconnection/RTCAudioTrack.mm # sdk/objc/api/peerconnection/RTCIODevice+Private.h # sdk/objc/api/peerconnection/RTCIODevice.mm # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm # sdk/objc/base/RTCAudioRenderer.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm
kanat pushed a commit to GetStream/webrtc that referenced this pull request Nov 22, 2024
* Update to m125. (#119) Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md webrtc-sdk/webrtc@b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization webrtc-sdk/webrtc@7454824 * allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk/webrtc#2) * release mic when category changes (webrtc-sdk/webrtc#5) * Change defaults to iOS defaults (webrtc-sdk/webrtc#7) * Sync audio session config (webrtc-sdk/webrtc#8) * feat: support bypass voice processing for iOS. (webrtc-sdk/webrtc#15) * Remove MacBookPro audio pan right code (webrtc-sdk/webrtc#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk/webrtc#29) * feat: add audio device changes detect for windows. (webrtc-sdk/webrtc#41) * fix Linux compile (webrtc-sdk/webrtc#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk/webrtc#52) * Stop recording on mute (turn off mic indicator) (webrtc-sdk/webrtc#55) * Cherry pick audio selection from m97 release (webrtc-sdk/webrtc#35) * [Mac] Allow audio device selection (webrtc-sdk/webrtc#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk/webrtc#80) * Allow custom audio processing by exposing AudioProcessingModule (webrtc-sdk/webrtc#85) * Expose audio sample buffers for Android (webrtc-sdk/webrtc#89) * feat: add external audio processor for android. (webrtc-sdk/webrtc#103) * android: make audio output attributes modifiable (webrtc-sdk/webrtc#118) * Fix external audio processor sample rate calculation (webrtc-sdk/webrtc#108) * Expose remote audio sample buffers on RTCAudioTrack (webrtc-sdk/webrtc#84) * Fix memory leak when creating audio CMSampleBuffer webrtc-sdk/webrtc#86 ## 3. Simulcast/SVC support for iOS/Android. webrtc-sdk/webrtc@b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. webrtc-sdk/webrtc@9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements webrtc-sdk/webrtc@a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. webrtc-sdk/webrtc@841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. webrtc-sdk/webrtc@fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. webrtc-sdk/webrtc@eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com> # Conflicts: # README.md # media/engine/webrtc_video_engine.cc # media/engine/webrtc_video_engine.h # modules/audio_device/audio_device_impl.cc # sdk/BUILD.gn # sdk/android/BUILD.gn # sdk/android/api/org/webrtc/RtpParameters.java # sdk/android/api/org/webrtc/SimulcastVideoEncoder.java # sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java # sdk/android/api/org/webrtc/VideoCodecInfo.java # sdk/android/src/jni/pc/rtp_parameters.cc # sdk/android/src/jni/simulcast_video_encoder.cc # sdk/android/src/jni/simulcast_video_encoder.h # sdk/android/src/jni/video_codec_info.cc # sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm # sdk/objc/api/peerconnection/RTCAudioTrack.mm # sdk/objc/api/peerconnection/RTCIODevice+Private.h # sdk/objc/api/peerconnection/RTCIODevice.mm # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm # sdk/objc/base/RTCAudioRenderer.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm * fix: duplicate simulcast entries * remove duplicate declaration * remove duplicate audioDeviceModule * fix: removed livekit's external audio processor * fix: add back simulcast factories * Fix missing RTC_OBJC_TYPE macros * Fix missing headers and Metal linking # Conflicts: # sdk/BUILD.gn * Fix Mac Catalyst `RTCCameraVideoCapturer` rotation (#126) * Fix set frame transformer (#125) * Fix webrtc_voice_engine not notifying mute change (#128) Looks like this line was missed during the m125 update. webrtc-sdk/webrtc@272127d#diff-56f5e0c459b287281ef3b0431d3f4129e8e4be4c6955d845bcb22210f08b7ba5R2289 Adding it back in so that mic is properly released when muted. # Conflicts: # media/engine/webrtc_voice_engine.cc * android: Allow for skipping checking the audio playstate if needed (#129) Pausing/stopping the audio track can lead to a race condition against the AudioTrackThread due to this assert. Normally this is fine since directly pausing/stopping isn't possible, but user is using reflection to workaround another audio issue (muted participants still have a sending audio stream which keeps the audio alive, affecting global sound if in the background). Not a full fix, as would like to manually control the audio track directly (needs a bigger fix to handle proper synchronization before allowing public access), but this will work through reflection (user takes responsibility for usage). * Allow to pass in capture session to RTCCameraVideoCapturer (#132) Expose initializers to pass in capture session to RTCCameraVideoCapturer so we can use AVCaptureMultiCamSession etc to capture front and back simultaneously for iOS. * Fix NetworkMonitor race condition when dispatching native observers (#135) There is a race condition in NetworkMonitor where native observers may be removed concurrently with a notification being dispatched, leading to a dangling pointer dereference (trying to dispatch an observer that was already removed and destroyed), and from there a crash with access violation. By ensuring dispatching to native observers is done within the synchronization lock that guards additions/removals of native observers protects against this race condition. Since native observers callbacks are posted to the networking thread in the C++ side anyway, there should be no risk of deadlock/starvation due to long-running observers. Bug: webrtc:15837 Change-Id: Id2b788f102dbd25de76ceed434c4cd68aa9a569e Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/338643 Reviewed-by: Taylor Brandstetter <deadbeef@webrtc.org> Commit-Queue: Harald Alvestrand <hta@webrtc.org> Reviewed-by: Harald Alvestrand <hta@webrtc.org> Cr-Commit-Position: refs/heads/main@{#42256} Co-authored-by: Guy Hershenbaum <hershi@meta.com> * Support for Vision Pro (#131) TODO: - [x] fix compile for RTCCameraVideoCapturer - [ ] fix RTCMTLRenderer ? --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> * Multicam support (#137) TODO: - [x] Return `.systemPreferredCamera` for devices (visionOS only). - [x] Use `AVCaptureMultiCamSession` only if `isMultiCamSupported` is true. - [x] Silence statusBarOrientation warning. --------- Co-authored-by: duanweiwei1982@gmail.com <duanweiwei1982@gmail.com> * tvOS support (#139) 17.0+ only atm --------- Co-authored-by: cloudwebrtc <duanweiwei1982@gmail.com> * Add isDisposed to MediaStreamTrack (#140) * chore: handle invalid cipher from key size. (#142) * Allow software AEC for Simulator (#143) ~Allow to use "googEchoCancellation" constraint for software AEC. For devices "googEchoCancellation" should be false to use VoiceProcessingIO.~ * Fix AudioRenderer crash & expose AVAudioPCMBuffer (#144) * fix: Fix bug for bypass voice processing. (#147) * chore: remove aes cbc for framecryptor. (#145) * Change audio renderer output format (#149) Instead of converting to Float, output original Int data without conversion. Output the raw format and convert when required. * Fixed issue with missing network interfaces on iOS (#151) Related issue: webrtc-sdk/webrtc#148 Cherry-pick : https://webrtc.googlesource.com/src/+/fea60ef8e72fb17b4f8a5363aff7e63ab8027b4f Fixed issue with network interfaces due to a missing return value in the "nw_path_enumerate_interfaces(...)" block. Exposed in iOS 18, RTCNetworkMonitor::initWithObserver will only enumerate the first interface, instead of all device interfaces Bug: webrtc:359245764 Change-Id: Ifb9f28c33306c0096476a4afb0cdb4d734e87b2c Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/359541 Auto-Submit: Corby <corby.hoback@gmail.com> Commit-Queue: Jonas Oreland <jonaso@webrtc.org> Reviewed-by: Kári Helgason <kthelgason@webrtc.org> Reviewed-by: Jonas Oreland <jonaso@webrtc.org> Cr-Commit-Position: refs/heads/main@{#42818} Co-authored-by: Corby Hoback <corby.hoback@gmail.com> * Custom audio input for Android (#154) # Conflicts: # sdk/android/api/org/webrtc/audio/JavaAudioDeviceModule.java # sdk/android/src/java/org/webrtc/audio/WebRtcAudioRecord.java --------- Co-authored-by: CloudWebRTC <duanweiwei1982@gmail.com> Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Guy Hershenbaum <hershi@meta.com> Co-authored-by: Corby Hoback <corby.hoback@gmail.com>
pblazej pushed a commit that referenced this pull request Jun 12, 2025
allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) Allow custom audio processing by exposing AudioProcessingModule (#85) Expose audio sample buffers for Android (#89) feat: add external audio processor for android. (#103) android: make audio output attributes modifiable (#118) Fix external audio processor sample rate calculation (#108) Expose remote audio sample buffers on RTCAudioTrack (#84) Fix memory leak when creating audio CMSampleBuffer #86 Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> (cherry picked from commit 7454824) # Conflicts: # audio/audio_state.cc # call/audio_state.h # media/engine/webrtc_voice_engine.h # modules/audio_device/audio_device_impl.cc # modules/audio_device/include/audio_device.h # modules/audio_device/mac/audio_device_mac.cc # modules/audio_device/mac/audio_device_mac.h # sdk/objc/api/peerconnection/RTCAudioTrack+Private.h # sdk/objc/api/peerconnection/RTCPeerConnectionFactory+Native.h # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm # sdk/objc/api/peerconnection/RTCPeerConnectionFactoryBuilder.mm # sdk/objc/components/audio/RTCAudioSessionConfiguration.m # sdk/objc/native/src/audio/audio_device_ios.mm # sdk/objc/native/src/audio/audio_device_module_ios.mm # sdk/objc/native/src/audio/voice_processing_audio_unit.mm
ipavlidakis pushed a commit to GetStream/webrtc that referenced this pull request Jul 28, 2025
* Update to m125. (#119) Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md webrtc-sdk/webrtc@b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization webrtc-sdk/webrtc@7454824 * allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk/webrtc#2) * release mic when category changes (webrtc-sdk/webrtc#5) * Change defaults to iOS defaults (webrtc-sdk/webrtc#7) * Sync audio session config (webrtc-sdk/webrtc#8) * feat: support bypass voice processing for iOS. (webrtc-sdk/webrtc#15) * Remove MacBookPro audio pan right code (webrtc-sdk/webrtc#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk/webrtc#29) * feat: add audio device changes detect for windows. (webrtc-sdk/webrtc#41) * fix Linux compile (webrtc-sdk/webrtc#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk/webrtc#52) * Stop recording on mute (turn off mic indicator) (webrtc-sdk/webrtc#55) * Cherry pick audio selection from m97 release (webrtc-sdk/webrtc#35) * [Mac] Allow audio device selection (webrtc-sdk/webrtc#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk/webrtc#80) * Allow custom audio processing by exposing AudioProcessingModule (webrtc-sdk/webrtc#85) * Expose audio sample buffers for Android (webrtc-sdk/webrtc#89) * feat: add external audio processor for android. (webrtc-sdk/webrtc#103) * android: make audio output attributes modifiable (webrtc-sdk/webrtc#118) * Fix external audio processor sample rate calculation (webrtc-sdk/webrtc#108) * Expose remote audio sample buffers on RTCAudioTrack (webrtc-sdk/webrtc#84) * Fix memory leak when creating audio CMSampleBuffer webrtc-sdk/webrtc#86 ## 3. Simulcast/SVC support for iOS/Android. webrtc-sdk/webrtc@b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. webrtc-sdk/webrtc@9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements webrtc-sdk/webrtc@a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. webrtc-sdk/webrtc@841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. webrtc-sdk/webrtc@fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. webrtc-sdk/webrtc@eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com> # Conflicts: # README.md # media/engine/webrtc_video_engine.cc # media/engine/webrtc_video_engine.h # modules/audio_device/audio_device_impl.cc # sdk/BUILD.gn # sdk/android/BUILD.gn # sdk/android/api/org/webrtc/RtpParameters.java # sdk/android/api/org/webrtc/SimulcastVideoEncoder.java # sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java # sdk/android/api/org/webrtc/VideoCodecInfo.java # sdk/android/src/jni/pc/rtp_parameters.cc # sdk/android/src/jni/simulcast_video_encoder.cc # sdk/android/src/jni/simulcast_video_encoder.h # sdk/android/src/jni/video_codec_info.cc # sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm # sdk/objc/api/peerconnection/RTCAudioTrack.mm # sdk/objc/api/peerconnection/RTCIODevice+Private.h # sdk/objc/api/peerconnection/RTCIODevice.mm # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm # sdk/objc/base/RTCAudioRenderer.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm * fix: duplicate simulcast entries * remove duplicate declaration * remove duplicate audioDeviceModule * fix: removed livekit's external audio processor * fix: add back simulcast factories * Fix missing RTC_OBJC_TYPE macros * Fix missing headers and Metal linking # Conflicts: # sdk/BUILD.gn * Fix Mac Catalyst `RTCCameraVideoCapturer` rotation (#126) * Fix set frame transformer (#125) * Fix webrtc_voice_engine not notifying mute change (#128) Looks like this line was missed during the m125 update. webrtc-sdk/webrtc@272127d#diff-56f5e0c459b287281ef3b0431d3f4129e8e4be4c6955d845bcb22210f08b7ba5R2289 Adding it back in so that mic is properly released when muted. # Conflicts: # media/engine/webrtc_voice_engine.cc * android: Allow for skipping checking the audio playstate if needed (#129) Pausing/stopping the audio track can lead to a race condition against the AudioTrackThread due to this assert. Normally this is fine since directly pausing/stopping isn't possible, but user is using reflection to workaround another audio issue (muted participants still have a sending audio stream which keeps the audio alive, affecting global sound if in the background). Not a full fix, as would like to manually control the audio track directly (needs a bigger fix to handle proper synchronization before allowing public access), but this will work through reflection (user takes responsibility for usage). * Allow to pass in capture session to RTCCameraVideoCapturer (#132) Expose initializers to pass in capture session to RTCCameraVideoCapturer so we can use AVCaptureMultiCamSession etc to capture front and back simultaneously for iOS. * Fix NetworkMonitor race condition when dispatching native observers (#135) There is a race condition in NetworkMonitor where native observers may be removed concurrently with a notification being dispatched, leading to a dangling pointer dereference (trying to dispatch an observer that was already removed and destroyed), and from there a crash with access violation. By ensuring dispatching to native observers is done within the synchronization lock that guards additions/removals of native observers protects against this race condition. Since native observers callbacks are posted to the networking thread in the C++ side anyway, there should be no risk of deadlock/starvation due to long-running observers. Bug: webrtc:15837 Change-Id: Id2b788f102dbd25de76ceed434c4cd68aa9a569e Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/338643 Reviewed-by: Taylor Brandstetter <deadbeef@webrtc.org> Commit-Queue: Harald Alvestrand <hta@webrtc.org> Reviewed-by: Harald Alvestrand <hta@webrtc.org> Cr-Commit-Position: refs/heads/main@{#42256} Co-authored-by: Guy Hershenbaum <hershi@meta.com> * Support for Vision Pro (#131) TODO: - [x] fix compile for RTCCameraVideoCapturer - [ ] fix RTCMTLRenderer ? --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> * Multicam support (#137) TODO: - [x] Return `.systemPreferredCamera` for devices (visionOS only). - [x] Use `AVCaptureMultiCamSession` only if `isMultiCamSupported` is true. - [x] Silence statusBarOrientation warning. --------- Co-authored-by: duanweiwei1982@gmail.com <duanweiwei1982@gmail.com> * tvOS support (#139) 17.0+ only atm --------- Co-authored-by: cloudwebrtc <duanweiwei1982@gmail.com> * Add isDisposed to MediaStreamTrack (#140) * chore: handle invalid cipher from key size. (#142) * Allow software AEC for Simulator (#143) ~Allow to use "googEchoCancellation" constraint for software AEC. For devices "googEchoCancellation" should be false to use VoiceProcessingIO.~ * Fix AudioRenderer crash & expose AVAudioPCMBuffer (#144) * fix: Fix bug for bypass voice processing. (#147) * chore: remove aes cbc for framecryptor. (#145) * Change audio renderer output format (#149) Instead of converting to Float, output original Int data without conversion. Output the raw format and convert when required. * Fixed issue with missing network interfaces on iOS (#151) Related issue: webrtc-sdk/webrtc#148 Cherry-pick : https://webrtc.googlesource.com/src/+/fea60ef8e72fb17b4f8a5363aff7e63ab8027b4f Fixed issue with network interfaces due to a missing return value in the "nw_path_enumerate_interfaces(...)" block. Exposed in iOS 18, RTCNetworkMonitor::initWithObserver will only enumerate the first interface, instead of all device interfaces Bug: webrtc:359245764 Change-Id: Ifb9f28c33306c0096476a4afb0cdb4d734e87b2c Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/359541 Auto-Submit: Corby <corby.hoback@gmail.com> Commit-Queue: Jonas Oreland <jonaso@webrtc.org> Reviewed-by: Kári Helgason <kthelgason@webrtc.org> Reviewed-by: Jonas Oreland <jonaso@webrtc.org> Cr-Commit-Position: refs/heads/main@{#42818} Co-authored-by: Corby Hoback <corby.hoback@gmail.com> * Custom audio input for Android (#154) # Conflicts: # sdk/android/api/org/webrtc/audio/JavaAudioDeviceModule.java # sdk/android/src/java/org/webrtc/audio/WebRtcAudioRecord.java --------- Co-authored-by: CloudWebRTC <duanweiwei1982@gmail.com> Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Guy Hershenbaum <hershi@meta.com> Co-authored-by: Corby Hoback <corby.hoback@gmail.com>
ipavlidakis pushed a commit to GetStream/webrtc that referenced this pull request Jul 29, 2025
* Update to m125. (#119) Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md webrtc-sdk/webrtc@b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization webrtc-sdk/webrtc@7454824 * allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk/webrtc#2) * release mic when category changes (webrtc-sdk/webrtc#5) * Change defaults to iOS defaults (webrtc-sdk/webrtc#7) * Sync audio session config (webrtc-sdk/webrtc#8) * feat: support bypass voice processing for iOS. (webrtc-sdk/webrtc#15) * Remove MacBookPro audio pan right code (webrtc-sdk/webrtc#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk/webrtc#29) * feat: add audio device changes detect for windows. (webrtc-sdk/webrtc#41) * fix Linux compile (webrtc-sdk/webrtc#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk/webrtc#52) * Stop recording on mute (turn off mic indicator) (webrtc-sdk/webrtc#55) * Cherry pick audio selection from m97 release (webrtc-sdk/webrtc#35) * [Mac] Allow audio device selection (webrtc-sdk/webrtc#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk/webrtc#80) * Allow custom audio processing by exposing AudioProcessingModule (webrtc-sdk/webrtc#85) * Expose audio sample buffers for Android (webrtc-sdk/webrtc#89) * feat: add external audio processor for android. (webrtc-sdk/webrtc#103) * android: make audio output attributes modifiable (webrtc-sdk/webrtc#118) * Fix external audio processor sample rate calculation (webrtc-sdk/webrtc#108) * Expose remote audio sample buffers on RTCAudioTrack (webrtc-sdk/webrtc#84) * Fix memory leak when creating audio CMSampleBuffer webrtc-sdk/webrtc#86 ## 3. Simulcast/SVC support for iOS/Android. webrtc-sdk/webrtc@b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. webrtc-sdk/webrtc@9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements webrtc-sdk/webrtc@a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. webrtc-sdk/webrtc@841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. webrtc-sdk/webrtc@fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. webrtc-sdk/webrtc@eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com> # Conflicts: # README.md # media/engine/webrtc_video_engine.cc # media/engine/webrtc_video_engine.h # modules/audio_device/audio_device_impl.cc # sdk/BUILD.gn # sdk/android/BUILD.gn # sdk/android/api/org/webrtc/RtpParameters.java # sdk/android/api/org/webrtc/SimulcastVideoEncoder.java # sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java # sdk/android/api/org/webrtc/VideoCodecInfo.java # sdk/android/src/jni/pc/rtp_parameters.cc # sdk/android/src/jni/simulcast_video_encoder.cc # sdk/android/src/jni/simulcast_video_encoder.h # sdk/android/src/jni/video_codec_info.cc # sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm # sdk/objc/api/peerconnection/RTCAudioTrack.mm # sdk/objc/api/peerconnection/RTCIODevice+Private.h # sdk/objc/api/peerconnection/RTCIODevice.mm # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm # sdk/objc/base/RTCAudioRenderer.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm * fix: duplicate simulcast entries * remove duplicate declaration * remove duplicate audioDeviceModule * fix: removed livekit's external audio processor * fix: add back simulcast factories * Fix missing RTC_OBJC_TYPE macros * Fix missing headers and Metal linking # Conflicts: # sdk/BUILD.gn * Fix Mac Catalyst `RTCCameraVideoCapturer` rotation (#126) * Fix set frame transformer (#125) * Fix webrtc_voice_engine not notifying mute change (#128) Looks like this line was missed during the m125 update. webrtc-sdk/webrtc@272127d#diff-56f5e0c459b287281ef3b0431d3f4129e8e4be4c6955d845bcb22210f08b7ba5R2289 Adding it back in so that mic is properly released when muted. # Conflicts: # media/engine/webrtc_voice_engine.cc * android: Allow for skipping checking the audio playstate if needed (#129) Pausing/stopping the audio track can lead to a race condition against the AudioTrackThread due to this assert. Normally this is fine since directly pausing/stopping isn't possible, but user is using reflection to workaround another audio issue (muted participants still have a sending audio stream which keeps the audio alive, affecting global sound if in the background). Not a full fix, as would like to manually control the audio track directly (needs a bigger fix to handle proper synchronization before allowing public access), but this will work through reflection (user takes responsibility for usage). * Allow to pass in capture session to RTCCameraVideoCapturer (#132) Expose initializers to pass in capture session to RTCCameraVideoCapturer so we can use AVCaptureMultiCamSession etc to capture front and back simultaneously for iOS. * Fix NetworkMonitor race condition when dispatching native observers (#135) There is a race condition in NetworkMonitor where native observers may be removed concurrently with a notification being dispatched, leading to a dangling pointer dereference (trying to dispatch an observer that was already removed and destroyed), and from there a crash with access violation. By ensuring dispatching to native observers is done within the synchronization lock that guards additions/removals of native observers protects against this race condition. Since native observers callbacks are posted to the networking thread in the C++ side anyway, there should be no risk of deadlock/starvation due to long-running observers. Bug: webrtc:15837 Change-Id: Id2b788f102dbd25de76ceed434c4cd68aa9a569e Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/338643 Reviewed-by: Taylor Brandstetter <deadbeef@webrtc.org> Commit-Queue: Harald Alvestrand <hta@webrtc.org> Reviewed-by: Harald Alvestrand <hta@webrtc.org> Cr-Commit-Position: refs/heads/main@{#42256} Co-authored-by: Guy Hershenbaum <hershi@meta.com> * Support for Vision Pro (#131) TODO: - [x] fix compile for RTCCameraVideoCapturer - [ ] fix RTCMTLRenderer ? --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> * Multicam support (#137) TODO: - [x] Return `.systemPreferredCamera` for devices (visionOS only). - [x] Use `AVCaptureMultiCamSession` only if `isMultiCamSupported` is true. - [x] Silence statusBarOrientation warning. --------- Co-authored-by: duanweiwei1982@gmail.com <duanweiwei1982@gmail.com> * tvOS support (#139) 17.0+ only atm --------- Co-authored-by: cloudwebrtc <duanweiwei1982@gmail.com> * Add isDisposed to MediaStreamTrack (#140) * chore: handle invalid cipher from key size. (#142) * Allow software AEC for Simulator (#143) ~Allow to use "googEchoCancellation" constraint for software AEC. For devices "googEchoCancellation" should be false to use VoiceProcessingIO.~ * Fix AudioRenderer crash & expose AVAudioPCMBuffer (#144) * fix: Fix bug for bypass voice processing. (#147) * chore: remove aes cbc for framecryptor. (#145) * Change audio renderer output format (#149) Instead of converting to Float, output original Int data without conversion. Output the raw format and convert when required. * Fixed issue with missing network interfaces on iOS (#151) Related issue: webrtc-sdk/webrtc#148 Cherry-pick : https://webrtc.googlesource.com/src/+/fea60ef8e72fb17b4f8a5363aff7e63ab8027b4f Fixed issue with network interfaces due to a missing return value in the "nw_path_enumerate_interfaces(...)" block. Exposed in iOS 18, RTCNetworkMonitor::initWithObserver will only enumerate the first interface, instead of all device interfaces Bug: webrtc:359245764 Change-Id: Ifb9f28c33306c0096476a4afb0cdb4d734e87b2c Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/359541 Auto-Submit: Corby <corby.hoback@gmail.com> Commit-Queue: Jonas Oreland <jonaso@webrtc.org> Reviewed-by: Kári Helgason <kthelgason@webrtc.org> Reviewed-by: Jonas Oreland <jonaso@webrtc.org> Cr-Commit-Position: refs/heads/main@{#42818} Co-authored-by: Corby Hoback <corby.hoback@gmail.com> * Custom audio input for Android (#154) # Conflicts: # sdk/android/api/org/webrtc/audio/JavaAudioDeviceModule.java # sdk/android/src/java/org/webrtc/audio/WebRtcAudioRecord.java --------- Co-authored-by: CloudWebRTC <duanweiwei1982@gmail.com> Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Guy Hershenbaum <hershi@meta.com> Co-authored-by: Corby Hoback <corby.hoback@gmail.com>
ipavlidakis pushed a commit to GetStream/webrtc that referenced this pull request Sep 11, 2025
* Update to m125. (#119) Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md webrtc-sdk/webrtc@b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization webrtc-sdk/webrtc@7454824 * allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk/webrtc#2) * release mic when category changes (webrtc-sdk/webrtc#5) * Change defaults to iOS defaults (webrtc-sdk/webrtc#7) * Sync audio session config (webrtc-sdk/webrtc#8) * feat: support bypass voice processing for iOS. (webrtc-sdk/webrtc#15) * Remove MacBookPro audio pan right code (webrtc-sdk/webrtc#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk/webrtc#29) * feat: add audio device changes detect for windows. (webrtc-sdk/webrtc#41) * fix Linux compile (webrtc-sdk/webrtc#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk/webrtc#52) * Stop recording on mute (turn off mic indicator) (webrtc-sdk/webrtc#55) * Cherry pick audio selection from m97 release (webrtc-sdk/webrtc#35) * [Mac] Allow audio device selection (webrtc-sdk/webrtc#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk/webrtc#80) * Allow custom audio processing by exposing AudioProcessingModule (webrtc-sdk/webrtc#85) * Expose audio sample buffers for Android (webrtc-sdk/webrtc#89) * feat: add external audio processor for android. (webrtc-sdk/webrtc#103) * android: make audio output attributes modifiable (webrtc-sdk/webrtc#118) * Fix external audio processor sample rate calculation (webrtc-sdk/webrtc#108) * Expose remote audio sample buffers on RTCAudioTrack (webrtc-sdk/webrtc#84) * Fix memory leak when creating audio CMSampleBuffer webrtc-sdk/webrtc#86 ## 3. Simulcast/SVC support for iOS/Android. webrtc-sdk/webrtc@b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. webrtc-sdk/webrtc@9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements webrtc-sdk/webrtc@a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. webrtc-sdk/webrtc@841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. webrtc-sdk/webrtc@fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. webrtc-sdk/webrtc@eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com> # Conflicts: # README.md # media/engine/webrtc_video_engine.cc # media/engine/webrtc_video_engine.h # modules/audio_device/audio_device_impl.cc # sdk/BUILD.gn # sdk/android/BUILD.gn # sdk/android/api/org/webrtc/RtpParameters.java # sdk/android/api/org/webrtc/SimulcastVideoEncoder.java # sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java # sdk/android/api/org/webrtc/VideoCodecInfo.java # sdk/android/src/jni/pc/rtp_parameters.cc # sdk/android/src/jni/simulcast_video_encoder.cc # sdk/android/src/jni/simulcast_video_encoder.h # sdk/android/src/jni/video_codec_info.cc # sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm # sdk/objc/api/peerconnection/RTCAudioTrack.mm # sdk/objc/api/peerconnection/RTCIODevice+Private.h # sdk/objc/api/peerconnection/RTCIODevice.mm # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm # sdk/objc/base/RTCAudioRenderer.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm * fix: duplicate simulcast entries * remove duplicate declaration * remove duplicate audioDeviceModule * fix: removed livekit's external audio processor * fix: add back simulcast factories * Fix missing RTC_OBJC_TYPE macros * Fix missing headers and Metal linking # Conflicts: # sdk/BUILD.gn * Fix Mac Catalyst `RTCCameraVideoCapturer` rotation (#126) * Fix set frame transformer (#125) * Fix webrtc_voice_engine not notifying mute change (#128) Looks like this line was missed during the m125 update. webrtc-sdk/webrtc@272127d#diff-56f5e0c459b287281ef3b0431d3f4129e8e4be4c6955d845bcb22210f08b7ba5R2289 Adding it back in so that mic is properly released when muted. # Conflicts: # media/engine/webrtc_voice_engine.cc * android: Allow for skipping checking the audio playstate if needed (#129) Pausing/stopping the audio track can lead to a race condition against the AudioTrackThread due to this assert. Normally this is fine since directly pausing/stopping isn't possible, but user is using reflection to workaround another audio issue (muted participants still have a sending audio stream which keeps the audio alive, affecting global sound if in the background). Not a full fix, as would like to manually control the audio track directly (needs a bigger fix to handle proper synchronization before allowing public access), but this will work through reflection (user takes responsibility for usage). * Allow to pass in capture session to RTCCameraVideoCapturer (#132) Expose initializers to pass in capture session to RTCCameraVideoCapturer so we can use AVCaptureMultiCamSession etc to capture front and back simultaneously for iOS. * Fix NetworkMonitor race condition when dispatching native observers (#135) There is a race condition in NetworkMonitor where native observers may be removed concurrently with a notification being dispatched, leading to a dangling pointer dereference (trying to dispatch an observer that was already removed and destroyed), and from there a crash with access violation. By ensuring dispatching to native observers is done within the synchronization lock that guards additions/removals of native observers protects against this race condition. Since native observers callbacks are posted to the networking thread in the C++ side anyway, there should be no risk of deadlock/starvation due to long-running observers. Bug: webrtc:15837 Change-Id: Id2b788f102dbd25de76ceed434c4cd68aa9a569e Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/338643 Reviewed-by: Taylor Brandstetter <deadbeef@webrtc.org> Commit-Queue: Harald Alvestrand <hta@webrtc.org> Reviewed-by: Harald Alvestrand <hta@webrtc.org> Cr-Commit-Position: refs/heads/main@{#42256} Co-authored-by: Guy Hershenbaum <hershi@meta.com> * Support for Vision Pro (#131) TODO: - [x] fix compile for RTCCameraVideoCapturer - [ ] fix RTCMTLRenderer ? --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> * Multicam support (#137) TODO: - [x] Return `.systemPreferredCamera` for devices (visionOS only). - [x] Use `AVCaptureMultiCamSession` only if `isMultiCamSupported` is true. - [x] Silence statusBarOrientation warning. --------- Co-authored-by: duanweiwei1982@gmail.com <duanweiwei1982@gmail.com> * tvOS support (#139) 17.0+ only atm --------- Co-authored-by: cloudwebrtc <duanweiwei1982@gmail.com> * Add isDisposed to MediaStreamTrack (#140) * chore: handle invalid cipher from key size. (#142) * Allow software AEC for Simulator (#143) ~Allow to use "googEchoCancellation" constraint for software AEC. For devices "googEchoCancellation" should be false to use VoiceProcessingIO.~ * Fix AudioRenderer crash & expose AVAudioPCMBuffer (#144) * fix: Fix bug for bypass voice processing. (#147) * chore: remove aes cbc for framecryptor. (#145) * Change audio renderer output format (#149) Instead of converting to Float, output original Int data without conversion. Output the raw format and convert when required. * Fixed issue with missing network interfaces on iOS (#151) Related issue: webrtc-sdk/webrtc#148 Cherry-pick : https://webrtc.googlesource.com/src/+/fea60ef8e72fb17b4f8a5363aff7e63ab8027b4f Fixed issue with network interfaces due to a missing return value in the "nw_path_enumerate_interfaces(...)" block. Exposed in iOS 18, RTCNetworkMonitor::initWithObserver will only enumerate the first interface, instead of all device interfaces Bug: webrtc:359245764 Change-Id: Ifb9f28c33306c0096476a4afb0cdb4d734e87b2c Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/359541 Auto-Submit: Corby <corby.hoback@gmail.com> Commit-Queue: Jonas Oreland <jonaso@webrtc.org> Reviewed-by: Kári Helgason <kthelgason@webrtc.org> Reviewed-by: Jonas Oreland <jonaso@webrtc.org> Cr-Commit-Position: refs/heads/main@{#42818} Co-authored-by: Corby Hoback <corby.hoback@gmail.com> * Custom audio input for Android (#154) # Conflicts: # sdk/android/api/org/webrtc/audio/JavaAudioDeviceModule.java # sdk/android/src/java/org/webrtc/audio/WebRtcAudioRecord.java --------- Co-authored-by: CloudWebRTC <duanweiwei1982@gmail.com> Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Guy Hershenbaum <hershi@meta.com> Co-authored-by: Corby Hoback <corby.hoback@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
No description provided.