Negotiating codecs when sender supports multiple options

[wkozyra95]

I want to send media over WHIP. My app supports multiple encoders, so I don't want to enforce a specific codec when creating a track. I want to resolve which codec to use based on the result of the codec negotiation. I would be grateful for any suggestions on how to approach it.

Current scenario with webrtc-rs 0.13

• create peer connection
• create audio and video transceiver
• create offer, send it to WHIP endpoint, receive answer
• set_local_description
• set_remote_description (fails at this point)
• replace both tracks with negotiated values based on RTCRtpSendParameters from RTCRtpSender

Failure scenario: For example, if the server supports only H264, but VP8 is first on our list of codecs (in MediaEngine), then set_remote_description will fail. If H264 is first on the list of codecs, it would work.

It looks like here https://github.com/webrtc-rs/webrtc/blob/master/webrtc/src/peer_connection/peer_connection_internal.rs when creating a transceiver, some codec is already assigned. My understanding is that behavior like that is expected for tracks, but not transceivers.

I'm able to make it work if, before calling set_remote_description, I call replace_track. However, this would require me to parse the offer and answer on my own and resolve what codec should be used. This would basically duplicate a resolution logic that is already in webrtc-rs, so I'm assuming that is not an intended approach.

Current scenario with webrtc-rs 0.11

We have this scenario working with webrtc-rs 0.11

We were able to make it work by:

• create peer connection
• create audio and video tracks
• create offer, send it to WHIP endpoint, receive answer
• set_local_description
• replace both tracks with None
• set_remote_description
• replace both tracks with negotiated values based on RTCRtpSendParameters from RTCRtpSender

In that version, creating tracks was necessary because if we used a transceiver ssrc was not included in the sdp offer. However, this approach does not work in the latest webrtc-rs because setting track track to None clears encodings and final replace_track fails with

webrtc/webrtc/src/error.rs

Line 199 in 6a62754

ErrRTPSenderNewTrackHasIncorrectEnvelope,

[rainliu]

Thanks for reporting this issue.

Do you have any existing example code or tests to reproduce this issue? If not, you may submit an example like https://github.com/webrtc-rs/webrtc/tree/master/examples/examples, so I can reproduce this issue with your example locally and fix it.

[wkozyra95]

Hi I created example here https://github.com/wkozyra95/webrtc/blob/repro_whip_codec_negotiation/examples/examples/whip/whip.rs. It will fail if you stream it to the WHIP server that supports H264, but does not support VP8. However if you move registering vp8 after h264 it would work.

By default, It's connecting to https://g.webrtc.live-video.net:4443/v2/offer (which is a Twitch url) and you need to provide token to run it. However you can switch send_offer with fake_send_offer here to mock WHIP sdp exchange, or test that with any other WHIP server that does not support VP8.

[rainliu]

Add support for multi codec negotiation #741 is merged.

by default, it is enabled.

If you want to disable multi codec negotiation, you can settingEngine.disable_media_engine_multiple_codecs(true);

[rainliu]

@wkozyra95 , please have a try whether this solve your issue. Thanks.

[wkozyra95]

Thanks for quick response. Unfortunately this change does not fix the issue. The example demo I provided still fails.

I'm not to familiar with the codebase, but based on the PR I think your change is making sure that there are multiple options for codecs after negotiation. This change might help in case where there are multiple codecs that both side of connection support.

The issue in my case is that codec is already locked in before negotiation. When transceiver is created it just takes first codec from the MediaEngine list, and if sdp answer does not include that codec then set_remote_description will fail.

[RaffaeleNachos]

Hi, I'm experiencing the same issue. Unfortunately, the PR didn’t fix it for me.

I managed to get it working by creating a custom RTCRtpSender.replace_track implementation that also updates the encoding.
Since, in my scenario, the first call to set_remote_description fails with ErrUnsupportedCodec, I also had to modify RTCPeerConnection.set_remote_description when is_renegotiation to prevent it from starting only RTP
and instead allow it to continue starting DTLS, ICE, and RTP
, as it should have done during the first set_remote_description call.

Here’s my working flow:

1. Create an audio track using codec A and add it to the peer connection.
2. Call create_offer with supported codecs A and B, then send the SDP via signaling.
3. Receive the SDP answer from remote, which includes only codec B, and call set_remote_description -> this fails with ErrUnsupportedCodec.
4. Catch the error and call the modified replace_track which replace codec A with codec B encoding
5. Manually perform renegotiation by calling create_offer again, then set_remote_description on the remote answer. (I have trouble triggering renegotiation automatically when switching codecs inside replace_track. I tried calling trigger_negotiation_needed(), but the peer connection on_negotiation_needed callback is not fired).

At the moment I don’t have a deep understanding of the codebase, so I’m not confident enough to open a PR myself.
Are there any updates or plans from the maintainers regarding this issue?