Steps to reproduce
We are developing a MCU application which can pass audio/video WebRTC traffic from various Browsers.
MCU is a linux app and after we upgraded to openssl 1.0.2 we hardly discovered that EDGE keeps failes to establish DTLS with error:
DTLS handshake failed hr=c004e00f
After investigating the issue we saw that EDGE keeps sending ClientHello and it seems to doesn’t like something in the ServerHello packet (ServerHello + Certificate + Server Key Exchange + Certificate Request + Server Hello Done) from our server. The same scenario (same certificate, dtls 1.0, etc) worked before upgrading from openssl 1.0.1e to 1.0.2.
Investigating deeper we found that EDGE doesn’t like the way our certificate is fragmented (in openssl 1.0.1e this wasn’t fragmented). We use a 4096 RSA self signed key and certificate which has around 900 bytes of data in the ServerHello packet.
If we use a smaller certificate (2048 RSA self signed) - the fragmentation is still there but there are less fragments and it seems to work.
It works again if we use an ECDSA certificate (which is considerable smaller).
Therefore we assumed that the problem comes from certificate re-assembly in your SChannel library.
The issue is still present with DTLS 1.2
We attached some pcap s in which it doesn’t work and some in which it works.
We look forward to your answer as this issue has a big impact on our business (we cannot force clients to downgrade certificates or request ECDSA one).
Comments and activity
In the pcap files: 10.150.67.4 is our MCU, 192.168.111.5 is EDGE. There are 2 streams (audio + video).
- Microsoft Edge Team
Changed Assigned To to “Steven K.”
Thank you for sending the pcap file. We will take a look. I will also check against any known issues.
This appears to be a known issue with OpenSSL and fragmentation. Can you confirm this is the same issue you are seeing?
One user mentioned that changing the MTU size corrected the issue. I am aware that is not a viable workaround, but it does point to the fragmentation of the ServerHello being the issue.
Thank you for your quick answer. We had already tried with MTU of 1350 bytes and it doesn’t work either. I don’t remember exactly if the number of fragments was smaller of not, but it didn’t work.
In our case, Edge is client and as far as I know you don’t use openSSL in EDGE.
Moreover, with the same certificates and fragmentation all other browsers works well (Chrome, Firefox, Opera, Safari).
We look forward to know if this issue will be fixed in the next EDGE release.
The way I am reading the network trace is that the “Server Hello” message from the linux server, which had the recent openSSL upgrade, has an issue, at least according to WireShark. WireShark shows "Reassembly error, protocol DTLS: New fragment overlaps old data". I have attached a screenshot. The confusing part, is that this is working in other browsers. Pointing the issue at Edge or the Windows side.
Edge (192.168.111.5) —Client Hello—> Linux Server (10.150.67.4)
// Linux Server understood the Client Hello and then responds
Edge (192.168.111.5) <—Server Hello— Linux Server (10.150.67.4)
// packet not understood
Can you provide a network trace/pcap for the same message exchange between the same Linux server and a browser where this is working properly? I.e. not using the smaller ECDSA certificate but the same as in the failure scenario.
Regarding the MTU size. I think the MTU would have to be around 6k. I see for each stream the combined Length is ~(2258 + 2233) Bytes. Perhaps just enable jumbo frames? I realize this would not be an ideal final solution.
We will continue to investigate and thank you for extra help,
Regarding Wireshark, there is a known issue that it doesn’t handle epochs well
The packet you see with “Reassembly error, protocol DTLS: New fragment overlaps old data” contains the same data as the first one (first Server Hello). You can see this if you select that packet and save it to a different pcap file and than open the pcap file separately. Wireshark will show that packet correct.
The size of this re-transmission is smaller as openssl decide to put less fragments in Certificate and Server Key exchange (one less fragment for each one) saving packetization space (13 + 12 bytes (headers) = 25 bytes per fragment). The re-transmission is 50 bytes smaller than the initial packet. But it is the same packet (different fragmentation).
However there are still too many fragments for EDGE to handle it correctly.
Regarding MTU - we will try it but there is not a viable solution as you said as the application is not a SaS and it will be deployed in client environment.
Regarding the trace when it works with 4096 RSA certficate - it will be with openssl 1.0.1e and is not fragmented at all.