Referer header incorrectly encoded for XmlHttpRequest

Fixed Issue #15561198

Details

Author
Karl-Johan S.
Created
Jan 22, 2018
Privacy
This issue is public.
Found in
  • Microsoft Edge
Found in build #
41.16299
Fixed in build #
17.17712
Reports
Reported by 4 people

Sign in to watch or report this issue.

Steps to reproduce

We are seeing a problem where the Referer header is encoded wrong for XHR requests when the current url contains non-ascii chars.

The following url illustrates the problem:

https://www.ostersundsbibliotek.se/jämtland-härjedalen?refId=FeAaCH&culture=sv

We load several resources on page load and when inspecting the Referer header we can see that “ä” is encoded differently for XHR and regular requests.

XHR
https://www.ostersundsbibliotek.se/api/search/media-types?no-cache-lang=sv

Referer: https://www.ostersundsbibliotek.se/j%C3%83%C2%A4mtland-h%C3%83%C2%A4rjedalen?refId=FeAaCH&culture=sv

Image
https://www.ostersundsbibliotek.se/svg/snow_hart.svg

Referer: https://www.ostersundsbibliotek.se/j%C3%A4mtland-h%C3%A4rjedalen?refId=FeAaCH&culture=sv

The complete requests from the developer tools are attached as images.

If we however browse to the page with encoded characters in the url it works as expected and both cases are encoded properly.

https://www.ostersundsbibliotek.se/j%C3%A4mtland-h%C3%A4rjedalen?refId=FeAaCH&culture=sv

The main problem that this causes is that the webserver rejects the whole request with a 400 Bad Request error (running ASP.Net Core with Kestrel behind IIS) since the headers are invalid.

Please see other feedbacks in userfeedback VSO for feedbacks that are tracked by this work item.

h3 {
color:#2f5496;
font-family:Segoe UI;
font-size:small;
padding-left:3%;
}

table {
font-size:small;
width:95%;
}

th, td {
font-size:small;
padding:5px;
}

td {
border:0.5px solid white;
}

th {
background-color:#8B8989;
color:white;
}

.feedback-leftcell {
width:20%;
color:#004d8b;
}

.feedback-row:hover {
background-color:#C4C4C4;
}

#feedback-banner, #feedback-footer {
background-color:#0078D7;
color:white;
width:100%;
}

#feedback-banner td, #feedback-footer td {
border:0;
font-size:larger;
padding:10px;
}

#feedbacksection-v5 {
font-family:Segoe UI;
border:5px solid #0078D7;
border-collapse:collapse;
}

#feedbackdetails-table, #feedbackanalysis-table, #feedbacklinks-table {
font-family:Calibri Light;
background-color:#f5f5f5;
}

            
        

        This Bug was created from feedback triage by stepfitz

    



Report Details



    
        Feedback Details

        Description

    

    
        Title

        Edge 400 OPTIONS requests when HTTP request refer header value is unencoded


    

    
        Description

        1. Open web page has an unencoded character in its address - eg https://www.foo.bar?value=español where ñ is unencoded
  1. Observe that any subsequent OPTIONS requests from that page to an API has the HTTP referer request header which contains this source URL with encoded character - eg Referer: https://www.foo.bar?value=español
  2. Observe that other subsequent API calls from the page are 400 in Edge and do not make it out of the browser.
  3. Observe if you encode the URL in the address bar then everything works as expected - eg https://www.foo.bar?value=espa%C3%B1ol

Note: For compat, Chrome auto-encodes HTTP request referers.
Note: Attached a sample Fiddler trace from my internal application where this occurs.

        Area Path

        UIF\Microsoft Edge\Browser crashes or stops working

    

    
        [Feedback Hub](https://aka.ms/feedbackhuburi/?ContextId=343&feedbackId=f6f5d169-5319-4a4f-9d59-ba7411176aed&form=1&src=1)

        View this Customer Feedback details and comments in the Feedback Hub app.

    

    
        [Feedback VSO](../7046540/)

        View this Customer Feedback in Feedback VSO.

    

    
        [Upvoted By](https://aka.ms/feedbackinternalupvoters?vsoId=7046540)

        Contact selfhosters that upvoted this issue.

    

    
            Metadata

            {"Metadata":[{"url":"https://microsoft.visualstudio.com/DefaultCollection/Universal%20Store/SFT-Dublin-Gamma/_workitems/edit/16852897"},{"ua":""},{"browserType":"Microsoft Edge"}]}


          



Analysis and Diagnostics


    
        Tool

        Description

    

    
        [Feedback Cabs](http://aka.ms/FeedbackViewer/?txtUifId=f6f5d169-5319-4a4f-9d59-ba7411176aed&section=Cabs)

        
            
                Looking For CAB files related to feedback?

All CABs for
this feedback can be accessed in the Feedback Viewer.

                Note:

CABs may take up to 12 hours
to process through the
telemetry pipeline. CABs age out and are removed after 60 days
due
to Watson retention policies.

        [Feedback Analysis](http://aka.ms/FeedbackViewer/?txtUifId=f6f5d169-5319-4a4f-9d59-ba7411176aed)

        
            
                Want to understand how this feedback is trending across various pivots?

                With the Feedback Viewer, you can view trending information for this feedback,
                including upvotes by build, region, time, device, and more. Try out the prototype
                version of our [WAAS Lightspeed](https://aka.ms/waaslightspeed/?txtUifId=f6f5d169-5319-4a4f-9d59-ba7411176aed)

that includes intelligent auto analyzed
insights, dimensional, detailed diagnostic and telemetry analysis

        [Related Watson Crashes](http://watson/User?Identifier=g:6825770728989634&Expand=true&StartDate=21-May-2018&EndDate=21-May-2018&DateTimeFormat=UTC&EventCategory=All)

        
            
                
                    Want to view Watson crashes that occurred on the same day from the user who
                    filed feedback?

The Watson Viewer will let you dive into crash data.

        [Device Drill](http://devicedrill/EventBrowser?dataSet=ALL&deviceId=g:6825770728989634&focusTimeStamp=2018-05-21T07:13:18&durationAfterMin=10)

        
            
                Want to view telemetry events from this customer's device?

Device Drill lets
you dive into the telemetry of the device this from which this feedback was filed.

Other Links


    
        Link

        Description

    

    
        [Feedback FAQ](http://aka.ms/FeedbackFaq)

        
            
                See the Feedback FAQ for more information about subjective user feedback.
            

        

    

    
        [Where's my CAB? Wiki](http://aka.ms/whereismycabwiki)

        
            
                If you were expecting a CAB but it does not appear in the viewer, please
                see the wiki for more information.
            

        

    

    
        [Translation Volunteers](https://aka.ms/feedbacktranslationvolunteers)

        
            
                For translation assistance, please see the translation volunteers wiki.
            

        

    

    
        [Reporting child pornography](http://aka.ms/FeedbackReportAbuseAndChildPornographyWiki)

        
            
                If attached screenshot(s) is child pornography you must report it using
                instructions listed at the link.
            

        

    






    
        
            Be Heard.
        

        
            [
                //aka.ms/feedback
            ](https://aka.ms/feedback)

Attachments

Comments and activity

  • Microsoft Edge Team

    Changed Assigned To to “Steven K.”

  • Hi,

    Will you provide a simplified repro for this issue?  I have looked at this issue and it appears the UTF-8 characters for the Unicode equivalent symbols, ‘ä’ in this case,
    are not getting converted properly.  For example:

    0xC3A4 - UTF-8(hex) for U+00E4 ä

    is being converted into two two byte UTF-8 characters: "0xC383 0xC2A4", which of course is not correct.

    I ask for a simplified repro because I see some HTML syntax issues on the site you gave me, which could cause the parser to misinterpret the source URLs.  I have attached a screenshot of the console and below is one example where there is a missing quotation mark ‘"’

       <div class="content-image rs_skip">
          <img src="https://cdn1.ostersundsbibliotek.se/images/5a2fc789193f640af8068179?crop=370x155" role=" presentation"/="">
       </div>

    The mark after the crop=370x155 is missing.

    It would be good to verify that the server is encoding the source files as UTF-8 properly and not a cp1252 encoded file but treated it as latin1 or some other similar issue.

    Hope you can provide another repro as this is an interesting issue.  Appreciate the submission and the help,

    Steve

  • Hi Steven,

    A repro can be found at https://rawgit.com/karl-sjogren/edge-15561198-repro/master/file-with-ä-in-filename.html.

    The source for the repro is available at https://github.com/karl-sjogren/edge-15561198-repro/

  • Hi, we have a similar problem, when call a REST-Service via XMLHttpRequest.

  • Reproduce Steps

    1. Navigate with Edge to https://www.festool.co.kr/서비스/서비스-올인클루시브/등록
    2. Open developer console and select the network tab
    3. Enter a device number, such as 200107, into the first input field
    4. See the results/errors in the network timeline
  • Microsoft Edge 40.15063.674.0
    Microsoft EdgeHTML 15.15063

  • The first request is a HTTP OPTIONS method and looks good (bad url, good result): https://ibb.co/iWza07
    The second request is a HTTP GET method and results a 500: https://ibb.co/ftO2f7

  • Hello all,

    Appreciate the comments and the repro steps.  I was able to reproduce this.  We will investigate this further.

    Steve

  • Microsoft Edge Team

    Changed Assigned To to “Venkat K.”

    Changed Assigned To from “Venkat K.” to “Scott W.”

    Changed Assigned To from “Scott W.” to “Nicolas A.”

  • We are in a catch 22 situation. We also use Kestrel as our web server but they don’t have any immediate plans to fix non-compliant headers:
    https://github.com/aspnet/KestrelHttpServer/issues/1144

    So in a nut-shell, our service is incompatible with the Edge browser.

  • Reproduce Steps

    1. Navigate with Edge to https://www.festool.co.kr/서비스/서비스-올인클루시브/등록
    2. Open developer console and select the network tab
    3. Enter a device number, such as 200107, into the first input field
    4. See the results/errors in the network timeline

    The first request is a HTTP OPTIONS method and looks good (bad url, good result):

    options_200

    The second request is a HTTP GET method and results a 500:

    get_500

    Microsoft Edge 40.15063.674.0
    Microsoft EdgeHTML 15.15063

  • The first request is a HTTP OPTIONS method and looks good (bad url, good result):

    options_200

    The second request is a HTTP GET method and results a 500:

    get_500

    Microsoft Edge 40.15063.674.0
    Microsoft EdgeHTML 15.15063

  • Microsoft Edge Team

    Changed Status to “Fixed”

    Changed Steps to Reproduce

  • Hi, we have a similar problem, when call a REST-Service via XMLHttpRequest.

    Reproduce Steps

    1. Navigate with Edge to https://www.festool.co.kr/서비스/서비스-올인클루시브/등록
    2. Open developer console and select the network tab
    3. Enter a device number, such as 200107, into the first input field
    4. See the results/errors in the network timeline

    The first request is a HTTP OPTIONS method and looks good (bad url, good result):

    options_200

    The second request is a HTTP GET method and results a 500:

    get_500

    Microsoft Edge 40.15063.674.0
    Microsoft EdgeHTML 15.15063

You need to sign in to your Microsoft account to add a comment.

Sign in