Update to Filtering and Sorting in the Mail, Calendar and Contacts REST API

Hi Partners and Developers,

We recently deployed a behavior breaking change to Microsoft Graph V1.0 and the Outlook V1.0 & 2.0 endpoints to block a specific class of requests that contain $filter and $orderby for messages, which can result in users being prevented from accessing their mailboxes in Outlook or any other app.  We realize that this breaking change impacted some of you whose apps & services relied on this behavior and we sincerely regret the pain that this has caused.

The remainder of this post explains the change we made so you can determine if this impacts your app or service and if so, the error you will receive, the modifications you must make to avoid the error and what we are doing to avoid such breaking changes in the future.    

Impacted Requests 

The behavior change impacts your app or service if they make a request that contains both query parameters $orderby and $filter, and meets any of the following conditions:

1. Fields appear in $orderby that are not present in $filter.

a. Example that doesn’t work because receivedDateTime is present in $orderby, but not in $filter:

https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$orderby=receivedDateTime & $filter=from/emailAddress/address eq 'Garth.Fort@contoso.com' & $top=10 & $select=receivedDateTime,from,subject,hasAttachments,bodyPreview

b. Example that works, after adding a $filter clause for receivedDateTime:

https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$orderby=receivedDateTime & $filter=receivedDateTime ge 1900-01-01T00:00:00Z and from/emailAddress/address eq 'Garth.Fort@contoso.com' & $top=10 & $select=receivedDateTime,from,subject,hasAttachments,bodyPreview

2. $filter clauses use fields also present in $orderby, but in an order different from that in $orderby.

a. Example that doesn’t work because hasAttachments and receivedDateTime don’t appear in the same order in $orderby as in $filter:

https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$orderby=hasAttachments,receivedDateTime desc & $filter=receivedDateTime le 2016-01-01T00:00:00Z and hasAttachments eq false & $top=10 & $select=receivedDateTime,from,subject,hasAttachments,bodyPreview

b. Example that works because because hasAttachments and receivedDateTime appear in the same order in $orderby and in $filter:

https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$orderby=receivedDateTime desc,hasAttachments & $filter=receivedDateTime le 2016-01-01T00:00:00Z and hasAttachments eq false & $top=10 & $select=receivedDateTime,from,subject,hasAttachments,bodyPreview

3. Fields, appearing in $filter and not in $orderby, come before those fields that are present in $orderby.

a. Example that doesn’t work because receivedDateTime (not present in $orderby) appears before from/emailAddress/address (present in $orderby):

https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$orderby=from/emailAddress/address & $filter=receivedDateTime ge 2016-01-01T00:00:00Z and from/emailAddress/address eq 'Garth.Fort@contoso.com' & $top=10 & $select=receivedDateTime,from,subject,hasAttachments,bodyPreview

  b. Example that works because from/emailAddress/address appears before receivedDateTime.

https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$orderby=from/emailAddress/address & $filter=from/emailAddress/address eq 'Garth.Fort@contoso.com' and receivedDateTime ge 2016-01-01T00:00:00Z & $top=10 & $select=receivedDateTime,from,subject,hasAttachments,bodyPreview

Error Returned for Failed Requests

{    "error": {      "code": "InefficientFilter",      "message": "The restriction or sort order is too complex for this operation.",      "innerError": {        "request-id": "<>",        "date": "<>"      }    }  }     

1. If your app is making requests that match the patterns explained in the previous section, please modify them to ensure the following:

1. All fields specified in the $orderby parameter are also specified in the $filter parameter.
2. Fields that appear in both $orderby and $filter appear in the same order.
3. Fields that are present in $orderby appear in $filter before fields that aren’t.

2. No changes are required and your app or service will continue to work fine if the app uses $orderby or $filter in either of the following ways:

1. Uses $filter but not $orderby.
2. Uses $orderby but not $filter.

Reason for the Breaking Change

We made a change in May 2016 to loosen restrictions on $filter for mail, calendar and contacts so apps can use filter operators such as “contains” and “startsWith”. As part of this change, we unintentionally also permitted the types of requests similar to (1a), (2a) or (3a) described in the section “Impacted Requests”. This slipped through multiple validation rings and ran  fine for users for months.  But recently we identified these types of requests were the root cause for a small number of users with large mailboxes being completely blocked from accessing their mail, calendar and contacts in any app including Outlook. The reason for this is that requests which match the criteria in the  “Impacted Requests” section can be very inefficient for the servers hosting the mailboxes, posing a severe load on the server. To protect users’ access to their mail, calendar and contacts, we had to quickly disable support for such requests.

Minimizing the Impact of Future Breaking Changes to Outlook APIs

We realize that these changes broke apps that relied on the impacted requests to work, and we are working hard to avoid such a situation in the future. Starting with Outlook APIs in Microsoft Graph, we’re changing our practices:

1. We will enhance test coverage to make sure inefficient requests aren’t unintentionally enabled. We will implement  test cases to verify that the types of requests that should be blocked are never enabled. 

2. We will not make a breaking change to production versions unless we need to address a security issue or significant service degradation. Barring emergency security patches or fixes to restore service for customers, we will provide app developers a reasonable lead time to make changes to their apps and services, to better accommodate any breaking changes.  In this incident, there was a process breakdown that resulted in developers discovering the breaking change only after encountering errors in production.  Our communication processes had been designed for intentional breaking changes in beta, and not in a production version. We have made changes to address this breakdown to ensure we’ll communicate in a timely fashion in the future.

3. We will introduce behavior changes first in beta and promote them to v1.0 only after several months. This will apply to breaking and non-breaking changes, with the exception of bug fixes. In this incident, the original change to add support for “contains” and “startsWith” was enabled in both beta and v1.0 at the same time. Had we restricted the original feature roll-out to beta, the breaking change would not have impacted the production version, v1.0.  

We apologize for the pain that this has caused for some of you. Please email us at msgraphpm@microsoft.com if you have any questions or concerns.