Cloud Build Pipelines: The Coalition

Microsoft's The Coalition Studios has made its way into the forefront of AAA gaming through its development of one of the top third-person shooter brands: Gears of War. The Coalition was eager to deliver new games to excited fans, but development was slowed by excessive wait times for shader and code compilation of the intricate levels, characters and vehicles. Adding to this, the third-party companies working alongside The Coalition often lacked the computing power necessary to effectively compile the shaders and codes. To overcome these internal and external challenges, The Coalition developers utilized IncrediBuild's unique process level virtualization technology to scale Azure performance capabilities far above that of a single Azure Virtual Machine. Without adding any on-premises equipment, The Coalition was ultimately able to deploy 700 cores, significantly reducing shader and code compilation times for themselves and their partners.

"IncrediBuild absolutely impacts our ability to iterate and move forward. We couldn't do anything without it."

Joe Vogt: IT Manager, The Coalition Studio

Business Challenge
With the release of both Gears of War: Ultimate Edition, a high-definition remake of the first game in the iconic series, and later Gears of War 4, the next chapter in the first-person shooter saga, 2016 proved to be an amazing year for Gears fans everywhere. To meet these incredible milestones, The Coalition Studio developers needed to overcome both internal and external production barriers.

Internally, frequent and massive use of Unreal Engine 4 (UE4) presented significant challenges, as told by The Coalition's IT Manager, Joe Vogt. "We use Epic's Unreal Editor 4 as a primary tool for content creation such as levels, characters and vehicles...The editor needs to compile all shaders beforehand, opening a level can take up to 30 minutes in some situations." With more than 100 team members using UE4 on a daily basis, a solution was needed to accelerate the shader compilation times. "You start doing the math and find out that with 30 minutes just to open a level, four to five times a day, you can spend a third of your day doing nothing," said Vogt.

In addition to their internal issues, The Coalition was also faced by external challenges. While The Coalition had the assistance of several third party outsourcing partners to provide the studio with augmented production capacity, some of these partners were smaller organizations that lacked the machines necessary to compile shaders and effectively utilize UE4. This situation left these partners in need of a solution.

Solution
As a first party studio inside of Microsoft, The Coalition Studio is able to use and benefit from the Azure cloud computing platform. Microsoft Azure provided The Coalition Studio with an easy and effective way to deploy hundreds of dedicated CPU cores to be used for accelerating local shader compilations.

The Coalition Studio then used IncrediBuild's unique process level virtualization technology to scale Azure performance beyond the capabilities of a single Azure Virtual Machine. The Coalition deployed dozens of 64-core Azure Virtual Machines (VMs) running IncrediBuild. By harnessing the power of Azure VMs, IncrediBuild creates an "on the fly" virtual environment where all those cores are working together for a single process execution. In this way, the platform was able to be scaled to meet the large process execution requirements and seamlessly accelerate UE4 shader compilation and level loading times.

Additionally, in the final six-month-crunch prior to game release, The Coalition was able to deploy a large Azure-based IncrediBuild farm with connectivity provided by Microsoft Azure ExpressRoute circuits. This allowed the Azure based agents to appear "local" to The Coalition's existing production coordinator. Without adding any on-premises equipment, 700 cores were deployed in this fashion and used throughout the final production push for Gears of War 4. The result was a significant reduction of both shader and code compilation times.

According to Vogt, "IncrediBuild absolutely impacts our ability to iterate and move forward. Our iterative cycle includes creating a build of the game, playing it, finding bugs, reporting the bugs, killing those bugs and then producing yet another build, sometimes two times a day. Rinse and repeat. It's all about how fast can a user get the latest version of all the data and get back to work. Without IncrediBuild, our development would grind to a halt."

To address the processing challenges of their third-party partners, The Coalition outsourced Azure and IncrediBuild resources for a short but crucial boost. "In one instance," recalled Vogt, "the partner was unable to provide any dedicated IncrediBuild farm capacity onsite, which caused huge waits for users when opening massive Gears of War 4 levels in UE4."

Regardless of the partner's initial resource limitations, by utilizing the flexible nature of Azure consumed services and IncrediBuild's parallel computing tech, The Coalition was able to grant their partners the power to scale their performance and enjoy unparalleled acceleration.

"We deployed around 160 cores to a public-facing Azure datacenter and configured a site-to-site VPN with the partner location." Vogt explained, "This allowed the small number of local Unreal Engine users to benefit from the IncrediBuild farm capacity in Azure to accelerate their work."

Since this was a short-term engagement, like many other micro- tasks that are part of developing a game, The Collation enjoyed Azure consumed services through hybrid cloud bursts. This allowed them to utilize Azure to boost IncrediBuild for the limited time that the boost was needed, rather than buying expensive hardware that would outlast its usefulness.

Benefits
The Coalition seemed to do the impossible by releasing two AAA games in one year. They were able to accomplish this in part by: 

• Scaling for maximum acceleration. By harnessing IncrediBuild's power The Coalition was able to create a 700 core Azure virtual super-computer, well beyond the limitation of Azure's most powerful single VM.
• Dramatically cutting shader and code build times through the unified use of IncrediBuild and Azure. This transformed UE4 level load times and enabled production staff to keep working instead of waiting.
• Overcoming limited partner resources. Through their allocation of IncrediBuild operated Azure cores The Coalition was able to boost the computing abilities of their under-powered third-party partners.

Utilizing real-time scaling for both internal and external needs. With this capability, The Coalition was able to increase computing capabilities during crunch time and scale back once the product was launched, without lasting, unnecessary hardware.
  

IncrediBuild adds yet another layer of scalability to Visual Studio, enabling developers to scale their applications to the cloud.

Mitra Azizirad: GM, Cloud Application Development & Data

Partner Overview

For 20 years, IncrediBuild has been the leading development acceleration solution. The award-winning plug-and-play solution runs on customers' existing hardware and dramatically speeds up builds, testing, packaging and more by up to 30x, by distributing tasks on machines across the local network. IncrediBuild empowers over 100,000 users from more than 2,000 leading companies to reach a fast time to market, save hundreds of developer hours and cut back on HPC costs. The company's strong partnership with Microsoft resulted in IncrediBuild's integration into Visual Studio.