DOOM Eternal accelerates online services with PlayFab
Renowned for technical innovation and having invented the first-person shooter genre, the online multiplayer deathmatch, and the modern game engine, id Software is one of the most storied game studios in the world. While creating DOOM Eternal, the sequel to the blockbuster 2016 reboot of the original DOOM, the development team at id used Microsoft Azure PlayFab to build several years’ worth of exciting features into the game in only one year and deliver an immersive multiplayer experience.
Getting DOOM Eternal to market so fast was a career achievement that was only possible because we built it from the foundation that Azure PlayFab provided. --Travis Bradshaw: Lead Services Programmer id Software
Building worlds that enchant
When development first began on DOOM Eternal, the hotly anticipated sequel to the id Software 2016 blockbuster, DOOM, the timeline included ambitious goals for building new online features into the game. It could have taken the development studio three years to complete all the work using the resources at its disposal, so id turned to Microsoft Azure PlayFab to expedite the release. The studio relied on versatile PlayFab features not only for an abbreviated timeline, but also for the features that contributed to an extensible architecture, contextualized telemetry data, and higher performance for a game that has garnered rave reviews.
Travis Bradshaw, Lead Services Programmer at the ZeniMax-owned id Software studio, appreciates the LiveOps aspect of development that PlayFab brings to the table. “We’re always improving our games,” says Bradshaw. “Being part of that iterative cycle of improvement is exciting.”
DOOM Eternal features live multiplayer and player engagement services to bring players together, reward them in real time, and track their progress. These required a sophisticated and specific set of tools and services to run, and PlayFab was there to deliver. “It had been a long time since we created an online title,” says Bradshaw. “We’d fallen behind competitors in our own space. And the fact that we hadn’t been in that space for a while didn’t mean that our customers and gamers would expect anything less from us.”
Bradshaw’s guiding principle is that it’s not enough to catch up—the way to succeed is to project where the industry is going and get there first. “We had little time,” he recalls. “Our need to get to market quickly and deliver at a high bar converged with Azure PlayFab. The way it delivers online features and functionality aligned with our ideas for next-generation game technology. That synergy gave us confidence.”
Caught between the short timeline and his lean team, Bradshaw faced a big choice: “I had to decide where to deploy our substantial expertise and talents for service programming. It was either reinvent the wheel, building an online platform from scratch to catch up with our competitors, or use Azure PlayFab to establish the foundation of what’s expected from online games today so that my team could do what it does best and make our game unique.”
Designing and deploying with cloud agility
Bradshaw’s team used the core platform features of PlayFab as the basis for the online features for DOOM Eternal, like CloudScript, a versatile PlayFab feature that runs a variety of functions in the cloud. “We leaned heavily on data APIs to hold state and CloudScript run by a trigger from PlayStream to flesh out the online systems,” says Bradshaw. “The way that PlayFab services are architected around PlayStream was a huge part of what drew us to PlayFab services.”
Because the PlayStream and Insights features of PlayFab offer two complementary profiles for sending data to the platform, id took a two-pronged approach to aligning its game clients with modern online services. The first strategy focused on driving player engagement and rewards through real-time communication and responses in cloud services—for example, enhancing the weekly challenge system to go beyond gains in response to a player completing a level by adding information about the minimum and maximum difficulty that players selected during a play session. Bradshaw’s team wanted to trigger multiple online game features from the same game event to react to happenings in the DOOM player experience. It achieved that by taking advantage of the flexibility of rule-based action processors in PlayStream. With logic offloaded from the client to the cloud, the DOOM architecture became extensible, removing the burden of client updates while enabling a creative design. “The resulting functionality is really quite clean, with minimal overhead to the game to prepare and serialize the single message to PlayStream, then two distinct rules that both trigger on that event to invoke CloudScript handlers for each of the two independent systems,” explains Bradshaw. “That’s how we’re able to provide these ‘just what’s needed’ events in near real time from the game.”
The second prong of the approach hinged on the PlayFab Insights capacity to ingest massive amounts of telemetry data with minimal overhead. All the events from the game client (PlayStream game events or Insights telemetry) and all the events from the PlayFab services themselves like sign-in events or statistic updates are delivered to the same cohesive data stream for analysis and unified data retention and management. That’s why Bradshaw’s team decided to add all its service telemetry for the game to the same stream, creating telemetry events from CloudScript or standalone services and logging them to the same endpoints that the game uses. “The resulting contextualization has really sped up troubleshooting on services, making it easy to correlate service behavior with reported game client behavior,” he says.
Choosing a LiveOps platform like PlayFab meant the id team could focus its energies on creating differentiated live services and superior player engagement, not on managing infrastructure. It also meant scalability on the level it needed to reach gamers all over the world.
Conquering lessons from recent history with Multiplayer Services
With no cross-platform game server hosting solution that offered the combination of affordability and expressiveness that id needed, the studio took what Bradshaw calls the “roll our own” approach for the 2016 version of DOOM. While this brought the immediate functionality that id needed, the costs of maintaining the system over time outweighed the benefits.
The team was initially pleased with what Bradshaw describes as a trivial effort to deliver nearby regions to players on all platforms with acceptable overhead costs when paired with a “sensible” reservation system architecture. “But as the typical exponential decay function played out on the player base to settle into the long tail of our core players, the overhead associated with managing the game servers began to overwhelm the actual compute,” recounts Bradshaw, “and at that point, the only responsible option was to reduce the number of regions available to players.”
Bradshaw’s team solved hard problems quickly, using manually triggered provisioning first and then load-based dynamic scaling. But that created a “death-by-a-thousand-cuts” scenario that had Bradshaw’s team spending more time on the multitude of small details that go into providing a great server reservation system for users: Quality of Service (QoS) checks to the regions, negotiating the best region for a party of players, gracefully handling failovers and race conditions, and other minutiae. “We found ourselves polishing—working on aspects that directly interacted with the lower-velocity aspects of game experience that had longer debugging cycles than our services-centric challenges,” says Bradshaw.
When it came time to plan for DOOM Eternal, id wanted a solution that would provide a lower maintenance burden at a more manageable cost. That’s when the id team started to lean into PlayFab Multiplayer Services. Without the substantial overhead that the reservation system imposed, the studio was free to keep more regions available to players without having to rationalize “upside down” returns from low-population regions.
“With Multiplayer Services, the polish that you need is already there,” Bradshaw says. Clients can make a latency measurement to endpoints in each of the regions that host game servers, building a profile of their QoS to each of the available regions. Those clients provide that list to the matchmaking service, in which the studio’s configured queues have a region selection rule that the matchmaker uses to find a best fit, and then return a match to the players. “Thanks to the wealth of regions available on Azure, we find that our players get their first-pick region almost every time,” adds Bradshaw.
The initial phases of development went quickly, which gave the id team time to focus on adding engaging enhancements. “Our fast start meant that we could focus more on the newer PlayFab features to fill out the more difficult online gaming challenges,” says Bradshaw. One goal was to improve the lobby experience from the 2016 version of DOOM. To do this, the team needed a cross-platform networking transport that could provide the relay needed to solve the traversal of challenging customer network topologies. PlayFab was there to help. “We built our lobby systems, including voice and chat, with PlayFab Party,” explains Bradshaw. “The added bonus of text and voice chat made it an easy win for our user experience.”
With all the exciting features developers were building into DOOM Eternal, the team occasionally ran ahead of PlayFab capabilities. This wasn’t a blocker, however, because PlayFab is intended to be an extensible and flexible framework that developers can customize to meet their needs. As an example, the id team adapted PlayFab Groups to serve as a cross-platform lobby system that supports the shared context necessary for successful matchmaking.
There were also a few areas where some online features were more complex than could be put together with basic data APIs and CloudScript, so the id team filled the gap with PlayStream and webhooks functionality to connect standalone id services. The studio needed the platform it chose to interoperate with its idTech services. “The initial connection with PlayFab and idTech took only a few weeks,” says Bradshaw. “It went so seamlessly and quickly that our confidence was very high.”
Crossing the finish line
The flexibility of PlayFab gave id the tools it needed to build Multiplayer Services exactly how it wanted. Bradshaw liked the flexibility that came with the standards-compliant web interface of PlayFab. This gave developers the option to build their own version of the SDK that aligned with their core game engine libraries and met the game’s extreme performance requirements.
The game launch in March 2020 coincided with the beginning of the COVID-19 lockdowns and soaring internet usage, but the PlayFab team was dedicated to supporting id. “We’d architected our game to be very resilient—it never went down,” says Bradshaw. “We had great performance for our launch during a global internet emergency, and I attribute that largely to the PlayFab team and the technology helping us meet the bar in extraordinary circumstances.” Again, his team called on PlayFab Multiplayer Services to help respond to a challenging situation in the moment, reallocating regions while the game was live without affecting players.
“We needed a great solution to meet our final delivery cycle stages and get to market on time,” says Bradshaw. “Honestly, PlayFab is a huge bargain compared to the challenge we faced. Getting DOOM Eternal to market so fast was a career achievement that was only possible because we built it from the foundation that Azure PlayFab provided.”