Record-breaking Fall Guys scales faster with Azure

Mediatonic launched Fall Guys: Ultimate Knockout, a battle royale game in August 2020. Fall Guys sold more than 11 million copies on Steam alone and is the most downloaded PlayStation Plus title ever. To support this massive volume of players and increase the rate to iterate new features and content, Mediatonic uses Microsoft Azure Kubernetes Services (AKS), Azure Cosmos DB, Azure DDoS Protection, and Azure DevOps.

"Fall Guys exceeded all expectations at launch. It sold more than 11 million copies on Steam alone and ranked as the most downloaded PlayStation Plus title ever. Azure provided all the reassurance we needed to scale elastically and quickly."

Adam Fletcher: Chief Technology Officer

Tonic Games Group

Massive success at launch was well beyond expectations

Mediatonic Games, a studio under the Tonic Games Group umbrella, developed Fall Guys: Ultimate Knockout as an internal pitch to publishers. Inspired by chaotic classic TV game shows such as Takeshi's Castle, it featured fun, simple, and easy-to-render jellybean characters. Once publisher Devolver picked up the game, Fall Guys released on PC and PlayStation 4 in August 2020. 

Fall Guys launched with massive success during the COVID-19 crisis, with player numbers well beyond expectations. The Mediatonic team quickly ramped up to develop even more enticing content and opened another studio location in Leamington Spa, a well-known hub for the United Kingdom gaming industry. The studio is estimated to create an additional 60 jobs by 2022. 

COVID-19 lockdowns had a significant impact on immediate demand for Fall Guys, with people often stuck at home and seeking relief and connection through an engagingly whimsical game. Its popularity accelerated franchising plans, shifting to three to six months instead of 12 months, while focusing on delivering new features to the playerbase—more levels, costumes, and customizations—with the audience providing constant feedback. Ultimately, Mediatonic is looking to reach as wide an audience as possible and continues to collaborate with high-profile IP holders, with plans to scale even faster.

Mediatonic had a long history of using multiple cloud platforms and a wide range of services, including fully managed, serverless, platform as a service (PaaS), and infrastructure as a service (IaaS). The team packages code as containers so that it can run practically anywhere. The primary requirements were:

• Container services for Kubernetes clusters
• A multi-model database that scales rapidly with zero downtime
• Solution for detecting and mitigating distributed denial-of-service (DDoS) attacks on the gaming platform
• Automation of software development and IT operations
   
  To continue to meet the needs of Fall Guys, Mediatonic adopted these Microsoft Azure solutions:
  
  

• Azure Kubernetes Service (AKS)
• Azure Cosmos DB
• Azure DDoS Protection
• Azure DevOps
  
  Thanks to this cloud architecture, the company can rapidly grow and scale operations.
  

Supporting an enormous resource quota at launch 

Although Mediatonic had launched large-scale games with well-known IPs with Microsoft in previous years, nothing was on the sheer scale and profile of Fall Guys. "Fall Guys exceeded all expectations at launch. It sold more than 11 million copies on Steam alone and ranked as the most downloaded PlayStation Plus title ever. Azure provided all the reassurance we needed to scale elastically and quickly," says Adam Fletcher, Chief Technology Officer, Tonic Games Group.

With the scalability of the Azure platform, Mediatonic can avoid paying for costs during quieter periods, but ramp up to meet enormous demands when needed. At the peak growth for Fall Guys, more than 25,000 people logged in every minute, with 2,000 new accounts created, and 1,000 new games started.

"When we launched Fall Guys, the game just exploded! After having conversations around how big this could get, we asked for the highest quota within reason from Azure. They provided that without a lot of additional processes," says Craig Bendell, Technical Director, Mediatonic Games.

Delegating infrastructure needs for small development teams

Mediatonic decided to use AKS based on its small platform engineering team and the need for a robust service-level agreement (SLA). The team wanted to take advantage of scalable Kubernetes architecture without having to dedicate a lot of effort to operating the infrastructure. At the time of evaluation, the company found AKS to be a much more mature offering than other options.

Ben Constable, a Principal Platform Engineer at Mediatonic, reveals, "Mediatonic already had some experience deploying unmanaged Kubernetes, but that created a lot of overhead for our small team. We found AKS to be the right solution for us because it is stable and provides a great experience with Azure support. It also allowed us to benefit from interoperation with other Azure services."

Mediatonic also chose Azure Cosmos DB as a multi-model database that scales infinitely as far as quotas will allow, with zero downtime. The company received hands-on help from the Azure team to implement perimeter protection for the system, using Azure DDoS Protection after launching. Azure architects highly recommended this service to detect and mitigate DDoS attacks on the gaming platform.

Combined with application design best practices, Azure DDoS Protection automatically detects and mitigates DDoS attacks on the Fall Guys platform. From the perspective of Mediatonic's server engineering team for platform and game service, all the hard work is handled by Azure, leaving the team more time to focus on creating award-winning games.

Shifting Kubernetes management to AKS

Mediatonic already had experience building and deploying applications using Kubernetes and scaling deployments to expected load. With AKS, the company can take advantage of its cluster's auto-scaling, so it gets all the benefits of a highly scalable, robust architecture without spending a lot of time on maintaining and deploying the infrastructure. Mediatonic uses Azure to take care of all basic "plumbing" concerns, such as managing Domain Name System (DNS) records. Mediatonic knew that since AKS comes with Azure SLAs, it was the right choice for solutions requiring highly resilient services.

With AKS, Kubernetes orchestrates virtual machine clusters and schedules containers to run on those virtual machines based on each container's available computing resources and requirements. The basic operational units for Kubernetes are containers, grouped into pods to be scaled to the desired state.

"It was an incredible load off our shoulders, having hands-on help from the Azure team," says Bendell. "A Kubernetes cluster was about to go out of support, and our timelines didn't allow us to complete the transition to a new version of Kubernetes or to a new cluster in time. Azure gave us top-notch support for that specific cluster for the extra few days we needed."

Enabling agile teams to focus on building games

With AKS, Mediatonic's agile team can concentrate on building game services rather than maintaining its platform. Kubernetes is an open-source orchestration software for deploying, managing, and scaling containers. Since AKS can easily define, deploy, debug, and upgrade complex Kubernetes applications, Mediatonic's server engineering teams could accelerate their time to market, and focus on writing their application code and delivering new functionality to players. 

Minimizing latency while reducing costs

Mediatonic selected Azure Cosmos DB as a fully managed NoSQL database for modern app development. Single-digit millisecond response times, automatic and instant scalability, and the SLA assure high availability. Azure Cosmos DB also provides faster app development and greater productivity, removing database administration through automated monitoring, management, updates, and patching.

Fall Guys deals with different data types and uses Azure Cosmos DB to store a variety of player metadata, such as profile info, inventory, and stats. Azure Cosmos DB can replicate data across multiple datacenters for latency and increased resilience.

The team can also configure Azure Cosmos DB consistency levels from very strong to eventual consistency. Lower consistency levels are useful to optimize solutions for the speed of writes. Higher consistency levels are useful for more critical information. Azure Cosmos DB offers five well-defined levels: strong, bounded staleness, session, consistent prefix, and eventual. Mediatonic's application developers can optimize their solutions for speed of writes or consistency of reads across all nodes using different consistency levels. These consistency levels are region agnostic and guaranteed for all operations regardless of the region from the reads and writes, the number of regions, or whether configured with a single or multiple write regions in the Azure Cosmos DB account.

"Mediatonic already had some experience deploying unmanaged Kubernetes, but that created a lot of overhead for our small team. We found AKS to be the right solution for us because it is stable and provides a great experience with Azure support. It also allowed us to benefit from interoperation with other Azure services."

Ben Constable: Principal Platform Engineer

Mediatonic Games

Achieving faster game development with Kubernetes

The modern agile development practices used by Mediatonic involve release planning, continuous integration and continuous delivery (CI-CD), continuous testing, and constant monitoring and feedback, all of which translate to a faster time to market for Fall Guys. Teams rely on Azure DevOps to manage requirements, run their teams using Agile or Scrum templates, build and test code, run CI-CD pipelines, and deploy working software.

Teams can also use any Git repositories, including ones hosted by Azure DevOps or GitHub, for their source code. From there, the build can be packaged and deployed to any container registry, like Docker Hub or an Azure container registry, or self-hosted Kubernetes clusters.

With Azure DevOps, Mediatonic's developers can manage their code's life cycle from requirements to deployment to have more time to focus on implementing the building blocks of their solution to provide new functionality to gamers. Developers can more easily contribute their code to a shared repository—both application and IaaS codes. CI-CD pipelines are extensible and can perform various tasks for the team, such as making sure the code builds, unit tests pass, and code quality meets the team's standards. This method ensures a complete function, high-quality, and maintainable code.

Automating code with pipelines in Azure DevOps

Mediatonic's team uses Azure DevOps to host their requirements, source code repository, and CI-CD pipeline to automatically handle building, testing, and deploying the code to appropriate environments. Achieving a high degree of automation means that developers can focus on the complexities of implementing game functionality. The automatic and consistent handling of routine tasks leads to a more stable and predictable system and improves the game's quality.

Server engineers push changes to their feature branches as they work on a specific user story. When ready, they can create a pull request to get their code merged into the main branch, triggering a CI-CD pipeline that packages the code as a container, pushes it into the container registry, and deploys it to a Kubernetes cluster for automated end-to-end testing.

Since Azure DevOps supports a wide variety of platforms and development environments, the Mediatonic team has developers running multiple operating systems and integrated development environments. The team uses Git tags to help control the flow of deployment of their code. Creating tags automatically builds and pushes out containers to development environments. Once releases have gone through their QA process, the full release cycle is managed from Azure DevOps.

Bringing Fall Guys to as many players as possible

Azure subscriptions and resource groups provide a more secure and sensible cloud resource model that help with ramp-up time for new engineers as Mediatonic continues to grow across its multiple studio locations. Azure also offers robust monitoring and logging solutions. Engineers can combine data from Azure Monitor with monitoring data from their own applications to get a unified view across the entire application stack. Mediatonic deploys Elasticsearch and Kibana on AKS to collect logging data, which couples the stability of AKS's managed service with familiar open-source technology.

Ultimately, Mediatonic wants to reach as many players as possible. Fall Guys offers Kudos and customizable cosmetic rewards reliably delivered to the players when they've accomplished a particular in-game task to keep players engaged. The Mediatonic team uses Azure Event Hubs to handle in-game events and respond accordingly. Using this flexible and extensive architecture, it built a framework for the design team to develop new and exciting features with content that keeps the player delighted. With Azure, Fall Guys can continue making headlines on Steam, PlayStation, and beyond.

Find out more about Mediatonic on Twitter, YouTube and LinkedIn, and Fall Guys on Twitter.

"When we launched Fall Guys, the game just exploded! After having conversations around how big this could get, we asked for the highest quota within reason from Azure. They provided that without a lot of additional processes."

Craig Bendell: Technical Director

Mediatonic Games