COVID-19 has been challenging for many disciplines, and game development is no exception. With the bulk of Microsoft's workforce now working from home, this posed a unique challenge for the content development teams within our Xbox Game Studios.
For our game developers to be able to work safely and productively from home, they need to be able to access and check-in content from/to various build systems within a reasonable timeframe. Downloading daily builds is a critical step in the development and validation processes. While development machines are typically collocated with build cache servers in the office and connected through high speed local area networks (40-80 Gigabit network), working from home requires engineers to download large builds onto their development machines using their existing home ISP network (40 Mbps to 1 Gbps download speed). The problem we ran into is how to best enable speedy downloads of ~300 Gb sized builds multiple times a day to 300+ developers located in the greater Seattle area (Washington state).
To address this challenge, teams in Xbox Game Studios partnered with Azure to create a fast and secure build transfer solution. Using Azure Blob Storage and AzCopy, the team successfully rolled out a proof-of-concept with 30 engineers to ensure the solution could scale. The proof-of-concept, which was developed and tested within a few weeks, exhibited reasonably good throughput of ~100-120 Mbps for offsite uncompressed download. This also stress tested the limits of Azure Blob Storage, Azure Networking and AzCopy by simulating simultaneous connections throughout the day.
These were the steps undertaken by the team to establish a fast and secure off-site build download:
- Created six premium blob storage accounts in the West US 2 region, to minimize network latency between engineers in Washington state and the Azure Data Center
Rewrote the build download scripts to call invoke AzCopy to enable secure and quick off-site download (Script 1)
Leveraged off-the-shelf compression techniques such as Gzip in combination with AzCopy's decompress copy feature to make the builds more lightweight where possible. Compression ratio of ~1.14 for content builds and ~4.8 for binary builds were achieved
Figure 1: Before COVID-19, typical set-up
Figure 2: During COVID-19, off-site build Storage + AzCopy set-up
While the proof-of-concept throughput was reasonably high for off-site development, a higher performance would have significantly sped-up the development process. Innovating further, teams used compression techniques coupled with AzCopy's decompress at destination feature, additionally reducing the build download time. Due to the overall redesign above, a 5X – 7X throughput improvement was achieved. Download times went from a 120Mbps using SMB over VPN, to around 600 - 700Mbps throughput using AzCopy and Azure Blob Storage. This reduced engineers' time to download a build from ~1hr down to 8-12 minutes, while working from home!
The above success indicates the possibility of a paradigm shift for content developers especially in the entertainment sector. While development of entertainment software such as those for gaming and movies relies heavily on strong on-premises infrastructure with high throughput networks and storage requirements, the work from home situation has inevitably forced innovation in off-site development strategies. Azure has demonstrated the ability to support such challenging use cases, which can eventually eliminate the need for content collaborators to be tied to a fixed workplace. The architecture suggested here is not only applicable to game development but also to any team that wishes to enable working from home where content distribution is essential to business continuity.
Using the custom script available here, and the Azure storage tutorials linked, you can instantly unlock a similar solution for your organization's work-from-home scenarios. See below for a sample script.
Script 1: Sample script