Supercharge games with Azure AI and reinforcement learning

Azure offers a full range of AI solutions to help you build, run and grow your games.

August 03, 2020
Girl with a machine gun, from a video game

​​​Azure offers a full range of AI solutions to help you build, run and grow your games. AI and machine learning can help accelerate game development by providing more realistic worlds and challenges as well as support automation and live operations. At Game Stack Live, Microsoft Research announced Project Paidia, a research effort aimed at exploring new opportunities created with AI based reinforcement learning in gaming. This project is grounded in a close partnership with Ninja Theory and their game Bleeding Edge, which is the perfect environment to research state-of-the-art AI agents. Working alongside Microsoft Research, Azure is bringing reinforcement learning to our customers and game developers in Azure Machine Learning. 

"We make ambitious games with small teams, where it's critical to maximize productivity. With Azure Machine Learning, we can bring multiple intelligent agents into a solution to learn rapidly and collaborate in the same way a player would." – Gavin Costello, Technical Director, Ninja Theory.​

What is Reinforcement Learning (RL)?

Reinforcement learning is an approach to machine learning to train agents to make a sequence of decisions. The entity that executes actions is the game agent, for example, a robot deciding on a path to walk.  RL works by: 

  • Providing an opportunity for the agent to enact a behavior, such as making decisions or choices.
  • Providing contextual information about the environment and choices, or the state in which the agent finds itself.
  • Providing feedback or rewards based on how well the behavior achieves a certain goal.​​​​​​​​​
diagram of agent-environment interaction in reinforcement learning

​​​​Image: Shows agent-environment interaction in reinforcement learning. 

Using Azure Machine Learning for RL

We recently added new reinforcement learning capabilities to Azure Machine Learning. It enables developers to train on powerful CPU or GPU clusters that are automatically provisioned and scaled to help manage costs.
You also get flexible support for your training scenarios such as:

  • Multiple training scenarios including Single Agent and Multi-agent RL
  • Access popular open source RL algorithms 
  • Use with a variety of gaming environments including Open AI Gym
  • Flexibility to use TensorFlow, ONNX and PyTorch models in-game
  • Unified tracking for experiments, monitoring of metrics and run comparisons

Azure Machine Learning reduces the complexity of running large scale experiments and makes RL and extension to the game development workflow, so game developers can focus on the task of creating engaging agents. 

Putting RL into action

Let's dive into an example of how you can train agents using Azure Machine Learning and a simplified gaming environment. 

1. Setup the problem: In the cooperative navigation task, (called 'simple_spreader'), agents are rewarded for spreading out and exploring landmarks in an environment without overlapping - the agents are the large blue circles and their goal is to each navigate to a separate black landmark without running into each other

2. Create the training script: The Python training script for the Cooperative Navigation sample is on Github.  It uses an algorithm called MADDPG which trains multiple agents together.  A typical experience involves iterative development using a combination of local or cloud hosted notebooks, and development tools such as Visual Studio Code. Azure Machine Learning notebooks enables rapid iteration using cloud resources.

3. Train the model, easily scaling to multiple nodes: After creating compute clusters in Azure Machine Learning, submit an agent training job using the Azure Machine Learning ReinforcementLearningEstimator. The following example sets up a training configuration to run on 4 worker compute nodes to collect training data. ​

estimator = ReinforcementLearningEstimator(
source_directory='files',
entry_script='particle_train.py',
script_params={
'--scenario': 'simple_spread',
'--final-reward': -400
},
compute_target=cpu_cluster,
environment=cpu_particle_env,
max_run_duration_seconds=3 * 60 * 60
)

train_run = exp.submit(config=estimator)

RunDetails(train_run).show()


​​Code Sample: RL training script

Azure Machine Learning automatically allocates and deallocates compute as needed and runs the training script.  

4. Visualize metrics, outputs and evaluate the model: In the beginning, the agents don't know what they are doing and don't get to the landmarks.  As the agents undergo training, they begin to make more progress toward landmarks. When the reward reaches more than about -400, the agents are able to successfully navigate cooperatively as shown below.​

animation showing tracking training progress in Azure Machine Learning

GIF: You can track progress of training in Azure Machine Learning.

graph showing how training reward increases over time with TensorBoard

​Graph: Shows how the training reward increases over time with runs in TensorBoard

More ways to add AI to your games

In this blog, you saw how reinforcement learning can be used to improve games. Here are some more examples of how the gaming industry is using Azure AI to accelerate game development and online experiences - Xbox is using reinforcement learning with Personalizer, to grow user engagement on their homepage by 40%. Big Fish Games is using Content Moderator to keep chat rooms safe, for Let's Dish, a new Match 3 cooking game. Cloud 9 is using Computer Vision to develop a game insights platform for competitive e-sports including League of Legends. Machine Zone, creator of Game of War: Fire Age, uses Translator to connect players across language barriers.

Azure AI is changing the way games are developed and played. We are excited to bring these tools to you and invite you to start your journey with Azure today!​​

Resources