Fortifying Teams with AI and Optimized Workflows

Last week, I had an opportunity to speak at SIGGRAPH, one of the computer graphics industry’s premier events that focuses on research, education, and skill development. I spoke with Munkhtsetseg Nandigjav, Associate Dean School of Animation & Motion at Savannah College of Art and Design, about my role as the General Manager for Gaming AI at Microsoft Gaming, our Responsible AI framework, and the ways that leaders can support their teams with AI to adapt to an ever-changing industry landscape.

Before I dive into the specifics of AI for Gaming and how I believe it can help change the industry we love for the better, I want to share a bit about my own background and why this matters so much to me.

Harnessing the good and positivity in technology

I’ve always been a technologist, ever since my dad brought home our first PC and I found myself writing little programs in BASIC or loading all these floppy disks to play King’s Quest. Even then, I felt the power of community that technology could bring. I remember I was a pretty nerdy kid, from an immigrant family from China, who always had the weird food at lunch and was way too shy to speak up. And at home we got a modem, where I could dial into local BBSes to access the internet – some of my best early friends I connected with through internet relay chat, playing online games together.

As a software engineer, I really wanted to get closer to WHY we were building the technology we were building, and whether that software would really resonate and be impactful to its users. So I did a masters in interaction design and embarked on a journey as a designer.

Ultimately, I’m an optimist who is passionate about seeing the impact of new technology, real tangible impact, for an individual, for a community.

Several years ago, I was lucky enough to be part of a BBC series exploring how technology and design could really create new solutions to support individuals in need. One of the stories I worked on was about a young woman named Emma who was diagnosed with early-onset Parkinsons. I was incredibly privileged to learn about her life and power and be able to invent new technologies that might support her in some small way.

I believe multidisciplinary technologies, whether it’s physical computing, electronics, 3D printing, or AI, can be harnessed for good and positivity and can help us change our world for the better. I think when working in technology, it’s always important to really understand the core problem you’re trying to solve.

When we talk about AI, we tend to talk a lot about the model advancements, number of parameters, or state of the art benchmarks. But to actually deliver any innovation that impacts people’s lives, there’s so much more collaboration across specialties like human-centered design, responsible AI, user research, product management, and software & prompt engineering.

What is AI for Gaming?

First, let’s take a look back: Artificial Intelligence as a practice and set of technologies is intertwined with the history of videogames. From rule-based AI systems that controlled in-game characters, like those pesky ghosts in Pac-Man or the toughest enemies in games like Elden Ring or Halo, to using genetic algorithms to procedurally generate dynamic game levels in games like Spelunky.

At Xbox we’ve been shipping AI in games for close to two decades, whether it’s from game studios or the teams that build the Xbox platform and apps. From using Bayesian inference for player skill ranking and matchmaking to AI mimicking your friends’ driving styles in the Forza franchise, AI is infused across our teams. 

Broad availability and accessibility of Generative AI has really been a major step forward in the capabilities of the technology. It’s more important than ever to have a focused Gaming AI team, where we are helping to support teams as they embark on their AI journey, guiding and setting a vision for AI that is responsible, inclusive, supportive, and empowering.

As with any new technological change, how the future unfolds is up to all of us and our teams. We really do need to think about how to apply this in support of players and game creators.

How AI is augmenting game development

As I said, AI as a field of computer science is deeply intertwined with the history of videogames. Games like chess or Go provided structures within which to advance the state of the art for AI – the intelligence level of an algorithm could be measured through the scoring in the game.

As games became more immersive, so did the complexity of AI to navigate these games. For example, Minecraft has been used as a research playground for AI agents, to complete tasks (like mining for diamonds) or building structures. We do a fair bit of this work at Microsoft, in collaboration with Microsoft Research – where there is a rich history of leveraging reinforcement learning to train agents that can play Minecraft. More recently, researchers have been looking at using language models to drive agents, like control a Minecraft character to chat with the player and take them on missions.

The field of Artificial Intelligence advanced from rule-based systems to neural network-based systems – and now with generative AI, we see interesting potential to innovate the experience of games. Players might be able to have deeper conversations, explore personalized play experiences, or puzzles and the world might change based on how they play.

We’ve already seen some indie developers pushing the boundaries of how generative AI can be implemented, such as ReLU studios in Korea, which released a game called Smoking Gun where players can chat with in-game characters, powered by language models, to solve a murder. Or the game Suck Up, from Proxima Studios, that gets players to play the role of a vampire, trying to talk their way into people’s homes. These examples are already showing new creative potential for AI in gaming. 

I think to truly allow game creators to explore the potential for AI, we need tools. We need to reduce the barriers to putting AI into games. We need to figure out the overhead cost of inference – which is still a question for many developers. And bringing down the latency for the optimal gaming experience.. Responsible AI has become a set of guidelines and tools with which to develop AI applications, and we needs to further expand these capabilities to game developer specific needs.

These are some of the challenges we are facing.

One of the projects we’ve embarked on is to collaborate with external startups who are rapidly turning AI into value-added tools. One such collaboration is with the company Inworld that is building tools to help game developers connect language models like GPT-4 to in-game NPCs to power conversations. At GDC this year, Ubisoft showcased an intriguing demo using language models that had a player plan a heist with a group of in-game NPCs.

Beyond NPCs, Microsoft Research also posited that one thing language models can do, is to take a lot of information and documents, reason over this information, and present it back in a summary, or a table, etc. The research and Xbox team collaboratively asked, what if we could use large language models to reason over game docs, and then create a custom interface bespoke to that particular game and story. We called this Narrative Graph. And earlier this year at GDC, we worked with Inworld to turn this research into a tool as part of the Inworld suite.

This is just the beginning of an exploration, and we think AI can be used to make tools that empower and augment the work of game creators.

The relationship between AI and human intelligence

Any new technology introduces new potential futures and questions about the best way to use it. The steam engine was first invented to be able to pump water out of coal mines in the 1700s. It took around 80 years after steam engines were widely used before someone applied the technology to moving people in a locomotive.

AI is advancing at a rapid pace, along with tools being developed to put these new AI foundation models to use across a variety of tasks. And I think right now there is an explosion of experimentation, people trying to figure out how to leverage AI for their businesses. And that can mean different things to different people.

I want to take a step back for a moment and consider what AI is, and the nature of innovation and really how society improves. There’s a really great essay called The Turing Trap from the Stanford Digital Economy Lab that talks about this.

Human intelligence is expansive and innovative. We’re more than the tasks we do. AI needs to become tools that augment our abilities, to help us do more. And in turn, even allow us to do new things we couldn’t do before. And the value for a business is that expansive value creation, with their teams at the core. We have to build AI tools that empower, not replace. Replacement by nature is reductive, it whittles down the potential to innovate. And is a short-sighted view.

How Gaming AI will empower players

We can see the ability AI has to empower players. We know one of the biggest reasons players stop playing games is that they’re just stuck, they can’t figure out how to get past a puzzle, or they just don’t get deeper into discovering all the different paths through a game.

Recently we created a video showcasing how AI might help players with assistance in games.  This is just a concept right now.

For this we worked with the game studio teams who really understood what their players wanted to get assistance in, and the best ways to play their games.

AI has the potential to benefit many and raise all boats, we have to really be guided by the north star of empowering, not replacing.

Using AI to support your team and develop talent

Most importantly, focus on empowerment, augment the work your teams are doing, freeing them up to innovate your business. Discover new tasks and new capabilities for your teams and your business this way.

This also means diversifying the exploration. AI isn’t a silver bullet. It can help in small but meaningful ways across the day, make incremental advancements in a complex workflow. And it’s important to experiment to find those areas where AI can really be helpful, vs just being a shiny new thing.

And invest in upskilling, training, and sharing best practices. For example, a really simple example is that I’ve been using AI Copilot features in Microsoft Teams to summarize meeting notes and action items. It’s a basic thing that takes time, and it’s great to be able to do it in a few seconds. It’s not accurate all the time, so I need to check it, but across the day it saves me time.

As you’ve probably heard many in the AI space say before, artificial intelligence is not something that will cure all of society’s challenges overnight. It’s up to today’s technical, creative, and business leaders to ethically use AI to help foster inclusivity, grow their teams (both in terms of their skillsets and their collaborative mindsets), and empower every member of their team to be the best that they can be.