Putting Your Game at the Center with Xbox Touch Controls

Touch controls are one of the top player-requested features for cloud gaming titles. Xbox touch controls unlock new ways for gamers to interact with their games, providing fans with even more choice in how and when they play. Working with game developers to bring touch controls to over 100 games in the Xbox Game Pass catalog, we've learned a lot along the way from working with our partners. Here is a little about the journey.

Benefits

Touch Controls with Xbox Cloud Gaming enable players to enjoy games wherever they are by removing the need for a console, PC, or controller. We have seen a tangible increase in usage and user sentiment across different genres for games that have implemented touch controls.

20% of our Xbox Cloud Gaming users use touch as their exclusive method of playing games. As such, it's important to us that the touch-enabled games we launch are relevant and, most importantly, play well with touch controls. We've seen, on average, a 2x increase in usage for titles available via Xbox Cloud Gaming, across genres, that implement touch controls, and the sentiment of users who play with touch controls is on average equivalent if not higher than on the Xbox Cloud Gaming platform all-up.

Additionally, we've seen even more engagement with touch controls for many games on an individual basis. Below are some of the 100+ titles where more than 30% of players exclusively use touch controls.

• DRAGON QUEST® XI S: Echoes of an Elusive Age™ - Definitive Edition
• Football Manager 2022 Xbox Edition
• Hades
• Minecraft Dungeons
• New Super Lucky's Tale
• Scarlet Nexus
• Yakuza: Like a Dragon

How it works

Xbox Cloud Gaming introduces many new and exciting opportunities to game developers. Running a fully powered console title on a mobile phone or low-powered PC simply has not been possible in the past. These devices include touch capabilities, and people have come to expect experiences to adapt and make full use of the form factor. Moreover, the promise of Xbox Cloud Gaming is one of accessibility and reach; cloud enables the incredible experiences that game developers craft to be enjoyed on-the-go, by customers who have never owned a console, and in new and emerging markets.

When thinking specifically about touch controls and mobile form factors, Xbox Cloud Gaming offers several mechanisms for developers to tap into the power of a range of devices. Because developing a console title often a massive multiyear investment, a core principle when thinking about how to leverage this new functionality is to recognize the hurdles of changing game code, and therefore provide a smooth on-ramp of development investment that provides progressively more benefit.

No Code Changes

The first level of investment for touch controls involves no codes for the game. Using the Touch Adaptation Kit, or TAK, game developers and designers can easily craft an overlay UI that provides touch controls for players and translates those actions into standard gamepad inputs that the title is already familiar with. This overlay is a simple JSON file that declaratively tells the system what touch controls to show on screen and how those controls map to a gamepad. With this system, a game developer is able to provide a more ergonomic and fun experience. To get started with the Touch Adaption Kit, see the getting started section at the bottom of this blog.

In just a few minutes, developers can take a sample layout from GitHub and customize it to their game's specific needs.

One key learning we've seen so far is that while a traditional gamepad control scheme often utilizes up to four simultaneous inputs, touch interfaces are much more constrained. Careful consideration of where to place controls and how to map them is critical to providing a fun experience. The touch adaptation kit is even helping experiment with new paradigms like allowing for motion controls in combination with touch.

The below Sea of Thieves touch layout shows all of this in action with its customized layout. The control placements and iconography allow players to control the game easily and intuitively. The TAK has a library of built-in icons for common actions to help provide players with the in game semantic meanings of different controls.

Sea of Thieves

Customized Layout Look

Taking the simple layout one step further in investment, is truly customizing the look and feel of the touch controls to match the game's own art style. Simply by using custom art assets, Hades with touch looks and feels like a native and natural experience with icons matching the game art. Again, this level of touch integration requires no code changes and can be completed even after a game has shipped.
 
 
Hades

"Another way it's thoughtful from Supergiant's perspective, we were delivered a working proof of concept. We didn't have to figure out all these different things. There are lot of smart people on the Microsoft side that had done that. For us it was just a light feedback pass. We've put so much love, time and effort into this game, it's amazing when you have a partner that's trying to honor that. It's really cool, and it has great potential. We say that because we've played with it, and we think it's good."
-Amir Rao - Studio Director - Supergiant Games

New Super Lucky's Tale

"Bringing New Super Lucky's Tale to xCloud touch wasn't just easy. It was delightful, involving little more than running the game and discovering how fun and intuitive it is to play on a phone. It's just one of the many innovations Microsoft is bringing to gaming right now and we couldn't be more thrilled with the opportunity it gives us to share Lucky with even more players!"
-Playful Studios

Minimal Cloud Aware Changes

Building on top the concepts from a single customized layout, Xbox Cloud Gaming allows game developers to dynamically switch layouts from game code. While this does require cloud aware game code changes, the investment is fairly minimal. By mapping different unique modes of the game to different layouts (think menu or driving or other times the control scheme would naturally change), a game is easily able to show the right layout for the right moment in the game. Just like above, these layouts also allow for custom styling and semantic iconography to make the touch controls natural and intuitive to players. A few examples of how this capability can be used:

• If the player is in a cutscene, non-required buttons can be removed from the screen and re-enter once the cutscene finishes.
• When the player transitions from exploration to a battle, the layout changes to ensure all of the appropriate buttons are on-screen in an intuitive format.
• If there are any actions that are unlocked at a later point in the game, keep them hidden to avoid any spoilers or player confusion.

The below images and quotes show this capability in action and what it took real studios to make use of it.

  
 
Yakuza: Like a Dragon

"Yakuza: Like a Dragon provides the best touch layout for each scene to bring better game experience even in game streaming. Specifically, this was achieved by using the Touch Adaptation Kit to easily place button positions and button icons in the desired way. For battle scenes that have evolved from traditional action battles to live command RPG battles, we chose icons that are intuitive and make actions easy to understand. Also, for the event scenes, we tried to avoid covering the touch layout to make it easier to immerse players in the events. The implementation is really easy. We're convinced that it has brought great gaming experience that makes game streaming easier to play and more immersive."
-Ryu Ga Gotoku Studio

Scarlet Nexus

"Since this title was not originally developed with the idea of being played on smartphones, we were not sure how well it would be able to handle controls on smartphones at first. The first problem we faced in development was that this game is an action game with complex controls. In particular, simultaneously pressing multiple buttons was a major challenge. In order to solve this problem, we added a dedicated button for simultaneous actions, so that you can use a single button to input instead of simultaneously pressing multiple buttons. In addition to the ease of operation, we also pay a lot of attention to optimizing the button layout for each scene, such as eliminating buttons as much as possible to make it easier to see the entire screen in cutscenes where we want to show. Because of our efforts in optimizing for Xbox Cloud Gaming, we were able to appeal to users who don't have Xbox consoles, and we felt that there was a lot of potential for future title development. "
-Scarlet Nexus Development Team

DRAGON QUEST® XI S: Echoes of an Elusive Age™ – Definitive Edition

"For the DRAGON QUEST® XI S, there're two game modes. We have adjusted the 3D mode by rearranging the buttons for better ergonomic accessibility, and the 2D mode by hiding most of the unnecessary buttons and enlarging the Directional pad to make it easier to play. Although we were already in the final stage of development when we were asked to support Touch controls, it was very easy to integrate into the game and create custom touch layouts."
-DRAGON QUEST® XI S: Echoes of an Elusive Age™ – Definitive Edition Technical Director

Cloud Aware Title

For developers looking to fully harness the capabilities of a player's device and provide a best-in-class experience that seamlessly adapts and transitions to different form factors, Xbox Cloud Gaming includes several additional techniques that a developer can leverage. From scaling UI in order to make it more readable, to creating a truly touch/mobile first experience, or to even handling the realities of latency with the cloud, Xbox Cloud Gaming has the developer covered. These more advanced techniques do require a developer to crack open game code and even possibly rethink some game or interaction design. The rewards, however, are a title that looks and feels like a truly native experience and a more adaptable design that may even benefit players in non-cloud scenarios.

UI Scaling

When your game is played via Xbox Cloud Gaming, your players may be experiencing it on screens ranging from 4k TVs all the way down to the phones in their pockets. User interfaces which look great on a large screen may be too small to work well on smaller screens. Cloud aware games can use the XGameStreamingGetStreamPhysicalDimensions - Microsoft Game Core | Microsoft Docs API to measure the physical size of the game as the player is seeing it. Using this information, a cloud aware game can calculate a sort of "virtual DPI" to help figure out what the appropriate scale for its user interface should be. For instance, if the game is rendering 1920 x 1080 pixel frames, and the player's device is rendering them with a size of 110 x 62 millimeters, the player is experiencing the game at approximately 443 DPI. With this data in mind, the game can then select the best size for their UI elements to make them the most usable for the player.

Dynamic Touch Control State

Games also have the ability to update properties of the touch layout at runtime to reflect the state of the game. Properties such as the visibility, look, or even the input that a control is mapped to can be updated to match the current game state.
A few examples of where this may be useful:

1. A title may allow a player to pick up and equip up different items in game. In order to reduce visual clutter on screen, a developer could use dynamic state to substitute out an appropriate art asset to match the currently equipped item and hide a game rendered hot bar.
2. A title may allow a player to remap controls in settings. By using dynamic state, a developer could propagate these changes to the touch controls and even offer some touch specific player customization.
3. A title may not need every control on screen because the player is out of ammo or hasn't learned a super ability yet. Rather than creating a layout for every possible player state, a developer could use dynamic state to hide or show controls as appropriate.

For more information on this capability and how to think of it versus switching an entire layout, check out the full documentation at: Changing Touch Control Layouts Based On Game State - Game Streaming | Microsoft Docs

Native Touch

In addition to using the Touch Adaptation Kit to render on screen controls, titles can also receive the raw touch inputs that a player is using to interact with the game. This allows you to build touch experiences which really feel native to mobile. Titles can build natural mobile interfaces, like tappable buttons on menus, pinch to zoom and pan to scroll on maps, or even tapping on the place that you'd like your character to walk to.


The Touch Adaptation Kit and native touch interfaces are not mutually exclusive - for example Minecraft Dungeons has a native touch interface in their menus, maps, and inventory system while using the Touch Adaptation Kit for the main gameplay loop. Additionally, a fully customized touch implementation requires only a relatively small investment of time and resources. For instance, a small team at Mojang spent only seven weeks to develop the fully custom touch interface for Minecraft Dungeons.

Football Manager 2022 Xbox Edition

"The relatively recent arrival of touch control into the mix has added a few challenges to the process, but it's also thrown up a few unexpected bonuses. For example, take the tactics screen (see screenshot). This is one place where the touch screen makes manipulation simpler and more intuitive even than the original mouse/keyboard combination. Touch controls also aid in the Match Day experience; shortcuts to Shouts, Instructions and Mentality can all be accessed effortlessly when quick changes are needed to turn the tide in a crucial match"
-From the desk of Stuart Boyd, Senior Console Producer at Sports Interactive

Getting started

Below are some of the resources you can use to get started with Touch Controls on Xbox Cloud Gaming:

• Introduction to game streaming - Microsoft Game Core | Microsoft Docs
• Getting started with touch - Microsoft Game Core | Microsoft Docs
• Xbox-GDK-Samples/Samples/xCloud/SimpleCloudAwareSample at main · microsoft/Xbox-GDK-Samples (github.com)
• Sample layouts to get started with touch: microsoft/xbox-game-streaming-tools: Tools for Xbox Game Streaming (github.com)
• XGameStreaming (API contents) - Microsoft Game Core | Microsoft Docs

We hope you have found this information helpful and inspiring! Cloud gaming is lowering the barrier of entry to gaming. By leveraging touch adaptation and cloud aware technologies, your game can reach more players than ever before, including players without consoles or even controllers. With Xbox Cloud Gaming, you can build a game once and play anywhere on any device, empowering your game to reach players at a global scale. So check out our public documentation! Take a crack at touch controls and cloud aware APIs. Get started with these touch layouts and cloud aware samples.

Reach out to your publisher or Xbox program representative if you are interested to learn more.