Sijo Jose is president, North America and India, at testing firm PTW.
The gaming industry is at a crossroads. Gamers expect more connectivity than ever, with as-a-service models transforming the industry similar to how they’ve revolutionized software for consumer and business applications.
Today’s “Netflixization” of gaming means audiences increasingly expect personalization, uncompromising performance and more choice than ever. The November launch of Google Stadia, one of the most ambitious projects in the ever-fragmented gaming landscape, offers significant lessons for this exciting future of gaming.
Viewed by some analysts to be one of the major shifts in gaming, Stadia showcases the critical nature of testing as studios look to thrive in an increasingly competitive market.
Surveys of gaming industry companies show between 15-40% of development is spent on quality assurance and testing, showcasing the critical nature of testing in ensuring the success of virtually every gaming title.
At PTW, we learned valuable lessons from Stadia’s launch, working across three titles for one of the largest and most storied gaming studios in the world. These insights offer a unique window into how Stadia and other platforms can drive a bright future for the global gaming community.
The Stadia support team was highly collaborative and offered what we found to be industry-leading supportSijo Jose
With increased volumes of games and varieties of platforms on the way, developers and platforms alike must rethink their approaches to testing, taking into account lessons from Stadia’s launch in order to delight players and exceed audience expectations.
Here’s what we learned from our own Stadia experience.
Collaboration and Readily Available Technical Support are Critical
Where did things go right? The Stadia support team was highly collaborative and offered what we found to be industry-leading support, making their staff available to our team as an on-call basis for the continuously updated SDK and the respective nuanced technical requirements.
They were professional, understanding and willing to answer technical questions throughout the process. While access to the partner portal was sometimes limited and unreliable, the Stadia support team made themselves available for troubleshooting on a variety of issues.
Without this close collaboration, the launch would have likely been marred by greater technical challenges and potentially less availability of gaming titles. Our experience shows partner support is critical to the success of titles, particularly to new platforms.
Ultimately, close collaboration and communication with the Google Stadia team provided a roadmap forward. The Stadia support team provided significant and ongoing support to our testing team, particularly across the technical requirements and documentation phase of our gaming titles, and we were happy with a successful launch for our three titles, with positive gamer feedback - in spite of broader platform issues.
Behind the Scenes: Stadia’s Uphill Battle for Testers
Our experience in testing offers a window into how the broader gaming industry can face the challenges of a platform launch. Advertised as a platform with unprecedented speed, accessible by just a browser, the vision of Stadia is an ambitious one.
Unfortunately, Stadia initially faced the challenges of latency issues and general title stability, both common issues for new platforms. While the community of fans continues to grow since its November launch, initial press reports found influential gamers were discouraged by its performance.
Addressing the ever-increasing expectations of gamers demands world-class testing. When executed properly, certification testing in particular has a tendency to uncover would-be-fatal flaws in game titles.
We found the technical requirements of Stadia were often a moving target, with parameters changing weekly in the lead-up to the launchSijo Jose
These issues can range from unacceptable performance issues (slow speed) and poor in-game experiences (bad game mechanics) to failures for meeting Stadia’s specific technical requirements. These challenges can render promising titles into flops.
Unfortunately, technical challenges with Stadia testing leaves studios vulnerable to these risks. We found the Stadia testing environment was more complex and unreliable than most platforms.
As we worked on titles for a leading studio, we interfaced closely with the Google team to ensure Stadia’s Software Development Kit (SDK) was installed and configured, with features deployed correctly in advance of the launch.
As we dug deeper, we found the technical requirements of Stadia were often a moving target, with parameters changing weekly in the lead-up to the launch.
With dense technical requirements from Google, our team found the technical documentation was often lacking for testing, likely because it was geared toward developers. As a result, testing meant liaising with support to answer performance questions that had no answers readily available.
From our experience, certification testing for Stadia is a grey area for many developers as well. As both the SDK and technical requirements were continuously updated, our experience was that games would be compliant with different versions based on the release date.
Finally, for the partner portal, which functions as the key point of access as testers, we found there was inconsistency between what worked for the accounts on partners’ side versus the access of testers accounts.
Because of these and other challenges, many certification issues arose and proved difficult to troubleshoot. Stadia allowed some titles to receive waivers for certifications in order to be able to launch games in time, requiring games to be patched within short amounts of time to meet all the technical specifications.
It’s clear that many exceptions and last-minute adjustments were necessary for the project’s November launch.
The Look Ahead
Stadia is at the forefront of monumental changes in gaming, from audience expectations of delightful and personalized gaming experiences, anywhere, to the Netflixization of gaming and unprecedented level of choice.
As a flagship example of these trends, lessons from Stadia’s launch provides a unique window into the state of the gaming market.
We’re bullish on the future of the Stadia platform and the trends it represents. However, with as-a-service revenue models exploding, audiences have heightened expectations for performance and connectivity that must be met.
Given the mixed reviews from gamers and critics alike, it’s clear Stadia must continue to build trust across the community and meet the unprecedented expectations of gamers for high performance, features and availability. Close collaboration with studios, testers, and all parts of the development ecosystem is critical.
Companies should pay more heed to testing, particularly given that it can represent a significant portion of the development process. We see a bright path forward for the industry and look forward to building upon our experience to support an exciting future for Stadia.