Rediscovering testing with Horizon: Zero Dawn
Recently Guerrilla Games launched its newest and first open world game: Horizon: Zero Dawn. This got us thinking. Is game testing any different than testing 'regular' software? Together with Ana Barbuta, QA Manager at Guerrilla Games, we held a Meetup to find the answer to this question. In this blog post, we reflect shortly on how we look at testing in general and how this view fits with what we've learned about testing games.
Automate the expected outcome
We all know that in the 'regular' software world testing has changed massively over the last years. The introduction of Agile in IT organizations was a big driver of this change. Features are delivered faster and therefore test automation has moved from being optional to something necessary. Automating our test means that we need clearly defined scenarios, or scripts so that they can be executed by a robot. By running these tests continuously we check that the application is still doing what we already knew it should do. It gives us confirmation.
But what if we do not have a clear expected outcome?
The obvious answer is that we spend more effort in refining our stories. Use methods like specification by example to find examples of behavior that your software needs to perform and then we automate these examples. This however still implies that we are able, with enough effort, to think about all these possible outcomes.
Testing the unknown
Like with 'regular' software components at the lowest level are still tested by making use of automated unit tests. These checks are making sure that some of the base mechanics are functioning as expected. This is however limited to small parts of the game engine. Horizon: Zero Dawn takes place in a vast open world, so how the mechanics interact with the visual world is of great importance. Not everything can be clearly defined as right or wrong here. Is the sun not too bright? Are some areas not too dark? It will depend on the context of when a player visits a certain area and what was designed for him to see or not. Feel and playability is an important aspect here. If a certain level is too difficult, is this a bug? Things to think about and in the context of test automation even more complex.
What we learned from Ana is that this is where game testing starts to become different. In an open world game like Horizon: Zero Dawn there are an infinite amount of actions and interactions possible. When you've played an open world game you've probably been stuck in a wall/tree/rock/... at some point. In an open world game, the player can go wherever he wants. This means that the player can do things that you as a designer, developer or tester did not foresee. Other applications might have a multitude of possible action combinations that a user can make, but in games, this open choice is at the core of what makes a game great. How do you test for these things?
We often forget, but in essence, testing is all about learning and discovering. It is about finding out how an application behaves in a certain condition. In some situations, we don't know all the possible outcomes, but we still need to test that the product works well. When tackling problems in testing we tend to immediately start thinking about tooling and frameworks. In game testing, we need to take a step back. Most likely the tools often used when thinking about test automation don't work. Horizon: Zero Dawn is all about UI, but good luck using a framework like Selenium... For me, it was refreshing to see that while testing games you really have to think like a tester again. Think about odd scenarios, risks and how everything fits together. It really feels like rediscovering what testing is all about.
Learning is an important aspect of Exploratory Testing and when used in the right way, supported by tooling and automation it can be a very effective to learn about your application.
Simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next . - Elizabeth Hendrickson
Exploratory Testing sets clear goals for your tests. It provides the needed focus and structure. It is however up to the tester how to best reach that goal and to find behavior that might be unwanted. While documenting the results new areas for testing will pop-up so that the learnings of one test might lead to input for the next. When done right, Exploratory Testing is a powerful test method to use in game testing.
Experience game testing
Ana and her team have come up with some really cool techniques that have helped them to deliver Horizon: Zero Dawn with great quality. By making use of tooling and Exploratory Testing they were able to effectively test the different aspects of the game.
We don't want to tell you all the details yet since you can still have the opportunity to hear it from Ana herself. We are hosting this Meetup for a second time on August 29th. Just join the Test Masters Meetup group and we'll keep you informed!
Hope to see you there!
Erik Zeedijk: Test Automation Consultant Xebia
Ana Barbuta: QA Manager Guerrilla Games