Knowledge Keeper Quest 4...

Playtesting

This week's lecture afforded us a guest speaker, Pejman who taught all about playtesting, the truth and myth behind it and how it's done in the industry nowadays. One of the most important things to remember with playtesting is that Quality Assurance is NOT the same thing as playtesting. While QA focuses on bugs, system fails or technical faults, playtesting is concerned with players and how they react to your game.

This blog is going to discuss what playtesting is, different methods of performing user tests as well as some important things to keep in mind when performing them. We will talk about what playtesting is, what it isn't and where most development teams trip up in testing their games or interpreting the data collected. 

Testing the waters

Playtesting deals solely with player interactions and reactions to your game. The data collected should reflect their experience and how much fun they had playing your game. It is important to identify what parts they liked and, more importantly, what parts they didn't. In playtesting, it is the designer's goal to evaluate their intended design and how the player perceives it by identifying whether the game was played the way the designer meant for it to be played. There should be healthy motivation behind playtesting and certain considerations taken when it is being conducted. The tester's main motivation should always be the player as it is important to keep in mind that you are making the game for another person and not yourself. This is why reading how the player (and not the tester himself) interprets your design is crucial for good playtesting.

Steady focus on the tester's purpose and motivation ensures that the designer doesn't fall into the trap of getting too emotionally involved in the game development process. The whole point of playtesting is to avoid this and to make sure that the designers focus on what their players want and not what they themselves want to see in their game. One of the best sources of motivation is the focus on creating a good-quality game with plans for prototyping using an iterative approach. This keeps the designer centered on delivering the most value for players who are looking for quality in their game purchase.

An important practice to follow is embedding everything the player may need to play or enjoy the game within its design. This comes from a realization that you won't be shipped with the game, which means you may not be able to explain things to the player and you can't expect them to know you reasoning behind certain design decisions. At the end of the day you need to implement all your thoughts in a way that makes sense to not you, but your players. The steps in doing this is begin with a good design idea, prototype it, evaluate it and user test it. User testing helps you focus on your goals. 

Testing...one...two

A typical user test follows these steps:

• You create your product.

• Your run test sessions.

• You list issues found during these sessions.

• You analyze these issues.

• You select which issues to report.

• You report those issues to the development team.

• You create a new version of your game.

• The cycle begins once more.

You could be an independent play tester who reports to a client/company, or you may need to communicate your findings to the programmers and the rest of the development team. The rule of thumb is that one cycle of user testing usually isn't enough. There is certain criteria to follow in regards to user testing including representative, accurate, specific, timely, cost-effective, actionable, and motivational. Let's discuss these a little further. 

Finding a representative user (your target user) is the first and most important criteria for user testing. If you aren't able to find one, your testing becomes invalid. You cannot consider yourself a representative user of your own game, just as you can't let your colleagues or friends test either. You also need to be accurate with your results. In terms of accuracy, there are several way to increase the validity of your results including multiple methods of testing. Specificity is crucial in gathering good user data as you need t be specific about what was good and bad and how to improve your game.

All user tests must be maintained within an understandable time-frame otherwise results of the playtest will not be useful anymore. You will have to meet deadlines so choose the appropriate method that doesn't waste excessive amounts of your time. Make sure your results are cost-effective. If there's a $100 issue to fix which costs you $1000 to fix, it isn't worth it. Stick to your pre-defined game budget. You would also need to ensure that your issues are really fixable and aren't too big. This is called being actionable to ensure that your issues don't have the potential to break the whole game. The user test should also be motivational in order to help you encourage your team to fix the bug. 

User test labs are generally designed to simulate comfortable and real living rooms. Generally, televisions and several cameras are present to capture gameplay feeds as well as player reactions. Additionally, there are often observation rooms used to monitor the player as they immerse themselves within the game. 

Let's talk

The various methods of user testing include focus groups, observation, surveys and questionnaires, interviews, heuristic evaluation, thinking aloud, diary studies, game metrics, and physiological metrics. Let's explore these further.

Focus groups are typically for 4-8 people and are a common approach in market research. The concept is to invite people to an exchange of ideas. This kind of testing is great for the beginning stages in game development. It is best to recruit users and discuss concepts and types of features to get some general feedback from users. Sometimes the feedback can be too much as members may get distracted from the main topic or idea. At the same time, you must be open to new and different ideas. You need a good facilitator to make sure everyone in the focus group gets a chance to talk. Someone who will make sure that dominating ideas won't push their way through. Focus groups are good opportunity to hear good ideas and to explore certain ideas on a deeper level. This method could also be used later on in the development process. For example, after they have played the game, the user may talk about how it made them feel.

Observation studies are one of the most common methods of user testing. You begin by creating a playable version of your game and then observing how the user plays the game. You really just need a game system and someone to watch. It provides good data and you get to see how users interpret your design. It can be intrusive for the player to have you watch over their shoulder and you need to make sure they know that you’re not looking at how well they play the game. They should have absolutely no pressure on them. It is important to know who is observing the gameplay. Usually you will have someone just conducting the session who isn't part of the development team. At least 2 game designers will then monitor the results. You need to be careful about your own interpretation of what the player is doing which is why having 2 people is ideal. Observing body language is very important for finding trends in players' emotions in relation to game moments. It can be time consuming to re-watch sessions for analysis.

Surveys and Questionnaires are easy and fast common methods to use. You would typically ask your users to fill out the questionnaires after they have tried out your game. You can often generalize surveys making them convenient for analysis. If you’re collecting it in scales, it’s even easier as you can use the information to make graphs and charts. You could use this method to collect some demographic information before you begin the game. After the gameplay is done, it may be used for overall experience testing. What’s the player's game type? Do they like to explore? Find out what your target audience likes and find a way to deliver this. You shouldn't interrupt players to ask questions, but there are acceptable times to pop in during gameplay (after the player dies). One of the most important considerations with surveys and questionnaires is that if you forget to ask something you don’t know the user’s opinion on it. Make sure you know what you want to know before you begin your questionnaires. Also make sure you frame it right – don’t put pressure on the user and don’t have them answer the way you want them to. It is also important to have clear consistent scales with your questions like the Likert-scale. It is impossible to follow-up a questionnaire because after collecting the results there might be something that you can’t explain, but you might not be able to get back into contact with your testers. You also need a large sample of people to conduct surveys or questionnaires. If you’re hoping to collect data online, it’s not that easy to find users to fill out a questionnaire. This is especially true of very specific questionnaires.

·   Interviews act as a one-to-one discussion between the developers and the testers. In playtesting, you usually get the user to interact with your product and then you sit down and talk about their experience. You need to know who you are interviewing. Maybe you’re making a game for doctors in hospitals, so you might want to interview professional doctors for your game (interview knowledgeable people and get their ideas). You usually interview players to go deeper and find out what matters to them. The guidelines of creating a good questionnaire works the same way for an interview. You need a good moderator in addition to having the questions that you want to ask setup beforehand. You could always be more flexible and ask more follow-up questions. Ask relevant questions. When you are questioning people, they need to be aware of the experience they had and remember it so that they might be able to explain it to you. Therefore, it is imperative that you don’t make play sessions too long. Your players might try to please you, so be wary of making them comfortable with criticizing the game. A good technique it to tell them it isn't your game and that you’re testing it for someone else. Let them know that you’re only interested in improving the game and not in knowing how well they did. Make sure you take notes – you will not be able to remember information later on. If you can record the Q&A session, even better! Don’t do the interview alone, get another team member to take notes. Have all your notes organized in a way you can review them (transcript) so that you can find common threads. Combine all the data and find the issues you’d like to fix.

·    

   Heuristic Evaluation deals with having design guidelines (questions) which help you evaluate your game. Have a game design expert evaluate your game and then compare the results to the list of heuristics you have. There will be a number of questions which will help you evaluate your game to see whether you’re succeeding or breaking the game. A problem with this approach is that it’s very subjective as it is based on the evaluator’s opinion. So having 2 or 3 members act as the experts is usually ideal. This method is cost and time efficient since you can be your own expert. Use this method at different stages of game development. When you have some ideas, compare the mechanics to your heuristic list and see if you are or aren't going against your list.

     Thinking Aloud can be done with observation. Ask your users to talk aloud about how they feel. The approach generates a lot of useful data as you can see how users interpret your design. The problem is it may be difficult for users to speak like this naturally. There are some users who might not be good at thinking aloud. They might do it for the first 2 minutes and then stop talking due to riveting gameplay. Try it and see how it works for your game/ in your setting. It’s a good idea to record the player’s comments and gameplay at the same time.

·        

    Diary Studies are suitable for observation over longer periods of time. You leave the user with the game and a journal to record their experience. Ask them to record this for a couple of weeks to allow them a longer period of time spent interacting with your product. After the study is over, you have a rich source of data which you have collected. It’s a good method of collecting data on player behavior and demonstrates how your game influences this. The hard part is having dedicated users to be disciplined in following the journal entries. It’s a good idea to keep in touch with your users throughout the study. Weekly meeting are a great idea. A Google doc to track entries may be helpful. The response can be slow since it’s a time-consuming study. Sometimes users are paid (at the end of the study) keeping them motivated to carry on through the study. Invite them back at the end of the study to interview them and discuss their experience. 

In Conclusion

As a play tester you need to know what you want know and how to figure this information out. Don't forget about the critical criteria for user tests and make sure you find appropriate representative users to participate in them. Pick methods based on your resource constraints and based on the questions you're trying to answer. And finally, listen to what your users say and compare this information what your intended game design was. Happy testing!