Tuesday, September 3, 2013

Learning to Ignore Playtesters

In a rare non-theory post, Max gives concrete advice for knowing when to make changes based on playtesting experiences and when to ignore problems that playtesters find.

Perhaps the core practice of human centered design is observing users as they interact with your product. Product designers have learned that asking users what they want doesn’t do any good, because they don’t know what they want. In both human centered design and game design, it’s the designer’s job to find out what the user wants and give it to them - in order to do this, most of the time designers need to remember to follow this adage: “Watch what they do, take what they say with a grain of salt.”

To help you know what to take with a grain of salt, here are 3 things you will see happen at playtests that you should consider very carefully before taking action on. Sometimes it’s just better to ignore your playtesters and playtest notes.

3. Undue Praise
Unless you have a one way mirror in your home or playtesting facility, it’s very likely that you’ve noticed this phenomenon in action - you present players with a half-baked prototype that has known issues, you watch them play and note several major gameplay problems, and then afterwards they tell you that the game was amazing and that they wouldn’t change a thing.
This is a bias triggered simply by the presence of the game’s inventor: some (but not all) playtesters want to tell you what they think you want to hear! There are a few ways to mitigate the impact of this bias. First, I always try to distance myself from the creation of the game by responding to the inevitable “Did you make this game?” question with the sometimes lie: “No, it was made by another member of our team.” This is not always an option, but even if you don’t have a team you can sometimes get away with saying “I’m testing this game for a friend.” Second, I try to physically distance myself from play. I like to sit in a corner looking bored and quietly taking notes. The hope here is that the players don’t see me as a vested interest. Finally, (and attached to both of the prior strategies), I do my best to make it clear that I am not an expert. If they ask questions about play I shrug or reply that I don’t know and that they should look it up in the instructions. I answer questions about the design process in a similar fashion.

What if you do all these things and players are still praising your game? Well, it’s possible that the praise isn’t biased and your prototype is hot stuff. However, there are always changes to be made to better a game, so letting this praise go to your head is a bad idea no matter what. While not all praise is undue, all praise is unhelpful. The best solution is simply to observe your players. I’ve watched a player stare out the window for the middle half of a game, and then afterward assure me that it was the best game she ever played. Clearly, the middle half of this game was a lull for her; the start and the end were engaging, but she lost direction and motivation in the middle. When I took notes for this playtest, this was what I took notes on, not on the fact that she said it was the best game she had ever played.

2. Design for Your Target Audience (Not Your Playtesters) 
After watching a playtest with specific issues, it is always tempting to change your game to fix them. However, before doing so you need to consider: are the problems you observed going to be universal for all audiences? Even more importantly: are your fixes going to make the game better for all audiences, and not worse? If you answer ‘no’ to either of these questions, you need to either come up with a solution that will make the game better for all audiences, or prioritize which audience means more to you -- that’s what a “target audience” is; it’s not the only audience you want (because every game wants to be playable by a wide swath of players), it’s just the most important one.

I see this all the time in my games at Tiltfactor. We love our playtesters, but because all of our playtesters are volunteers this usually means that they are game enthusiasts, and so the feedback they give and the observations we make are almost always going to be biased towards hardcore audiences. In the end we always do tons of testing with our target audience (whether that’s medical professionals or teenage girls), but until that point while we test with our core testers, we have to use our experience and intuition to ignore comments and features that might eventually make the game worse for that target audience. At this point I’ve lost count of how many times one of our core testers has recommended adding a new mechanic while I smile, nod, agree, and write down: “SUGGESTION WAY TOO COMPLEX FOR TARGET AUDIENCE!”

These gamers are some of my favorite people but if I design a game for them it won't be for my mom.

1. New Games are Always Hard to Learn
Depending on the complexity of your game and the complexity your target audience is comfortable with, players may have more or less trouble learning your game, but they will without fail always have some trouble. The trick here is being able to determine when your playtesters are having trouble because learning a new ruleset is always rocky, and when they’re having trouble because your game has too much front-loaded information to take in, is explained poorly in the instructions, or is just generally too complex.

Unlike the other two types of feedback that you need to take with a grain of salt, this one is observational, which makes it all the more tricky to decide when your game could be made easier to learn and when to ignore.

I generally find that the best practices for dealing with this hurdle are simply: designing for easy learning as much as you can initially, fixing specific hangups in the instructions, and finally observing repeat players over multiple playtests. If players don’t have much trouble the second game and all the other parts of the game are working, I find that it’s alright at this point to overlook some amount of initial learning confusion.

Notes from playtests are always useful, and you should always record everything you see. Just note that not everything you hear and observe from playtesters should translate to changes in your game. Sometimes following their feedback or advice can waste your time, sometimes they need much more coaxing to relate the true problems they were experiencing, and other times following their advice can even hurt your game. Knowing when to listen to your playtesters and their experiences and when to ignore them is hard to learn, but I hope this primer will help designers avoid some common pitfalls.