Monday, August 26, 2013

The Psychology of Rewards in Games

In this week’s doozy of a post, Max discusses what’s going on psychologically when games reward their players. During the discussion, Max explains reward schedules, the overjustification effect, and what both psychological phenomena mean for game designers. He also tackles the age-old question, “are variable ratio reward schedules inherently nefarious design?” and answers with a resounding “no!”

Over the past 2 years I’ve had the privilege of working with at Tiltfactor alongside Dr. Geoff Kaufman, who has a PhD in social psychology and is our head of research at the lab. In addition to the fantastic insights he has made running formal studies on the games we’ve been developing (publication pending on those), I’ve learned amazing things about psychology, formed many theories about how it influences game playing, and developed a list of things game designers need to keep in mind. I’ve devoted a lot of thought recently to one of game design’s most hotly debated psychological topics, reward schedules, and how they relate to what psychologists call the “overjustification effect.” But don’t worry if you’ve never heard of either of these things! I’ll explain them both before discussing what they mean for game designers.

These theories, while most obviously applicable to digital games and tabletop roleplaying games, are important for all types of game designers to understand, especially in an era where even traditional board and card games are becoming more digital.

Reward Schedules
In the 1930s, Burrhus Frederic "B. F." Skinner, a psychologist at Harvard, invented an Operant Conditioning Chamber (better known as a Skinner Box). The concept was simple: put a rat in the box. Let the rat pull the lever in the box. Sometimes give the rat a food pellet for pulling the lever. Study what conditions cause the rat to pull the lever more or less often.

Of course Skinner Boxes also include the capacity to shock the creature because scientists are freaky like that.

The applications in game design become clear when you look at what Skinner and other psychologists found. While experimenting with pigeons, researchers found that the pigeons were more likely to push the lever more often when there was a only a chance that they would receive a reward, even more often than when they always received one. Specifically, they were most active when the chance of receiving a reward was 50%. This is an intermittent reward schedule: it gives a chance at payoff for any given action. Specifically, they found that the most effective reward schedule was a variable ratio reward schedule - inserting randomness into the equation such that there could be many pulls of the lever with no payoff, but the average payoff is set.

If this behavior can be extended to humans (and let’s be honest, it can), we can be controlled to perform an activity more often simply by giving us a chance at a reward instead of promising us a guaranteed reward. We tend to know this intuitively: it’s why we gamble. And tons of games already use these principles. For example, slot machines are basically Skinner Boxes for humans. Zynga is notorious for using variable ratio reward schedules in their social games like Farmville. Even World of Warcraft uses them by having killed mobs only drop the loot you need for quests some of the time and not all of the time.

Skinner Box for people

The use of variable ratio reward schedules in game design is often panned, however, for being “nefarious.” Detractors’ reasoning goes: if the game designers had chosen to use simple fixed reward schedules (where for each action there is a promised reward), the players would play a certain amount. With variable ratio reward schedules, the players are more active. Thus, the game designers are “tricking” the players into playing more than they really want to, and usually also spending money.

Before I go into the ethics of using variable ratio reward schedules in games, I want to talk about another crucial phenomenon that is often overlooked in discussions of morality and reward schedules: the overjustification effect.

In 1973 psychologists Mark Lepper and Richard Nisbett conducted a fascinating experiment with kindergarteners. They observed that the children were intrinsically motivated to draw —that the kids enjoyed the activity of drawing pictures for its own sake without any need for external payoff. They split the children into three groups. The students in the first group were promised ribbons as a reward for drawing. The students in the second group were given ribbons, but were not promised them beforehand. The third group was left alone to draw in peace. While the experimenters handed out ribbons, all three groups drew comparable amounts of drawings. Then they stopped giving out ribbons. The third group, as would be expected, continued to draw the same amount (after all, nothing had changed.)  The first group, however, had a significant drop off in the amount of drawing they did.

Psychologists theorize (and they’re pretty certain at this point) that what happened is as the first group received ribbons, they shifted their motivations from “I’m drawing because I like drawing” (intrinsic) to “I’m drawing because I want a ribbon.” They were still motivated to draw so long as they were receiving their rewards, but once the rewards were removed the motivation did not snap back to being intrinsic.

This is an often overlooked phenomenon that occurs in many sorts of games. I only have personal anecdotes to share, but if you play many video games I’m sure you’ll be able to identify times when the overjustification effect happened to you.

For myself, after beating Diablo 3 I continued to play and sell my rare items for money on their real money auction house. I built up quite a store of rare items that I was slowly selling off. I played this way for several weeks, but then something changed: Blizzard released a patch making newly dropped rare items much more powerful. This made all of my collection worthless, and I quit after reading about the patch. I haven’t played since. What happened here was I shifted my motivation from “I like playing Diablo 3” to “I like selling items on the auction house.” This was all well and good until Blizzard took away my chance to cash in on my efforts, and then I simply quit.

In another personal example, I played League of Legends, all the while working towards unlocking a single summoner spell: the renowned “Flash” spell. As soon as I unlocked it I quit. I didn’t even play one game with it unlocked. I had shifted my motivation for playing the game to “I want to unlock Flash,” and as soon as I did I found I had no motivation left to play.

Cue 1000 screaming fan boys on how overpowered Flash is and how I should totally play with it.

In both of these cases the games presented me with a goal to work towards that undermined my intrinsic desire to play the game. From a designer’s point of view this is fine, so long as the extrinsic motivations (rewards) remain. So long as I could continue selling items on the Diablo 3 auction house, I would keep playing. Unfortunately, there are two obvious occurrences that take players’ rewards away. The simplest one, exemplified by my League of Legends story, is that the player can achieve the reward. The other problem, as shown in my Diablo 3 anecdote (and many others) is the reward becoming obsolete. Between these two issues it’s difficult for a designer to add fixed reward schedules into a game without risk of the overjustification effect coming into play.

Giving Rewards
Giving rewards in games is desirable. Designers want to give players rewards for numerous reasons, including reinforcing player behavior, increasing players’ feelings of mastery, scaling difficulty over the course of gameplay, and scaffolding mechanics and player abilities. So how can designers give rewards with the perils of triggering overjustification looming overhead and threatening to make their players lose interest in the game? Lepper and Nisbett’s second group of kindergarteners (the group I didn’t reveal the results for) give us a hint at the answer. Recall that this group was the one that were given ribbons after drawing, but were not promised them beforehand. Once the rewards were removed, this group continued to draw at the same rate as group number three.

This gives designers the solution to providing rewards while avoiding shifting players’ motivations to solely wanting rewards! Don’t let the player know for certain that she’s going to get a reward —also known as variable reward schedules. It’s pretty straightforward: when a player knows what she’s going to receive by means of a reward, she can play only for that reward. It’s much harder to make that motivational shift when the reward is uncertain. This is why games like Dota 2 give items as rewards after games fairly infrequently, at (almost) unpredictable times and with random quality: this way players can enjoy the surprise of the reward without banking on it.

Dota 2 gives payoffs when the "battle experience" bar up at the top fills up.  I couldn't for the life of me tell you how many battle points I have, which means it's as good as random and Valve is doing a great job!

My point here is that not only are variable ratio reward schedules not inherently evil, they are actually good game design practice. Variable ratio reward schedules don’t trick players into playing more than they really want to, they trick the players’ brains to prevent them from shifting their justification for playing to external factors. This means that these reward schedules keep the players playing because they find the experience itself fun and not just to get to the next reward, AND THIS IS EXACTLY WHAT GAME DESIGN IS! Game design is the process of making systems that players find interacting with fun and worthwhile without external justification. Omitting variable ratio rewards when they could be included is just like ignoring other established design theory (say scaffolded learning or positive feed back loops) that we’ve discussed on Most Dangerous Game Design previously: it’s simply shoddy design practice.

Can designers abuse the psychology of reward schedules? Of course! Most of human psychology can be abused and often is. However, even when using psychology, “tricking” players into having more fun still counts as giving players more fun.  I know this entire arguments tends to raise peoples' hackles, so please feel free to argue with us in the comments, as always!


Lube said...

I strongly disagree with your opinion about variable ratio rewards being a mechanic to not shift the justification of playing to external factors, and I honestly Im amazed at the videogames example you are using.. If you are really going to use videogames example to prove your point, or to explain it, you should really get into them, because for example nobody in the entire world would play lol to get to use Flash...

In reality unblocking summoner spells as you lvl up, makes you really weight them and learn to use the more simple ones before you delve into this OMG IS FLASH thing you got going on.

And I cannot fathom how would you get to your conclusion after talking about the Nepper and Nisbet experiment, where they show that when people play just for the sake of playing, they dont look at external factors for keep on with this activity.

Ramenhotep said...

Hi Lube,

I would love to hear specifically what you disagree with about my thesis! The point of the flash example is not that the player (in this case me) was ever initially playing with for the purpose of getting flash, or indeed ever admitted to himself that he was playing to get flash. Instead what happened was that motivation for playing shifted subtly over time towards earning battle points in pursuit of progression and away from playing for its own sake (although the game was still enjoyable). Part of the point is that players cannot consciously decide their motivations for doing an activity, or shift them at will. If players could this wouldn't be an issue. I cannot for certain say that this phenomenon (the over justification effect) has impacted anyone else's gaming experiences beyond my own, but I highly suspect this is a common occurrence.

The point of the Nepper and Nisbet experiment is not, as you say, that "they show that when people play just for the sake of playing, they dont look at external factors for keep on with this activity." Their conclusions are in my understanding quite the inverse - that when people look at promised external factors, they don't (indeed often can't) do an activity for its own sake. In effect, the variable ratio rewards allow the designer to provide the external factor but remove the promise, to help players not focus on it.

I agree that my examples may not be the strongest, but they are all authentic experiences that I've had. What do you think?

Anonymous said...

Great article. But I can see how the previous guy jumped to his own conclusions instead of shifting perspective. I feel like you didn't dive into the psychological examples enough. I feel like I understood what you were saying, because I already knew half of it. In other words my context carried me through it.

What was most valuable for me was the clean defense. Stuff like "Tricking people into having more fun is the same thing as giving them more fun" is conversation gold. I just hope I can remember it when I need it.

My site btw:

Ramenhotep said...

You're right, budleiser! In the end, I can only really speak from my own experiences, and if someone hasn't had those experiences (or I can't write well enough to have them recognize those experiences), then they won't get much out of the post.

Unknown said...

Interesting article! I´m a psychology student (finishing my bachelor soon) and am looking into game design right now to, well, maybe at some point create my own game, do research, basically, having a job that is related to video games. I also liked how you pointed out how playing for external rewards might be unethical and that you think it´s better to have the player play because he likes the game, not "exploiting" the way the brain works.
Will definitely check out your blog more often! :)

Unknown said...
This comment has been removed by a blog administrator.
Unknown said...

The reason why you are wrong is in a pay to play environment RNG is used as a revenue generating mechanism it only exists to artificially extend ones subscription by keeping the items they desire just out of reach and instead of investing in meaningful content to keep players engaged to continue to pay to play they use this cheap trick.

Unknown said...

Example Square Enix actually invests money into Final Fantasy XIV in order to allow them to develop quality content on a consistent 3 month basis and players knowing that the developers actually care about giving them that is what keeps people subbed.

Ramenhotep said...

Sir salisbury,

If RNG is only used as a revenue generating mechanism, why would games like Skyrim, or No Man's Sky, or ____insert any RPG name here___ use RNG? Games without subscriptions or microtransactions don't have much of a revenue incentive to keep you playing after you stop finding the game fun, and yet many still use variable reward schemes. Why would they do that if there wasn't another benefit?

I'll concede that I may have overstated my argument in the post (it's several years old and my zeal has mellowed a little bit). Can RNG and variable rewards be used exploitatively to milk players for revenue? Absolutely! But I argue that variable rewards can have legitimate design uses, and can be used without exploiting players. Of course, as you point out, variable reward schemes aren't a substitute for content and other good design elements, but I would argue that they can be used as a complement to quality content.

Unknown said...

In single player games RNG is fine because you buy the game up front there is no monthly fee like MMO which in cases lik swtors new Galactic Command system which ties end game gear to a sub only based RNG exp bar is clear designed to get as much out of players for as long as possible.

. said...

Hi Max,

I enjoyed reading this article.

However, you made an error when you wrote this sentence: "Once the rewards were removed, this group continued to draw at the same rate as group number one."

If you recall, the group number one drew less after they received their reward. Not more.

I looked at the study, and it does say that group two's interest in drawing did not decrease (as the group one's did). So the study supports your logic. But that sentence you wrote above does the opposite.

As an aside, you might want to think twice about enrolling your toddlers in a preschool on the Stanford Campus - geez. Or any place that has 1-way mirrors for observation. The whole idea of treating kids as lab subjects seems pretty messed up, to me (but a decent premise for a sci-fi show - I'm thinking of you, Fringe). Who knows what kinds of weird fetishes they induced in these kids. "Would you like to come with me to the surprise room?" Yikes! The racist part (they excluded the kids who weren't white!) was especially egregious, in my opinion.

Fun study.

Ramenhotep said...

Hi .

Oops, you're totally correct. It should read "Once the rewards were removed, this group continued to draw at the same rate as group number three." My mistake and thanks for catching this-I'm going to fix it in the text.

Let it forever be known that it originally read "Once the rewards were removed, this group continued to draw at the same rate as group number one," mistakenly.

Fortunately the Institutional Review Board reviews for skeezy experiments these days, even if they do make our lives much harder. Skeezy or not, though, I really really want some one way mirrors in our lab.

Kevin Q said...


The very suggestion that humans can be controlled through this sort of task vs. reward method is preposterous as well as completely false.

Playing a game to complete an objective isn't the same as being a rat in a cage.

By your theory humans wouldn't mind being enslaved because you can train them to do what you want in order to receive food and rest and water and life.

Ramenhotep said...

Hi Kevin,

I am absolutely not suggesting that playing a game to complete an objective is the same as being a rat in a cage.

I am talking about an experience that I have had many times--the shifting of motivation away from intrinsic and towards extrinsic, and the difficulty of switching that motivation back. The phenomenon of overjustification is something observed routinely in humans.

I am hypothesizing that a pattern of behavior that we observe in rats with respect to rewards may be a solution to the overjustification problem. Pavlov initially did classical conditioning experiments with dogs, but those experiments are still relevant to human cognition and behavior: they simply have to do with the way our brains are wired when it comes to rewards. Similarly, I believe that Skinner Boxes can give us insight into how the human brain works.

Does the fact that are brains are not 100% under our cognitive control at all times make us slaves? Of course not. But as a game designer my job is to make an engaging experience for the player. In the context of a game, the overjustification effect is literally the process of disengaging from that game. So it's my job to do what I can to help players not experience overjustification, and continue to engage with a game if they want to do so.