[Long] Why your feedback is important, but your reasoning isn't

This thread is about expert feedback versus layman feedback. But first, I'm going to tell the story of my buying PoE.

No, not this game, but Pillars of Eternity, a much-hyped Kickstarter cRPG. So I played it for a while on its hardest difficulty, rerolled several times, and then just stopped playing it.

The stopping part was a little confusing to me. Why did I do that? I was pretty darn hyped about the game. I thought the story was pretty good. But I guess I had burned myself out on rerolls.

As I often do with games which hype me but I'm not currently playing, I hopped on the corresponding forums. After a few days of posting, I eventually came across a thread lining to a review of the game. A very negative review.

It was a big moment for me, because reading Roxor at RPG Codex trash the game solved my mystery. I now understood why I wasn't playing Pillars anymore.

In short, the gameplay sucks. Bad.

Does this mean that I'm some kind of weak-willed dittohead who only agrees with what a talking head has to say? I certainly don't think so. I think it has a lot more to do with vocabulary and the ability to communicate our ideas. Are the dialogue options in Pillars often Biowarian? Yes, they are. But I would have never, ever used Biowarian as a word on my own, while the reviewer did. So it never really impacted my own, internal review of the game. I simply never put it into words.

This isn't a new or unusual circumstance. For years, researchers on individual taste have known that laymen have several barriers to giving good feedback. These barriers, in turn, have had huge impacts on things like focus group testing, leading to some of the biggest business disasters in history.

1. Short and Sweet

Perhaps the most famous of these disasters was New Coke, the 1985 attempt to change the Coca-Cola formula. Why? Fear.

See, the Coke folks had conducted lots of blind sip tests to mirror the "Pepsi Challenge" campaign which their competitor had put in TV advertisements at the time. Two cups, no labels, and Pepsi was winning, 57% of the time.

Coke took this as evidence that Pepsi had the superior formula. So they work on a new cola formula which out-Pepsis Pepsi at the Pepsi Challenge. This is a heavily tested product with thousands of taste tests performed before rerelease, and all the tests indicated a winner.

As we know, it failed horribly.

Why? The core reason was the design of the Pepsi Challenge itself. Coke bought the line sold by the advertisement - that a sip test is a valid means of estimating the actual marketplace behavior of customers.

As it turns out, sip tests are a horrible indicator. Without a significant amount to test over a significant period of time, sweet beverages always win... and Pepsi was the sweeter cola. Pepsi had devised a test in which Pepsi would perform much, much better than it would in the actual market, and then tried to spin those test results to the public.

Not every test we come up with, even if designed with the best intentions, is always going to give accurate results of customer satisfaction. For example, with all of the balance changes being made in the current Closed Beta, are testers rerolling more and running maps less? If so, should GGG adjust things so testers run more maps and get more data on the new mapping experience? The danger here is that, like New Coke, the focus might be too much on the sip of pre-maps play, favoring a formula which is sweet early on but poorly equipped for the long haul.

2. Roasted in the Dark

So far, closed beta invites have mostly followed a random system. For the record, I think this is absolutely the correct way to go when it comes to raw, empyrical data, such as how much time is spent in which zones or when players choose to either stop a party session of continue playing past their bedtime. (Getting normal sleeping hours from each beta participant, via webform, should be a priority).

Unfortunately, it is very, very difficult to trust what random users have to say on the forums.

For example, coffee companies are well aware of a problem with focus groups. If you ask people what kind of coffee they prefer, the majority are going to say they like a rich, hearty roast. And for 25-27% of them, this is actually true. But the vast majority prefer weak, milky coffee; they just don't want to say it. The coffee companies know this, because they know what customers actually buy in the supermarkets, when no ears are there to judge.

Which means: when it comes to coffee, asking people what they want and giving it to them is actually a disaster.

Perhaps nowhere is this more true than in the echo chamber that is the contemporary internet forum. In such an environment, which ideas you (dis)favor can have a direct and immediate impact on how others respond to you. You can find users championing ideas, not using their own reasons as evidence for an argument, but instead pointing to the raw popularity of an idea, as if the popular opinion is always the correct one (rich, hearty roast for everyone). And so on.

This is particularly evident in ARPGs when it comes to changes to established builds or mechanics. Players with currently overpowered builds overwhelmingly are seen opposing nerfs. But is it actually true that playing a challengeless, roflstomping character is more enjoyable than a more balanced one? I am inclined to believe that nerfed builds are often (not always) more fun to play post-nerf than before, but rather than helping to guide the designers towards a reasonable nerf that leads to more enjoyable gameplay afterwards, players usually defend their unbalanced characters vigorously... and we can only wonder how many unfun overnerfs could have been prevented if this players had provided more honest feedback in the first place.

3. Jammed communications

However, selfishness or fear of embarrassment are not the core reason why people give bad feedback on their coffee. Perhaps a deeper reason is that they aren't comfortable with the words.

In a 1991 paper, Wilson and Schooler, two psychology researchers, asked random college students who volunteered to evaluate a selection of jams - that is, fruit preserves. The jams were selected from a list in a 1985 issue of Consumer Reports, wherein a panel of experts had already rated them.

With the first group in the study, they asked the volunteers to simply rank the jams on a 1-10 scale. The result was that the experts at Consumer Reports were pretty close - it wasn't exact, but the correlation in ranking was strong.

The second group was the interesting one. This time, they asked participants to write down reasons for their rankings on paper, then the ratings would be collected while the reasons would be thrown away. The result was that the second group was wildly off from the first group (and expert's) consensus.

So the process of trying to reason out one's opinions, in this case, actually leads to warped responses. Thinking about the preference corrupts the preference.

Is this true for everyone? Of course not. The experts for Consumer Reports were able to describe the reasoning behind their rankings. And they're writing about jams, of all things.

The difference is vocabulary. If you ask an expert to explain his opinion, it isn't a problem, because he has the tools to communicate them. But if you ask a layman, they don't have the tools to describe differences in texture, or particular tastes. All they have is a gut reaction, and if you force them to explain it, what they end up with will more often than not be a betrayal of that gut feel, not an affirmation of it.

And this is why I, like so many gamers, profit from having reviewers out there. If you would have asked me about Pillars of Eternity, really forced me to explain, before reading the Codex review... I really don't know what I would have said. I probably would have just made something up, halfassed. Poor feedback.

4. Conclusions
A. You need to test the right things. Beta testers should absolutely have the ability to skip to Merciless, so interested testers could get as much mapping data as possible.
2. Peer pressure is a feedback killer. A good community for feedback embraces diversity and makes those with uncommon opinions feel at home.
3. Players are a good source of knowing if something about a game is broken or not, but an atrocious choice of providing reasons why something is broken or not... and if you push them for reasons, they might tell you broken things aren't broken.
4. Asking a forumer to explain his feedback is actually counterproductive to good feedback, unless that forumer is an expert.
5. GGG should care more about getting beta impressions from gaming journalists than from streamers.

This post was heavily inspired by speeches by Malcolm Gladwell.

So...is the EB thread doomed?

(in an additional point: maps are accessible maybe to the 1% of the playerbase; how much feedback GGG needs from that part? Shouldn't they focus first in the leveling phase before working properly on the endgame?)

Good read.
Unfortunately, those who really need to read this, wont.

Ha, Pillars gameplay sucking? Stopped reading right there, you usually have good posts, but this was probably the most stupid thing you ever posted.

Pillars is one of the best games to be made in last couple of years. Everything in it is superb, even gameplay.

Also regarding Coke, many believe Coca Cola Co. did it on purpose so they could switch from using sugar canes to fructose suplement without people really noticing. It's a tinfoil theory but they did switch from using sugar canes.

Edit: ok now I read all and the whole post is not good. Sorry, but it's not. Thinking that game journalists would be better reviewer than streamers, in todays age, when we KNOW half of the journalists are paid to write what the owner/director wants or they just don't have enough time to properly play and rest the game - you really think they will give better feedback than someone who plays 10h per day and has been doing so for last couple of years?

Your regular journalist would go through Normal and reviewed the game without ever going into Merci.

"

A. You need to test the right things. Beta testers should absolutely have the ability to skip to Merciless, so interested testers could get as much mapping data as possible.

Why just mapping data.
From a dev perspective: knowing too much about the game can lead to bad results in term of financing and support.
About crowfunding:
"The second process is pluralistic ignorance (or social influence) in which people tend to rely on the overt reactions of others when interpreting an ambiguous situation. In this case, individuals look for cues in the environment that can help them determine whether action is necessary. As noted by Cialdini (2001: 100),
we “view other behavior as correct in a given situation to the degree that we see others performing it.” Astrong bystander effect occurs when no one helps because everyone believes that no one else perceives anemergency"

If a lot of people knew about the situation of mapping or solo vs multiplayer advantage in term of mapping, there would be a lower amount of players willing to pay for 1000$ supporter packs. Typical supporters who never have the time to play the game or to nolife it.

"

3. Players are a good source of knowing if something about a game is broken or not, but an atrocious choice of providing reasons why something is broken or not... and if you push them for reasons, they might tell you broken things aren't broken.

One of the things that are bad imo but when i am asked for reasons i can't tell, are:
a. Designing a unique purchases should not last more than 4 months to be fullfilled. It still amazes be to know that a Diamond supporter could stay 1 year or 2 without seeing the result of his work.
b. Reserved

"

5. GGG should care more about getting beta impressions from gaming journalists than from streamers.

It's interesting to see how Kripparian did not fall for this thing yet. He knows that 2 days impressions are not 100% honest feedback.

Just to clarify, if the closed beta were 5 months or some other awfully long duration, I would not consider auto-advancing testers to maps. It's the relatively short duration involved.

"

Odoakar wrote:

Your regular journalist would go through Normal and reviewed the game without ever going into Merci.

By "average" here, you apparently mean "trash." Game reviwers who don't investigate endgame/postgame content probably won't be game reviwers a few years from now. I'd hope GGG would be more interested in the opinions of competent reviwers than incompetent ones.

Oh, and they are out there.

"

Odoakar wrote:

Ha, Pillars gameplay sucking? Stopped reading right there, you usually have good posts, but this was probably the most stupid thing you ever posted.

Pillars is one of the best games to be made in last couple of years. Everything in it is superb, even gameplay.

Also regarding Coke, many believe Coca Cola Co. did it on purpose so they could switch from using sugar canes to fructose suplement without people really noticing. It's a tinfoil theory but they did switch from using sugar canes.

Edit: ok now I read all and the whole post is not good. Sorry, but it's not. Thinking that game journalists would be better reviewer than streamers, in todays age, when we KNOW half of the journalists are paid to write what the owner/director wants or they just don't have enough time to properly play and rest the game - you really think they will give better feedback than someone who plays 10h per day and has been doing so for last couple of years?

Your regular journalist would go through Normal and reviewed the game without ever going into Merci.

The whole point is that a gaming journalist has the tools to communicate the reasoning behind the feedback, its not about the worth or value of the feedback, its about the worth or value of this reasoning. The raw feedback is valuable regardless.

The developer has to glean and interpret, doing the reasoning work/guess work on behalf of the streamer or player and then choose what to do with it, rather than being able to take what the journalist says literally and trust the reasoning behind it and make a more direct correlation to the correct course of action.

I disagree however, as there plenty of disciplines that can provide the same/sufficient tools of communication. In general you cant generalize, and have to take it on a case by case basis assessing each piece of feedback and whether the quality of communication warrants trusting/valuing the reasoning behind it. But that is far less efficient of course than choosing before hand what sources of feedback qualify.

To further explain my point of view:


1) Auto-advancing testers to maps = auto-advancing potential big supporters to maps = auto-advancing players who might never reach that part of the game otherwise = Casual potential big supporters know for sure that they don't like the mapping system or any other random aspect of the game(Trading?) = preventing social influence by famous respectful nolifers praising the mapping system = less money.
2) Devs know better anyway.


I still believe in the Beta phase ability to lead the game into great improvements/features.

"

I still believe in the Beta phase ability to lead the game into great improvements/features.

Oh, as do I. I don't mean to say that problems with interpreting feedback mean feedback doesn't matter. Actually, quite the opposite; I doubt PoE could survive much longer without something similar to this beta where they hunker down and work on big changes.