Topic: Review Etiquette: What counts and what shouldn't?

[UncleBob]

So, with the recent "scandal" surrounding the DLC in Injustice, one of our resident reviewers suggested changing his review score.

Now, this conversation happened on Twitter, with about 8 different people in the chain of replies, so it didn't leave much room for discussion.  I thought it might be interesting to move it here and to make it a bit more broader.

The basic question comes down to: "Why are you reviewing this game?"

Do you want to share your opinions on how well (or how poorly) made the game is?
-or-
Do you want to influence the buying decisions of your readers?

These can sometimes lead to the same goal - but they can sometimes be very different.

For example, Injustice may be an awesome game - but it's quite apparent the Wii U version is... lacking.  Even without the DLC.

So, do you review the game based on the merits of the version you're reviewing?  Or, do you review it based on what else is out there on the market?  Even then, for fighting games, Wii U doesn't have much for it, so do you review it based on what's on other systems?  If I don't have a 360 or a PS3, Injustice might just be the best fighting game I can buy right now.  If I have a 360 or PS3, would it be a disservice to me, as the reader, if you failed to mention that "Oh, hey - this game is awesome - and you can get a better version of it on that other system."

More to the actual conversation - do you review the game based on promises made regarding future support?  If I'm reviewing Professor Layton, then surely the fact that they're bringing another 365 puzzles to the game as DLC should be worth mentioning, right?  Isn't "longevity" or "replayability" usually something measured in reviews - and doesn't DLC (paid or otherwise) play a large part in that?  Heck, if New Super Luigi U is given as a free download to anyone who owns New Super Mario Bros. U, isn't that worth going back and updating the old reviews with "Oh, hey - now this game has freaking twice as many levels."?

[Halbred]

Let's not forget that it says "DOWNLOADABLE CONTENT" on the back of the box. You're (I'm) reviewing a feature set, even if that feature set is (at the time) assumed. If we don't get Lobo on Thursday, we can pretty much discount that feature and know that WB GAMES ARE LIARS.

The game becomes less desirable AFTER THE FACT. How do you treat THAT in a review? I even brought up the DLC in my review because I was excited about it. We thought DLC was coming back then. So naive. The Twitter conversation was basically me being really goddamn angry. I remain angry, but I don't know if I'd actually change the score anymore. The 8.5 was based on the game I played, not the content I thought might come down the pipe later. Even so, I'd probably rewrite part of the review.

[ShyGuy]

I hope there is class action lawsuit for fraud.

[nickmitch]

I missed it. What happened?

[Oblivion]

If it says DLC on the back of the box, and that story is CONFIRMED for no DLC for Injustice on Wii U, then you probably have a right to sue the **** out of them.


However, DLC is still an optional experience and shouldn't effect the score of the core game. I feel like the only time you can change a score is if you play more of a certain aspect and it makes the game flow better, or if you revisit it and have a change of heart, etc.

[Spak-Spang]

I think that the game should be reviewed singularly as the product you can buy as one purchase.  If the content is compelling and fun it should get a high review on that merit. 

However, part of the review should be comparing the game to the ports since the game doesn't live in a bubble.  So in the opinion or other comments of the review mentioning another version has DLC or better frame rate or graphics is important for a gamer to make the best decision on which version of the game to buy. 

However, that should not affect the actual score of the stand alone product. 

[NWR_insanolord]

I think the game should be reviewed for what it is, but at the same time, in the case of a multi-platform release, I think it makes sense to let readers know when a particular version is missing features that the other ones include. Like it or not, our reviews are going to be used as aides to purchasing decisions, and I don't see the harm in including bits of information that help with that as long as most of the review is focused on what is there.

[Pixelated Pixies]

This DLC that is coming to other platforms but not to Wii U is chargeable right?
 
If so, it's my opinion that the score shouldn't change. You're still well within your rights to be angry about it, and to write and report about it, but the review should stand to assess the quality of the main product. Chargeable additional content which is optional should not in my opinion factor into a review of a game.

[Ian Sane]

In the case of a multi-platform review you need to take into account that some of the readers own multiple systems and may base their decision of which version to get on the review.  Therefore if one version is missing a feature the others have it should be mentioned in the review and should affect the overall score.

In the case of Injustice there is effectively a missing feature in the Wii U version but it was not known at the time the review was written.  I would consider an update to the review because of that.  Typically you wouldn't find out this sort of thing after the fact.  It's a rare scenario.

Regarding DLC, if the content is as substancial as actual new levels and such I would say it's worthy of its own review.  I wouldn't re-review NSMB U when the Luigi DLC comes out but rather just review the Luigi DLC and perhaps state in the review that you would recommend NSMB U even more now that it has extra content.  DLC is effectively an expansion pack and PC games have had those for a long time now.  Whatever approach was typical for such games then should be applied now.

[Ceric]

On the whole Review Etiquette thing I think Scoring to an individual still gets my goat.  I mean not even at a site level.  When you give something a 3 and admits its better then a game you gave a 5 to I find that the height of Hypocrisy.

[NWR_insanolord]

It's not really feasible to have one unified scoring scale for an entire site. A site is going to be staffed by a variety of different people, and they're going to have very different tastes. I know for instance Zach Miller and I would have strikingly different opinions on a lot of games. That's why Metacritic's approach is silly, ascribing taste to a site, when really it's a hodgepodge of individual tastes.

[Halbred]

J.P. is right. He gives a game a 10 if it has soccer in it; I would lower the score for merely having the word "soccer" appear in the instruction manual.

[Pixelated Pixies]

That's why Metacritic's approach is silly, ascribing taste to a site, when really it's a hodgepodge of individual tastes.

Metacritic is only as silly as the people who use it incorrectly. Metacritic does not 'ascribe' taste to a site, it only reflects the mean average and then presents those individual scores on a scale for people to reference (and hopefully read). The only thing about Metacritic that I disagree with is that there have instances where a score has been appointed when no numerical value was provided by the reviewer. That's just silly.

[NWR_insanolord]

And I think it's silly to modify the scores at all in their presentation to the user. If one person wrote every review for a site, adjusting for biases would make at least some level of sense. I appreciate the value of Metacritic as a place to go for easy access to a wide variety of reviews, but I take issue with it misrepresenting the content it's aggregating.

[Pixelated Pixies]

...but I take issue with it misrepresenting the content it's aggregating.

I'm still unclear. In what way does Metacritic misrepresent the content? Are you saying that the assigning of a numerical value misrepresents the review?

[Ian Sane]

I liked Metacritic when it was just some site to get an average score from and have an idea of what games were getting good reviews and were perhaps worth checking out (or avoiding if their was a trend of negative reviews).  The problem is when the average score is treated like gospel to the point that publishers will award bonuses based on the Metacritic score.  It's pretty cool as just something for fun but now it actually affects the lives of real people, which is probably not the original intention.  Basing bonuses on review scores also now introduces a major incentive for bribing reviewers.

[Ceric]

I think My point flew over the head of J.P. On that one at least being directly under my post it looks that way.

On the whole Review Etiquette thing I think Scoring to an individual still gets my goat.  I mean not even at a site level.  When you give something a 3 and admits its better then a game you gave a 5 to I find that the height of Hypocrisy.

So in JP Case if Pro Evolution Soccer 2013 3D comes out, was going to use Fifa but 9 is hard to be flexible with, and reading the review JP loves it vastly better than Pro Evolution Soccer 2012 3D but, goes on to give it a 5 that Gets me.  Now if any other reviewer does *shrug* flexible scoring system.

[NWR_insanolord]

...but I take issue with it misrepresenting the content it's aggregating.

I'm still unclear. In what way does Metacritic misrepresent the content? Are you saying that the assigning of a numerical value misrepresents the review?

I want Metacritic to show games as getting the score the site actually gave it, and not some bullshit adjusted score that's based on faulty reasoning.

[Pixelated Pixies]

I want Metacritic to show games as getting the score the site actually gave it, and not some bullshit adjusted score that's based on faulty reasoning.

I understand where you're coming from. I do find it a little disconcerting when scores are seemingly pulled from no where. On the other hand, if the Metacritic system didn't equate a 4/5 with a 80/100 then they wouldn't be able to compare scores, and that's kind of the whole point of the site.
 
The problems that I have with Metacritic, however, are more in how it's being used as a resource, and not necessarily how that information is being surfaced. Firstly, I think people who refer to Metacritic scores without actually reading at least some selection of the reviews aren't getting an accurate picture (and that responsibility lies with them). Secondly, as Ian said a few posts back, developers and publishers are aware that certain people do put stock in the 'Metacritic score' alone, and they will often, therefore, use that score as a barometer of success (which can have repercussions in terms of bonuses etc). In both cases though I feel the stupidity lies with the users, be they individuals or companies.