Topic: EDITORIALS: On Ratings

[Maverick]

Quote
Originally posted by: Professional 666
7.0 - Dragon Blade

7.5 - Zelda PH

GOTTA LOVE THAT 0.5 INCREMENT

Haha, it also seemed that some of the sentences in this editorial were referring to the "backlash" of the Phantom Hourglass review.

[NinGurl69 *huggles]

The fight rages on.

[BigJim]

Yep,  IGN caters to the enthusiast crowd, and perhaps their reviews are fine for that group of people.

That being said, I think if they want to also review market-expansion games, non-games, etc, they should either adjust their scales accordingly to those titles, or create an off-shoot brand that is geared towards the mainstream where games can be properly represented and content is more accessible to the other 90%.

At the VERY least, since we are on the internets where anything is possible, a very short summary review can be made available on the first page, with the "enthusiast" version on the following page(s), if they care to read them... rather than the summary getting stuck in at the end, which seems entirely backwards to me.

I have to reference and give props to GamerDad.com. They're doing a relatively good job, and I would like to see more of their type of reviews rather than watching sites compete to be more and more elite.

Edit Note: Not all IGN reviews are long, extensive, cluster-F's of "look how smart I am" techno-babble, they're just an example. I've seen much worse self-indulging reviewers at other sources.

[that Baby guy]

I have barely read anything here, but I'm already decided.  Numbers mean too much in too many other ways to work for reviews, mostly because of school.  The numbers you receive in school are based on accuracy.  A long time ago, people figured out that it's better to be more than half-accurate for passing marks.  Would you trust a doctor, teacher, surgeon, news anchor, or anything else that only was right half of the time?  Of course not.

But here's the thing.  You can't judge the accuracy of a game, a movie, a book, or any kind of art.  That concept is ridiculous.  An "art" can't be accurate.  It can't be right or wrong.  It can be tasteful, entertaining, captivating, innovative, and a number of other things.

The number-based score is invalid when it comes anything apart from accuracy.  It only works for quantitative measurement.  A game can't be 70% accurate.  The only way this would work is if the reviews all had a solid, concrete equivalent to compare.  For example, if you were to rate out of 100, a score of 85 would mean that there are 84 games out of every hundred that aren't as well done as that one, and 15 that are better.  Now, obviously, since average scores tend to be in the 70's and 80's, this isn't true.  So it's time to scrap the system.

Did you know that NWR has a few phrases that define their ratings?  Why doesn't NWR just scrap the numbers, and substitute in those phrases, with perhaps a little extra changed for each number?  It would give meaning, and would really show that NWR rates based on what they feel is right, not based on what each individual deems the correct score for a type of game.  It would give reason where before there were just numbers.

Check out this: http://www.nintendoworldreport.com/policy.cfm for more information.  You'll see that NWR really doesn't rate with numbers at all, but they do it to match industry trend.  I say that NWR should forget trend, and instead, go the next step, and forgo numbers when it comes to the overall rating.  It might be worth it to use numbers in the smaller categories for a little while, but hopefully, in time, you can encourage the entire readership to take time to understand why a game is worth their time, or, perhaps, why it isn't.

Anyways, I'm sure Evan's right, but I beat him to eat over in General Gaming about a month ago.  I just am not such an eloquent writer...  

[mantidor]

Numbers and reviews have become meaningless for me, with some scarce exceptions. My best review is forums in general, I see the buzz of a game, I kind of know the people behind the buzz and I kind of know their tastes, so I can measure fairly accurate if I would enjoy the game or not based on their comments, of course is an enormously complex system that I doubt is of any of use except for me and other handful of people.

[GoldenPhoenix]

I support ratings, personally I don't think there is much you can do to get around it. Most people have grown accustomed to it and although review scores can differ from time time, usually they are fairly close to each other across the board giving some sense of what the game is like. Now could alot of that sense be more fictional than real? Perhaps but I know I feel much better with a review that has a review score than one that doesn't. In fact I go out of my way to read review scores that may differ from the pack in order to see why that particular reviewer gave it a low or high score. Also I think scores can also be helpful if you know the reviewer and can better determine what the score means quality wise and how much emphasis they put on certain things.

[Jonnyboy117]

Quote
Originally posted by: IceCold
If reviews are numerical, I think there should always be a "Tilt Factor" or "X-Factor" similar to what GameSpot has, except they implement it quite poorly.

I complained about GameSpot's review system for years, but just recently they overhauled it to eliminate all subcategory scores.  Also, the overall score (now the only score) is only given in .5 point increments, so there won't be any more 6.8s or 7.9s.  It's a much, much better system than what they had before.

[Jonnyboy117]

Many of you have probably heard that I did not use scores in my reviews at the Nformant, and when PGC started to discuss what would become our review format (in early 2001, before we imported the first GBA games), I was very vocal in suggesting that we avoid scores altogether.  However, I was a lowly Staff Writer at the time, and although my opinion was registered and respected, other viewpoints prevailed.  I think most of my superiors at that time agreed that "it would be nice" to omit scores, but they felt the practical benefits were more important.

It's funny, because now I am Reviews Editor, and I still believe review scores are a bad idea.  But I'm not sure that I would want to even try to get rid of them now, because they are part of the institution of NWR.  It would even be technically difficult to create a new type of review article without the scores and have the new reviews indexed along with the older ones.  So I don't know if we could ever eschew scores now that we are standing so deep in them.  But certainly, I agree completely with the spirit of Evan's editorial.  If I had a new website and could start from scratch, I would definitely review games without scores.  That's not going to happen, though.

[ShyGuy]

I think an ice cream flavor rating system would be best. Phantom Hourglass is rated Orange Sherbert Swirl and Dragon Blade is rated Mocha Chocolate Chip. Is Orange Sherbert Swirl better than Mocha Chocolate Chip? I KNOW I PREFER THAT FLAVOR OF ICECREAM

[shammack]

Quote
Originally posted by: ShyGuy
I think an ice cream flavor rating system would be best. Phantom Hourglass is rated Orange Sherbert Swirl and Dragon Blade is rated Mocha Chocolate Chip. Is Orange Sherbert Swirl better than Mocha Chocolate Chip? I KNOW I PREFER THAT FLAVOR OF ICECREAM

This is a fantastic idea.

[k_bukie]

Breaking down art objectively is almost impossible.  Breaking down technical aspects of a game objectively is easy.

I'd almost suggest going to a "Recommended for everyone", "Recommended for fans", and "Not Recommended" review for all games, but even the VC games get criticized and compared for that simple system (lol Kid Icarus).

Maybe the best thing would be listing the pros and cons of a game, and just let the reader decide for him/herself whether it's worth it.

[GoldenPhoenix]

Quote
Originally posted by: k_bukie
Breaking down art objectively is almost impossible.  Breaking down technical aspects of a game objectively is easy.

I'd almost suggest going to a "Recommended for everyone", "Recommended for fans", and "Not Recommended" review for all games, but even the VC games get criticized and compared for that simple system (lol Kid Icarus).

Maybe the best thing would be listing the pros and cons of a game, and just let the reader decide for him/herself whether it's worth it.

Wait is this topic going to turn into "Is art subjective?".

[Djunknown]

I weep for people who just look at the rating of a review without reading it...

Quote
Do journalists ever give 10s to games outside of established franchises?

They have. Sometimes they come out of left field. Soul Edge was just some 3d fighter, it was playing 3rd fiddle to Virtua Fighter and Tekken. But when Soul Calibur came on Dreamcast, one could say that game was so good, it almost single-handeldy killed the arcade.

Quote
Reviewing a game's graphics, sound, or control too is a nonsensical idea: does a high polygon count within the framework of realism look better than a simple and striking fantasy design?

One of the underlying assumptions we make for console games are: is it using the hardware's power effectively? I don't want to speak for everyone collectively, but Wii games that visually look PS2's is insulting to me. While the Wii may lag behind the 360/PS3, in my mind there's no excuse for last gen graphics.  The visuals need to match the vision as close as possible.

Games like Wind Waker hit it right the nose, despite the misgivings when it was first revealed. Karl's (and one of mine) favorite game, Killer 7 does this as well. Metroid Prime 3, to use a more recent example, should ideally be the rule instead of the exception. When done right technically, we can further discuss the artistic merit. But if [realistic sports game 2KXX] misses the realism mark by a wide margin, than that needs to be addressed.

One aspect that just about all developers got right is sound. Sound at a technical level thankfully isn't an issue anymore. Instead the reviewer can discuss what they think of the sound effects, the scoring, the voice overs when its applicable.

Control should always be taken into account. Ultimately, all video and computer games live and die by the interface. This is what separates itself from the rest of popular media. You interface and interact with it. You can have the greatest production values in the world, but if you can't interface with properly, what good is it for? Lair is a prime example. While Suda 51 and Hideo Kojima are considered gaming's mad scientists/oddballs if you will, they wouldn't in their right mind ship a game with purposefully broken controls.

Quote
Furthermore, other forms of media also pretty much only get ONE overall rating. Movies don't get separate ratings for acting, special effects, directing, soundtrack, stage sets, etc. That would just be silly.

But they do get awards for each individual category  That's an interesting way of looking at it. I don't think Ebert and Roeper nitpick movies like that every single time.

[that Baby guy]

Quote
Originally posted by: Jonnyboy117
Many of you have probably heard that I did not use scores in my reviews at the Nformant, and when PGC started to discuss what would become our review format (in early 2001, before we imported the first GBA games), I was very vocal in suggesting that we avoid scores altogether.  However, I was a lowly Staff Writer at the time, and although my opinion was registered and respected, other viewpoints prevailed.  I think most of my superiors at that time agreed that "it would be nice" to omit scores, but they felt the practical benefits were more important.

It's funny, because now I am Reviews Editor, and I still believe review scores are a bad idea.  But I'm not sure that I would want to even try to get rid of them now, because they are part of the institution of NWR.  It would even be technically difficult to create a new type of review article without the scores and have the new reviews indexed along with the older ones.  So I don't know if we could ever eschew scores now that we are standing so deep in them.  But certainly, I agree completely with the spirit of Evan's editorial.  If I had a new website and could start from scratch, I would definitely review games without scores.  That's not going to happen, though.

Well, what about putting a standard phrase with each score.  For a 9.5 or 10, there would be this phrase, or one similar next to the score: "We think this title is a nearly flawless game, fun, and innovative for all.  Do not miss out on this great experience!"

For an 8 or a 7.5, it could be, "We believe this title has substance, but not everything is of the highest quality.  If you have interest in the genre or series, this game is probably for you.

For a 5, which NWR defines as the borderline, the phrase could be, "This game isn't the best or even close, but if you absolutely enjoy the genre, you may want to consider spending some time with it."

And so on.  That way, there would still be numbers, but you'd have guidelines so people would understand what they're looking at, so hopefully, they won't overreact quite so much.

[GoldenPhoenix]

At least NWR's rating for PH was dead on. That is my 2 cents

[UncleBob]

Quote
Originally posted by: Maverick
Haha, it also seemed that some of the sentences in this editorial were referring to the "backlash" of the Phantom Hourglass review.

To be fair though, for some of us, the problem wasn't with the "rating", it was with how the review itself was written.

[wulffman04]

All ratings, are in a sense, flawed. No one person can take a game(or anything for that matter)  for what it is and not put in his own opinions about it. Then again, are ratings just opinions of experienced people?

-I can hear Evan holding back his hand as he desires to reply to these messages.

[Kairon]

Why not just add NWR's "Recommended for Everyone," "Recommended for Fans" and "Not Recommended"  as an additional label for each game? It's not scarily new, it's just applying the same thinking of the VC's impressions to the game reviews. Even keep in the point score if you like, just prominently display the "label" at the top of the article, away from the score.

[Jonnyboy117]

Quote
Originally posted by: thatguy
Well, what about putting a standard phrase with each score.  For a 9.5 or 10, there would be this phrase, or one similar next to the score: "We think this title is a nearly flawless game, fun, and innovative for all.  Do not miss out on this great experience!"

For an 8 or a 7.5, it could be, "We believe this title has substance, but not everything is of the highest quality.  If you have interest in the genre or series, this game is probably for you.

For a 5, which NWR defines as the borderline, the phrase could be, "This game isn't the best or even close, but if you absolutely enjoy the genre, you may want to consider spending some time with it."

And so on.  That way, there would still be numbers, but you'd have guidelines so people would understand what they're looking at, so hopefully, they won't overreact quite so much.

I have intentionally avoided score guidelines like the plague.  We don't even have internal staff definitions for the numbers.  I think reviewers will be more accurate to themselves if they are forced to choose a number based on instinct rather than an established rubric.  And I see no proof that providing a clearer definition of numbers to the reader would lead to less controversy or overreaction -- look at how much hand-wringing takes place over most reviews at GameSpot, IGN, 1up, and other places where the numbers are pretty well defined.

Quote
Originally posted by: Kairon
Why not just add NWR's "Recommended for Everyone," "Recommended for Fans" and "Not Recommended" as an additional label for each game? It's not scarily new, it's just applying the same thinking of the VC's impressions to the game reviews. Even keep in the point score if you like, just prominently display the "label" at the top of the article, away from the score.

One of the problems with this proposal is that "Recommended for Fans" has never been clearly defined, and it's going to be ambiguous no matter what.  Also, adding this kind of system would not really help matters in the context of Evan's editorial, because we are still treating reviews like recommendations.  I do believe reviews should include recommendations in addition to actual criticism, but I fear that institutionalizing it in this way would make the recommendation seem overly important to both the reader and the reviewer.  In other words, it might seem clearer than numerical scores to some people, but it will make things more confusing for others, and it doesn't solve the root problem either way.

[that Baby guy]

Quote
look at how much hand-wringing takes place over most reviews at GameSpot, IGN, 1up, and other places where the numbers are pretty well defined.

Really?  NWR does a much better job defining their ratings numbers than any of those places.  I can rarely match up the written reviews with the scores in any logical fashion at those sites.  That's something I love about NWR, that the number always matched the written review.  At the above sites, I feel like everything is 7.something or 8.something, just because.  If it isn't in that range, it's a major license or a highly publicized game.

[TheYoungerPlumber]

At one point we tried to put better definitions next to numbers, but it's just too difficult. For example, I think most on staff see 7 or 8 as good with some serious setbacks...but why a 7 instead of an 8?  That's really up to the reviewer.

[GoldenPhoenix]

I think at IGN Wii the ratings are being a bit more defined.

[that Baby guy]

You can have a quip used for several different numbers, especially if the staff agrees there's not much difference the two scores, but that right there tells you a reason why numbers alone are no good.  Why a 7 instead of an 8?  If the staff can't answer that question, then why should they be using the system that makes them answer it?

[TheYoungerPlumber]

Soooo. I think the answer there is that individual reviewers have their own ideas of 7 vs 8.  I didn't say there was no difference.  It's just that my 7 may be Karl's 8.  For me an 8 is a game that is fun to play and is probably worth buying, but has some notable shortcomings. A 7 for me has more critical shortcomings that muck up what aspects of the game are fun, making it at least a rent-before-you-buy.

[IceCold]

Quote
A 7 for me has more critical shortcomings that muck up what aspects of the game are fun, making it at least a rent-before-you-buy.

That's basically the industry standard, but NWR has been giving games like you just described 5s instead of 7s recently.