Author Topic: EDITORIALS: On Ratings  (Read 28119 times)

0 Members and 1 Guest are viewing this topic.

Offline Jonnyboy117

  • Associate Editor
  • NWR Staff
  • Score: 37
    • View Profile
    • Nintendo World Report
RE:EDITORIALS: On Ratings
« Reply #50 on: October 17, 2007, 07:14:18 PM »
Quote

Originally posted by: thatguy
Quote

look at how much hand-wringing takes place over most reviews at GameSpot, IGN, 1up, and other places where the numbers are pretty well defined.


Really?  NWR does a much better job defining their ratings numbers than any of those places.  I can rarely match up the written reviews with the scores in any logical fashion at those sites.  That's something I love about NWR, that the number always matched the written review.  At the above sites, I feel like everything is 7.something or 8.something, just because.  If it isn't in that range, it's a major license or a highly publicized game.


I will take this as a major compliment -- the thing I stress most is that the numbers should match the review.  But the miraculous feat (in my opinion) is that we can do this without actually defining the scale at all!  I think what you're saying is that the scores are defined on a per-review basis by the reviewer, in that he or she is writing the entire article to define the number score (or better yet, choosing the number that most intuitively matches the article).  That is totally different than putting a chart on the site so everyone can see "5.0 = Average, 8.0 = Great, etc."
THE LAMB IS WATCHING!

Offline that Baby guy

  • He's a real Ei-Ei-Poo!
  • Score: 379
    • View Profile
RE: EDITORIALS: On Ratings
« Reply #51 on: October 17, 2007, 07:31:32 PM »
My biggest problem is that NWR is the only site that I've found that's actually capable of doing so, though, so it creates a horrible unbalance to shuffle between the mushy corporate reviews and the cleaner-cut NWR ones.  For everywhere else, every game seems to earn about a 7.5, but NWR would hypothetically adequately give this game something around a 5.0, which actually makes sense.  It's not really NWR I have the problem with, it's everyone else.  But then, because everyone else is flawed, it creates an invalid perception in the industry, and that's why people act like a score around 7 is a terrible thing for a game like Wind Waker, when it's really praise with a slight warning, IMO.

As far as those phrases go, perhaps each rater could summarize their recommendation/feeling about the game in every review in one sentence, even though the reviews usually finish with this, anyways, repeating the statement by the numerical score could encourage people to actually read the reviews before they react.

I think what I'm trying to say is that the way other sources rate games has skewed the population's perceptions of these numbers.  However, NWR seems to be more fair, sensible, and even when it comes to ratings, and manages to use the full spectrum quite handily, unlike said other sources.  Because of the population's ignorance and NWR's steady scales, misconceptions and misjudgments are typically made, usually because most people expect to see padded scores.  I think just a brief sentence written by the reviewer could quickly alleviate some of the concerns readers have in this manner.  Does that make a little more sense?

Offline Kairon

  • T_T
  • NWR Staff Pro
  • Score: 48
    • View Profile
RE: EDITORIALS: On Ratings
« Reply #52 on: October 17, 2007, 07:41:41 PM »
If the entire world was crazy, and you were the only person who wasn't somehow off the deep end, wouldn't that make you aberrantly normal, and clinically insane?

Or would you simply be the only one who doesn't see the emperor's new clothes?

But what about the adage that if you can keep your head while everyone else is losing theirs, there's something that you're missing?
Carmine Red, Associate Editor

A glooming peace this morning with it brings;
The sun, for sorrow, will not show his head:
Go hence, to have more talk of these sad things;
Some shall be pardon'd, and some punished:
For never was a story of more woe
Than this of Sega and her Mashiro.

Offline KDR_11k

  • boring person
  • Score: 28
    • View Profile
RE: EDITORIALS: On Ratings
« Reply #53 on: October 17, 2007, 11:22:52 PM »
I think discussing "aesthetic experience" is kinda missing the mark. Treat games like art all you want but people treat them as entertainment, something they use when they're bored and they expect it to be FUN. This is the goal metric, how much fun does the game provide. I don't buy games for their creativity, their graphics or their game mechanics, I buy them for the fun that results out of the combination of all of these aspects. If aesthetic experience is what gives you enjoyment then by all means base your rating on it! However, don't pretend we have to be snobby about some external value like "artistic merit" and rate that over enjoyment. If you didn't enjoy it, say so, don't pretend that the art must be experienced despite being unpleasant to play.

I do agree with the idea that ratings are flawed but I disagree that the cause is number inflation or whatever. I simply think we need more ratings for a single game, divided as seen appropriate for the specific game. I've seen reviews pan a game for being similar to the previous one and give a worse rating for that but does that help me when I've never played the first and want to decide which one to buy? Really, you need a rating for every "demographic" that you think has to be distinguished. Take Wii Sports. How do you rate that? Party or casual play: Awesome, Conventional play: Uninteresting and short. Guilty Gear? Advanced players of the genre: Awesome, newbies or people not used to the genre: stay away. More Brain Training: Newcomers: Buy this!, Owners of the previous one: may not be worth the money. Add as many ratings as considered appropriate but more than 2-3 usually won't make sense. Trying to give one rating valid for all people is futile since the values will diverge and you can only average them by considering their relative numbers.

Abolishing ratings and telling people to read the text makes it hard to get a quick glance of whether this game is worth considering (say you have a pool of ten and want to narrow it down to 2-3, reading all the reviews will waste a lot of time) and it also compensates for the reviewers potential inability to express the exact quality of a game in writing. The shortcomings and highpoints listed in a review all have a weight but that weight is often insufficiently expressed, a game can have 100 bad points and only 10 good ones but still be awesome to play. The final rating shows how the reviewer weights the advantages and disadvantages together and considers the overall experience. It may be a crutch but would you tell the man with two broken legs to drop his crutches? Reviewers aren't necessarily capable of bringing the quality of the game across in a wall of text, they may need a number to do so.

In closing, I rate this article -1. It misses the point and demands changes that benefit noone except game developers who suck at creating games and rely on the crutch of claiming it has artistic value not seen in the gameplay. :P

EDIT: It must apparently be mentioned that with "fun" I don't mean "colorful", "cartoony" or anything like that, I mean a game that provides an experience you like when you play it. Some people apparently think a game can be good without being fun by being scary, sad or whatever else emotion they can think of, in my vocabulary fun, enjoyment, etc describe how positive you feel towards playing a game. Horror is fun but that doesn't mean you sit there laughing, just that you think "that was good" after it's over.

Offline shammack

  • "This space intentionally left blank."
  • Score: 1
    • View Profile
RE:EDITORIALS: On Ratings
« Reply #54 on: October 18, 2007, 03:21:09 AM »
Quote

Originally posted by: KDR_11k
Reviewers aren't necessarily capable of bringing the quality of the game across in a wall of text, they may need a number to do so.


Just wanted to highlight the absurdity of this statement.

I don't have much sympathy for people who are too lazy to read a review and are only willing to look at a number when deciding what game to buy.  If that's the only factor going into their decision, it's pretty much a crapshoot because the numbers are so arbitrary and meaningless anyway.  If they buy a game and it's not what they expected from the 8.7 review (or whatever), they will whine.  If they didn't have the 8.7, they would have had to read the review and find out what to expect from the game, and maybe put some actual thought into their decision.  I don't see how creating a bunch of sub-ratings is any easier to understand than saying, "The multiplayer is a lot of fun at parties, but fans of traditional adventure games may be disappointed," or something like that.

Every game has different things that it does well or badly, and the only effective way to convey those things to the reader is to talk about them.  You just can't get that information across in shorthand.  Yes, you could write a review with all that stuff in it and still slap a rating on the end, but when you do that, that rating becomes the focus, and people ignore everything else and say, "This game got a 7.6 and the other game only got a 7.4!  I'd better go with this game!"  That should not happen.

Offline Nick DiMola

  • Staff Alumnus
  • Score: 20
    • View Profile
    • PixlBit
RE: EDITORIALS: On Ratings
« Reply #55 on: October 18, 2007, 04:03:57 AM »
Well there are alot of comments here already but  I figured I'd throw in my 2 cents. The review system should be as follows:

Rating
- Recommended for Everyone
- Recommended for Fans
- Not Recommended

List of Pros & Cons

Review Text

Multiple reviewers should at least provide their rating and list of pros & cons. If multiple reviewers can, they should provide a body of text.


With a system like this you put the most important points forward from a variety of people, and readers can distinguish some sort of value based on what each reviewer felt the pros and cons were. Also if a breakdown is given of each subcategory (ie Graphics, Lastability, Presentation, Controls), Graphics should now be "Art Direction." Chances are, if the Art Direction is crap the graphics will be too, but vice-versa it won't necessarily be true (ie Cubivore, Katamari Damacy, etc).

Otherwise, great write up Evan, I think you touched on some very good points.
Check out PixlBit!

Offline Smash_Brother

  • Let me show you my poké-balls
  • Score: 3
    • View Profile
RE:EDITORIALS: On Ratings
« Reply #56 on: October 18, 2007, 05:46:53 AM »
The key problem I see with reviews aiming to describe a game is that they seldom describe what the game offers an individual gamer and instead explains what it offered the reviewer. The problem is that I have no idea if the reviewer and I see eye to eye, or if the reviewer has been influenced by outside sources or not.

I had an idea a while back which could break down a review to the point where a reviewer would basically need to describe what the game offered in a much more granular scenario...

Five Scores, The Hows, Comparisons and Reviewer Notes

Five Scores:

Casual Gamer: The game's basic ability to involve gamers of average skill and entertain them.

Hardcore Gamer: The game's ability to force players to push hard to accomplish difficult but satisfying goals in the game.

Non-Gamer: The game's ability to be played by those who do not play games.

Party Gamer: The game's ability to involve and entertain a group of people.

Fan Gamer: The game's ability to please and entertain fans of the franchise.

The Hows (for each score):

What does the game accomplish in each category and how? An immediate and straightforward explanation of what the game offers and HOW it offers this.

Comparisons:

"Players who enjoy this game may have also enjoyed these other games..."

Reviewer Notes:

The reviewer's summary which comes after all of the aforementioned information.

A Sample Review

Wii Sports

Casual: 9.0
WSports is a completely new experience, one which will bring new and veteran gamers alike to their feet in excitement with an amazingly new but simple and intuitive way to play games. All gameplay is motion-based, using the Wii Remote's motion-sensing ability to translate your movement into on-screen action. Roll a ball in bowling, throw a punch in boxing, swing a bat in baseball: it's all really that simple.

Hardcore: 8.0
WSports offers a reasonable amount of goals for the veteran gamer, including increasingly difficult opponents as your skill increases and earning medals in minigames which may take you a long time to achieve, but in the end, it doesn't feel quite as satisfying and fulfilling as some other challenging games.

Non-Gamer: 10.0
For people of any race, age or gender, WSports marries the most simple and intuitive control scheme with fun and engrossing gameplay. This is the ultimate non-gamer game.

Party: 9.0
Newer gamers will spend a longer amount of time with WSports, but veteran gamers will no doubt find themselves hungering for more depth. Still, WSports is the definition of a party game for new and casual gamers as it will have people curious within minutes of watching the game (and the players) in motion.

Fan: 8.0
If you're a fan of sports games, the appeal of WSports will be immediate. However, certain missing key elements may discourage you from enjoying the game for longer periods of time, such as Baseball's lack of base running control or Golf's lack of club variety.

Those who enjoy Wii Sports might have also enjoyed:
Tennis games
Baseball games
Boxing games
Bowling games
Mario Party games
Games which focus on simple control schemes (Katamari, Super Monkey Ball, etc.)


Reviewer's Notes:
Wii Sports finds the perfect balance of accessibility and enjoyability, owing to its simple control scheme and fun, time-honored sports games. Had the game employed an online mode, it would easily find itself with a higher hardcore score, as this would keep hardcore players plugging away at the game for years to come as they try to scale leaderboards and outdo their opponents around the world. As it stands, it's an excellent piece of software to own and the best means of proving to people that the Wii's unique control scheme worlds wonderfully. A recommended buy for any gamer.


The idea is that a gamer can immediately check which score matters most to them and read the associated paragraph if they want. Games and their appeal are too large and too varied to choke them down to one defining score so it would be easier and more telling to rate individual facets of a game's appeal instead of trying to meld them all together into a score which certainly doesn't account for the variety of gaming tastes in the world.

Beyond that, the "if you enjoyed..." section, I feel, does far, FAR more for the gamer in the way of buying decisions because it immediately brings out some game types, genres and specific titles which share common likable traits with the game being reviewed.

It's not perfect, but no review system is and I think this would still be a great deal better than the standard "scroll through the Wall of Text to find the final score" system which is employed by most of the review industry right now.
"OK, first we need someone to complain about something trivial. Golden or S_B should do. Then we get someone to defend the game, like Bill or Mashiro. Finally add some Unclebob or Pro666 randomness and the thread should go to hell right away." -Pap64

Offline Ian Sane

  • Champion for Urban Champion
  • Score: 1
    • View Profile
RE: EDITORIALS: On Ratings
« Reply #57 on: October 18, 2007, 05:53:28 AM »
"Why not just add NWR's 'Recommended for Everyone,' 'Recommended for Fans' and 'Not Recommended' as an additional label for each game?"

I don't think those labels work for new games.  What does "for fans" mean for a brand new IP?  Fans of the genre?  Fans of the developer?  These labels work well for old games because "for fans" can mean people who liked the game back in the day.  It's a game that not everyone will like but if you liked it in the past you'll still want it.  That just does not apply for new games.

The brief format of the VC reviews also better fits cheap games.  So NWR recommends a game, I buy it and don't like it.  Had they been more detailed in their "review" I would find out about this one part of the game that sucks.  Big deal.  I'm out like 10 bucks at the most.  With a $60 purchase maybe I want some more detail in case something that I really won't like or really will like comes out in the details.

Offline Nick DiMola

  • Staff Alumnus
  • Score: 20
    • View Profile
    • PixlBit
RE: EDITORIALS: On Ratings
« Reply #58 on: October 18, 2007, 05:58:23 AM »
I think you are touching on a really nice system S_B, but it might be extremely hard to remove yourself (as the reviewer) to the point where you can see how a game fits the needs of a crowd you are not a part of. The marketing words for describing a gamer don't help either, but I can definitely see what you're going for.

The review system is extremely broken and I think as time has passed, some game publications are realizing this and trying to make strides toward revising the system to one that really fits the needs of the readers. Anything at this point that will change the system would be welcomed.
Check out PixlBit!

Offline Smash_Brother

  • Let me show you my poké-balls
  • Score: 3
    • View Profile
RE:EDITORIALS: On Ratings
« Reply #59 on: October 18, 2007, 06:03:20 AM »
Quote

Originally posted by: Mr. Jack
I think you are touching on a really nice system S_B, but it might be extremely hard to remove yourself (as the reviewer) to the point where you can see how a game fits the needs of a crowd you are not a part of. The marketing words for describing a gamer don't help either, but I can definitely see what you're going for.


I agree that this is the challenge that should befall every reviewer "Even if I didn't like it, it had _____ going for it and thus I expect ______ fans to enjoy it."

I've heard of reviews of RPGs where the reviewer basically starts the review by announcing that he hates RPGs. While I'm not a fan of RPGs either, I could still give some indication of what points of the game RPG fans would probably like.

The idea would be to communicate what the game has to offer everyone instead of using the reviewer as a "filter" which then explains what the game offered him/her.

Basically, it would establish what the game has and leave it to the reader to decide if they'd like that or not.
"OK, first we need someone to complain about something trivial. Golden or S_B should do. Then we get someone to defend the game, like Bill or Mashiro. Finally add some Unclebob or Pro666 randomness and the thread should go to hell right away." -Pap64

Offline Nick DiMola

  • Staff Alumnus
  • Score: 20
    • View Profile
    • PixlBit
RE: EDITORIALS: On Ratings
« Reply #60 on: October 18, 2007, 06:33:06 AM »
Hell it would even be nice if game reviews were done in a group setting. I want to hear all aspects and viewpoints so I can place myself somewhere in the group and figure out what the game offers me in particular. I really like the whole idea of multiple viewpoints depending on gamer type. That combined with an effective rating system and a write up describing the major features of the game would be awesome. I don't really read reviews anymore to figure out what I want. The combination of previews and videos from the collection of sites out there I can usually feel the game out well enough to make a decision before the review even hits. Most of the time when I read a review I use it as a validation tool, however most of the time it is ineffective because my tastes often do not line up with those of the reviewer.  
Check out PixlBit!

Offline BigJim

  • Score: 0
    • View Profile
RE: EDITORIALS: On Ratings
« Reply #61 on: October 18, 2007, 07:24:24 AM »
Recommended for Casual Players
Recommended for Enthusiasts
Recommended for All
Not Recommended

Score (1-10, no half points) & Review

?

"Enthusiasts" can encompass fans, fans of genres, hardcore players, etc. based on the context of the game, or in whatever context the reviewer uses to justify the selection (they should explain). And I use "Players" rather than "Gamers" to be able to encompass casual gamers and non-gamers.

I don't know if I would replace 5 ratings (graphics, sound, controls, etc) with another 5 ratings. I would probably give a game just 1 overall numbered score rating, using the targeted demographic (see above) as the intellectual scale in which it's judged, but S_B may be onto something too.

But certainly reviewers need to take themselves out of their reviews. It's not about them. They should be educated on the topic/genre, but still able to be as neutral as possible so they don't come at it with either a gripe or a glowing ball of fanboy sunshine.          
"wow."

Offline KDR_11k

  • boring person
  • Score: 28
    • View Profile
RE:EDITORIALS: On Ratings
« Reply #62 on: October 18, 2007, 07:43:40 AM »
Quote

Originally posted by: shammack
I don't have much sympathy for people who are too lazy to read a review and are only willing to look at a number when deciding what game to buy.  If that's the only factor going into their decision, it's pretty much a crapshoot because the numbers are so arbitrary and meaningless anyway.  If they buy a game and it's not what they expected from the 8.7 review (or whatever), they will whine.  If they didn't have the 8.7, they would have had to read the review and find out what to expect from the game, and maybe put some actual thought into their decision.  I don't see how creating a bunch of sub-ratings is any easier to understand than saying, "The multiplayer is a lot of fun at parties, but fans of traditional adventure games may be disappointed," or something like that.

Every game has different things that it does well or badly, and the only effective way to convey those things to the reader is to talk about them.  You just can't get that information across in shorthand.  Yes, you could write a review with all that stuff in it and still slap a rating on the end, but when you do that, that rating becomes the focus, and people ignore everything else and say, "This game got a 7.6 and the other game only got a 7.4!  I'd better go with this game!"  That should not happen.


I don't want to have to go through a whole bunch of text when the game's total crap anyway. Why do I have to read half a page when Score: 1.0 would have been enough to get it across that the game is so horrible that you'll want to hurt yourself? Similarily, if I have three games, two rated 7.0 and one rated 9.0 I know immediately which one to look into, without the scores I have to read each review individually and hope the reviewer is good enough to make a great game sound better than an average game in the text. Scores are for the first selection, the review text is for more information. A good score tells you this game is worth looking into, a bad score tells you immediately that the only purpose of reading the review would be to make fun of the game. Using scores for direct comparisons is silly, especially for 0.1 differences and going with a 100 point scale is just silly but the rough value of the score is still highly useful. There are more quality levels of games than just "must buy", "rental" and "avoid", a score lets you decide your personal threshold for each of these. E.g. I won't buy a game at full price if it's rated 7 because that usually leads to disappointment and when I think the text of a review sounds good but the rating is low it usually turns out that the issues I thought I could overlook are really too large to ignore.  

Also I think S_B's five ratings are too many, few games will have such different meanings to each group to warrant a separate score but then you might find a split that you cannot express with just these. Use multiple ratings where appropriate, not because you have to fill out a form.

Offline IceCold

  • I love you Vanilla Ice!
  • Score: 2
    • View Profile
RE:EDITORIALS: On Ratings
« Reply #63 on: October 18, 2007, 02:05:27 PM »
Quote

Originally posted by: KDR_11k
I think discussing "aesthetic experience" is kinda missing the mark. Treat games like art all you want but people treat them as entertainment, something they use when they're bored and they expect it to be FUN. This is the goal metric, how much fun does the game provide. I don't buy games for their creativity, their graphics or their game mechanics, I buy them for the fun that results out of the combination of all of these aspects. If aesthetic experience is what gives you enjoyment then by all means base your rating on it! However, don't pretend we have to be snobby about some external value like "artistic merit" and rate that over enjoyment. If you didn't enjoy it, say so, don't pretend that the art must be experienced despite being unpleasant to play.
*salutes*
"I used to sell furniture for a living. The trouble was, it was my own."
---------------------------------------------
"If your parents never had children, chances are you won't either."
----------------------------
"If it weren't for electricity we'd all be watching television by the candlelig

Offline Smash_Brother

  • Let me show you my poké-balls
  • Score: 3
    • View Profile
RE:EDITORIALS: On Ratings
« Reply #64 on: October 18, 2007, 02:32:13 PM »
Quote

Originally posted by: KDR_11k Also I think S_B's five ratings are too many, few games will have such different meanings to each group to warrant a separate score but then you might find a split that you cannot express with just these. Use multiple ratings where appropriate, not because you have to fill out a form.


In some cases, the scores will be N/A, but even with single player games, there's a definite "party" value if the game can entertain a crowd watching you play it.

I think the perspective is necessary because trying to apply a single numbered score to a game in the hopes of establishing how an individual will enjoy it is absolutely futile because the review knows nothing about its sole audience member, that being the reader.

When you come to a review looking to learn about the game, you might find that the game offers a shallow single player experience which is only 5 hours long, BUT that the game and its characters are immensely entertaining to watch so it could occupy a room full of people for a whole night.

Moreover, my system isn't perfect, but it at least gives readers five options so they can align themselves with a particular type of gamer and hopefully get a better idea about how they would enjoy the game.

After all, no one knows their tendencies better than they do and a review which only offers a lone number is attempting to somehow bridge that gap without any prior knowledge, something which just ain't gonna happen.
"OK, first we need someone to complain about something trivial. Golden or S_B should do. Then we get someone to defend the game, like Bill or Mashiro. Finally add some Unclebob or Pro666 randomness and the thread should go to hell right away." -Pap64

Offline KDR_11k

  • boring person
  • Score: 28
    • View Profile
RE: EDITORIALS: On Ratings
« Reply #65 on: October 18, 2007, 10:20:20 PM »
Five categories for every game will just lead to arbitrary numbers. Who the hell would test an SP game for its ability to entertain a crowd of spectators? Don't force the reviewer to give numbers he didn't really want to be separate because then it gets arbitrary and the reviewer might feel compelled to enter a different score into each box just so he can't be accused of copy & paste. Really, only add meaningful divisions. I've had enough situations where I was filling out a form and most of the given options didn't make a difference while the ones that would weren't on the form. Never mind that readers might be confused which one of the load of options meets them when all they know is "I play my RPGs for the story".

Offline oohhboy

  • Forum Friend or Foe?
  • Score: 38
    • View Profile
RE:EDITORIALS: On Ratings
« Reply #66 on: October 18, 2007, 10:59:32 PM »
Quote

Originally posted by: NewsBot

This is a crisis moment for modern art, where the melding of consumerism and artistry has created a group of movie-goers, musicheads, and gamers who believe that art can be rated on an objective scale. The idea of applying ratings to paintings or sculptures (even modern ones) is pretty unthinkable. Though it has taken thousands of years, no one today would question the cultural validity of paintings or sculptures as a medium.


Rubbish. For as long art has existed, we have applied ratings and scales to it. For things like classical art, there is no limit to that scale. An item can be brought for a million dollars or a hundred million. Is that not an arbitrary value tied to a scale we call money?

Quote

Video games are subject to the shackles of ratings more so than other arts due to a couple reasons: one is their high cost. A gamer may only be able to purchase one game every paycheck, or every month, and the difference between a 9.0 and a 9.5 suddenly becomes important. Another reason for excessive ratings in the game world is their status as software. Since they are a program that must perform certain functions, problems like a lack of polish in graphical presentation, poorly designed controls, or simple bugs and errors can all be treated as quantifiable leaps that the user should or should not have to make, in the reviewer's mind. Yet when a journalist reviews a game under our current system, he must also attempt to apply numbers to the game's artistry and his overall level of satisfaction, in the hopes of giving a solid purchase recommendation to the video game world.


A painting or statue made out of rotting apples would have no value what so ever. The same can be said for games that are made up almost completely out of cut-scenes or have controls so poorly designed that it renders the game unplayable. You might say that the rotting apples are a statement and that has value in itself, but would appear asinine to everybody else.

Quote

Most reviewers would admit to being concerned more with the artistry of a game than with its functionality as software; these two pieces are necessary parts of a review, but by this time in gaming history, functionality should be a non-issue. Slowdown and control glitches will always be with us, but a reviewer must comment on them only insofar as they hinder the experience of playing the game. The game's goals as an aesthetic experience must be paramount in the reviewer's mind.
   

While true in a perfect world, you said it yourself, "functionality should be a non-issue". You may have an absolutely fantastic work of art, but if you display it next to a cesspit, it will detract from what's really there.

Quote

Yet numbers dominate our discourse; if a reviewer rates a game lower than his peers, he is seen as having an incorrect position. And though every journalist may strive to write about a game before applying a rating, the overall score that comes at the end of the review can never fully be out of his mind. It is supposed to be a reflection of where he thinks the game falls on a scale of 1 (for terrible) to 10 (for incredible). It can supposedly be compared to his other reviews: if he gave a 9.0 to a game I didn't like, then I have no reason to believe that his 6.0 for a different game is accurate.
   

While that is a problem, and it would be impossible to remove the reviewer's personality and bias, doing so would a be disservice. As long as the reviewer has made clear their dissatisfaction and why in a review, there should be no reason why a game shouldn't as along it is justified.

Quote

Reviews can never be fully separated from their rating: the philosophy of numerical scales forces reviewers to give reasons why the game is better than an 8.0 but less than a 9.0. Though this may aid the purchase recommendation part of the review, it does little to encourage dialog about a game's actual merits. The score is a straw man to argue against, with the game's aesthetic qualities mere support for why it was deserved.    


Limited resources and time demand review scores. It quantifies how much a person my pay for a game. If a game is 9-10, I am far more likely to pay the full price for the game let it be 50 - 60 dollars depending on the console. This also extends to budget games. This works, because games have a local fixed ceiling in price. An understanding that if prices rise above a certain point you will be punished for lower sales.

Quote

Even Roger Ebert (who has no doubt that movies are art and most video games aren't) claims that his stars and his thumb are worth less than his written review, yet he will only put four-star movies on his top ten list each year. Similarly, when the “Game of the Year" hype contests roll around, scores are a main part of the debate. Is it possible for a 9.0 average game to pull ahead of all the 9.5s and 10s to steal the contest? Does anyone truly believe that these year-end lists are anything more than phoned in months in advance?    


Check back to your own end of year awards. You don't mention scores and scores shouldn't be mentioned. The moment that happens it becomes a genitalia swinging contest. Anyone who mentions scores at that point is a fool since you are already at the top, the cream of the crop. A special region where scores are rendered irrelevant due to the fact that everyone has collectively agreed that it was worth "Full price".

Quote

Reviews without ratings are less satisfying for readers because they do not supply the tidy summary of a game's worth that is expected under the current conditions. A review without a number cannot be compared to another review instantly, and the reviewer cannot be looked down upon by the public until his words are read. Many reviewers may feel pressure to not give the “wrong" score for a beloved franchise installment, hoping instead to say things that are in line with other reviewers. If he is the standalone aberration on MetaCritic, he will be fighting consensus and dismissed.    


While certainly true that this does happen, if actually read most of those outliner scores, they are horribly written reviews, in general. However, this can go the wrong way, where the hype of a game can override even the body of the review render the entire system defunct. *Cough* Halo *Cough*

Quote

Yet ratings never make sense. The Bit Generations titles are so simple that a rating of 8.5 doesn't mean the same thing as an 8.5 given to a Zelda game; the first may be too high for a simple game, while the second too low for a much more complex game. Does any reviewer honestly look at Tetris and Zelda and say “Zelda is better" as if the two could be compared? When Nintendo releases the next console Zelda or Mario game, is a score of 9.0 going to dissuade you from purchasing it? Do journalists ever give 10s to games outside of established franchises? Even within genres, comparing two very similar games like Okami and Zelda seems fruitless if we must conclude that one is superior over the other. The only comparison that seems appropriate is whether a new Zelda game is as good as the previous ones (in which case I may have to revise my score for Twilight Princess).    


The problem is that you base this assumption that both games are worth the same monetarily and serve the same function and have the same function. If two similar games get the same score, then tough luck to the buyer deciding between the two, or for the optimist, hey look, two games I can buy and I will be happy with it regardless of choice. This is especially true if the both games were reviewed by the same reviewer.

Quote

Reviewing a game's graphics, sound, or control too is a nonsensical idea: does a high polygon count within the framework of realism look better than a simple and striking fantasy design? No game is worth less for having blocky graphics if it works in context with the story; not all games can have their graphics measured in the same way.    


What happens if it is not within the context of the story, your screwed right? Well, yeah. If I, with no thought what so ever, threw paint against a wall it would have value under your definition. The only thing I had demonstrated would be my sheer laziness. If I decided instead to place some cohesiveness to the entire thing, then there maybe something of value here. Rouge-likes, with their ASCII graphics work because there is an understanding, like a book, you are meant to imagine what is happening. Never the less, this does not excuse developers who could have done better. Your next paragraph affirm this.

Quote

Instead of writing about whether a game fulfills my preconceptions for what a good game looks like, sounds like, and plays like, I should be compelled as a reviewer to rate the aesthetic experience I had. Though this is a subjective statement of my opinion, it can be qualified by my appraisal of a game's graphical and aural design, as well as my opinion of how successful the game was at creating a world, delivering a feeling of suspense, showing me beautiful images, giving me a sandbox to play in, telling a story, or whatever else the game may have tried to do. No single philosophy of game design is correct, and with as many artists as there are in the game industry we ought to encourage them to take their individual ideals as far as possible. This is why games like Metal Gear Solid and Super Mario 64 can both be praised for their different visions of what video games can do.    


Those preconceptions are formed by your very past experiences that allow you to rate any aesthetic experience. Without it, you would think and accept Bartz horse riding is a good game.

Quote

Removing scores from reviews will not prevent us from discussing games, comparing disparate genres, or discussing objective quality. Instead, it will allow journalists the freedom to examine a game as a holistic and inclusive experience, an exercise that has been constricted for decades by universal participation in scoring. Having to quantify a game's graphics, sound, control, and fun factor are roadblocks to true discussion. The best art you will ever see cannot be summed up in an essay, or a review. To this day people are discussing the aesthetic experience known as Michaelangelo's La Pieta, or Stanley Kubrick's 2001: A Space Odyssey. As soon as we believe that we can fully know and understand these works of art, we have lost the ability to ever know anything about them. Only in the ongoing discussion of how video games affect us, and what keeps us coming back for more, can we break through the meaningless numbers and make gaming journalism into something more than just software reviews and purchase recommendations.


Stanley Kubrick's 2001: A Space Odyssey I found was rubbish even though I saw it for free. Regardless of whether you assign a score or not, people will/can discuss the game. That will never be in danger. The real problem is that you have assigned something different to the score than what it should be. It is indeed a purchase recommendation regardless of whether you like it or not as there isn't infinite amount of time or money or will to play all games. Why play a game even if you got it for free if you don't derive any enjoyment what so ever?

Edit. Fixed bold.
I'm Lacus. I'm fine as Lacus!
Pffh. Toilet paper? What do you think cats are for?

Offline KDR_11k

  • boring person
  • Score: 28
    • View Profile
RE: EDITORIALS: On Ratings
« Reply #67 on: October 20, 2007, 03:37:11 AM »
As for the art comparison, let's take architecture. Architecture is art, there are many famous artwork houses around the world. That does not mean that functionality is not important. An art-house that is not structurally sound and likely to collapse is not a good idea, neither is a house with an interior that's completely unusable because it's built like a videogame level or something. And contrary to what Sony thinks, if the architect places the doors wrong people WILL complain.

Offline Nick DiMola

  • Staff Alumnus
  • Score: 20
    • View Profile
    • PixlBit
RE: EDITORIALS: On Ratings
« Reply #68 on: October 31, 2007, 06:58:09 AM »
Looks like Kotaku is running a similar Editorial to what Evan has put together here.
Check out PixlBit!

Offline Nick DiMola

  • Staff Alumnus
  • Score: 20
    • View Profile
    • PixlBit
RE: EDITORIALS: On Ratings
« Reply #69 on: October 31, 2007, 07:06:35 AM »
I'm pretty sure the mass of gaming enthusiasts out there is starting to realize that there is an obvious flaw in the system of reviews. We need to streamline this process and put together a better system. Lots of good ideas have been put forward in this thread, it would be nice to see some of these ideas get implemented and put to good use.
Check out PixlBit!

Offline vudu

  • You'd probably all be better off if I really were dead.
  • NWR Junior Ranger
  • Score: -19
    • View Profile
RE: EDITORIALS: On Ratings
« Reply #70 on: October 31, 2007, 07:31:21 AM »
Oh, I know you didn't just copy and paste the entire article in your post.  That's a big no-no.

Favorite quote:
Quote

Completely made up rule of thumb: The more numbers a reviewer uses, the more they are trying to authenticate their own bullshit.
Can I get a Hell Yeah?
Why must all things be so bright? Why can things not appear only in hues of brown! I am so serious about this! Dull colors are the future! The next generation! I will never accept a world with such bright colors! It is far too childish! I will rage against your cheery palette with my last breath!

Offline Nick DiMola

  • Staff Alumnus
  • Score: 20
    • View Profile
    • PixlBit
RE:EDITORIALS: On Ratings
« Reply #71 on: October 31, 2007, 08:00:14 AM »
Quote

Originally posted by: vudu
Oh, I know you didn't just copy and paste the entire article in your post.  That's a big no-no.

Favorite quote:
Quote

Completely made up rule of thumb: The more numbers a reviewer uses, the more they are trying to authenticate their own bullshit.
Can I get a Hell Yeah?


Good point vudu, I removed the text from my post, didn't remember that it was a rule.
Check out PixlBit!

Offline Kairon

  • T_T
  • NWR Staff Pro
  • Score: 48
    • View Profile
RE:EDITORIALS: On Ratings
« Reply #72 on: October 31, 2007, 08:05:20 AM »
ummmm... Hell yeah?

I mean, I certainly expect the world to turn into a utopia of millions of Svevans writing long thesis-like reviews that lack scores, but ...omg...

Quote

Originally posted by: oohhboy
Rubbish. For as long art has existed, we have applied ratings and scales to it. For things like classical art, there is no limit to that scale. An item can be brought for a million dollars or a hundred million. Is that not an arbitrary value tied to a scale we call money?


Oohhboy! You just gave me an awesome idea!

... why don't we rate games... using MONEY!

Why isn't the review score a monetary amount that the reviewer says would be a "fair price" for the game? That way we wouldn't have 9.0's or 8.0's... we'd have games that are great for $30 but cost $50, and games that people should and would pay $60 for (Zelda titles for example) but only cost $40 (Zack & Wiki)!

...bad idea? ... yeah, probably... but it's a new one at least!
Carmine Red, Associate Editor

A glooming peace this morning with it brings;
The sun, for sorrow, will not show his head:
Go hence, to have more talk of these sad things;
Some shall be pardon'd, and some punished:
For never was a story of more woe
Than this of Sega and her Mashiro.

Offline vudu

  • You'd probably all be better off if I really were dead.
  • NWR Junior Ranger
  • Score: -19
    • View Profile
RE: EDITORIALS: On Ratings
« Reply #73 on: October 31, 2007, 08:20:10 AM »
Because money's not an absolute value.  If I have more money than you do I may be willing to part with more money for a game I enjoy just as much as you.  We're not communists.

Can I get a Hell Yeah?
Why must all things be so bright? Why can things not appear only in hues of brown! I am so serious about this! Dull colors are the future! The next generation! I will never accept a world with such bright colors! It is far too childish! I will rage against your cheery palette with my last breath!

Offline Smash_Brother

  • Let me show you my poké-balls
  • Score: 3
    • View Profile
RE:EDITORIALS: On Ratings
« Reply #74 on: October 31, 2007, 08:46:37 AM »
Quote

Originally posted by: KDR_11k
Five categories for every game will just lead to arbitrary numbers. Who the hell would test an SP game for its ability to entertain a crowd of spectators?


...which is what the N/A is for when the score wouldn't apply to a game.

Quote

Really, only add meaningful divisions. I've had enough situations where I was filling out a form and most of the given options didn't make a difference while the ones that would weren't on the form. Never mind that readers might be confused which one of the load of options meets them when all they know is "I play my RPGs for the story".


This is like suggesting that we shouldn't have horoscopes for different signs because people are too stupid to figure out which sign they are.

You're missing a very simple point here and that is that basically anything is better than a player reading a review and not knowing that the reviewer sits 100% opposite them on all their tastes and opinions. By asking that the reviewer attempt to place him/herself in the shoes of someone who might have enjoyed the game, it forces them to consider it in a light that might be more appropriate for the review.

And even if it isn't as effective as it could be, it's STILL worlds better than not trying whatsoever which is what most reviewers currently do.

If the purpose of a review is to save the reader from a potentially poor buying decision, then that goal is best accomplished via attempting to cover different bases.

Quote

Never mind that readers might be confused which one of the load of options meets them when all they know is "I play my RPGs for the story".


Most reviews have 5 scores already, covering the graphics, sound, etc. If this didn't cripple the minds of those attempting to decipher them, than a different 5 scores won't do so, either.

And if someone enjoys RPGs for the story, it would be mentioned under the normal section that the RPG offers a deep story and shown under the hardcore section if it's difficult.
"OK, first we need someone to complain about something trivial. Golden or S_B should do. Then we get someone to defend the game, like Bill or Mashiro. Finally add some Unclebob or Pro666 randomness and the thread should go to hell right away." -Pap64