Author Topic: How do you consider ratings?  (Read 19924 times)

0 Members and 1 Guest are viewing this topic.

Offline that Baby guy

  • He's a real Ei-Ei-Poo!
  • Score: 379
    • View Profile
How do you consider ratings?
« on: September 14, 2007, 09:09:14 PM »
I was thinking a little bit about numerical ratings for video games.  I can't seem to figure out how it works, honestly.  There are so many different systems out there, as well as so many different games, and so many different methods used, that I think the industry needs some sort of shock to let people have a clearer understanding of what numerical or other ratings mean.

With movies, there's a similar problem, but since this industry has been around longer, people seem to have been able to earn an understanding.  You have Roger and Ebert, with their thumbs, star ratings, and online percentage ratings.  Roger and Ebert seem to actually be the least influential, IMO.  With a two point system, as well as hundreds or thousands of questionable calls, they just aren't very trustworthy.  Besides, one of them bad-mouthed games, too.  The next, probably better system currently used is online rankings.  Several sites like Yahoo, Fandango, and others gather up professional critic reviews, user reviews, and all the accompanied ratings to provide readers a look at what several think of the movie, as well as a consensus score in a few different categories.  To me, this seems like a nice way to go, and the percentages assigned often fall in the same category as school grades do.  The third is an older, more well-known system, which uses anywhere from one to four stars to rate a movie.  A one-star movie is deemed the worst, while a four-star movie is considered the best.  This system usually pairs the star-ranking with a mini-review or info-snippet, so movie-viewers can decide whether the movie and rating is appropriate to them.

Now, I have to ask how video games should be rated.  What would be the most efficient way?  What do you like?  What do you dislike?  What don't you understand?

For me, I hate that nearly every website and magazine out there uses a scale of ten or five, and very few have interpretations to what these numbers actually mean.  To further muck these up, the reviewers themselves do not have a consensus on what numbers mean.  Don't get me wrong.  I'm not griping about NWR here.  I find the entire industry is in this state.  With so many different reviewers under each publication, so many different genres, and so many different sources out there, it seems impossible to learn much about any specific title by its rating.  To add fuel to the fire, I've seen stories, heard rumors, and read speculation that sometimes, the reviewer is forced to alter their numbers by employer, without changing the content of the review.

I personally think that the best way to rate a video game might be to adopt something more similar to the star system.  A general rating as well as an information quip could be useful.  I second-guess that rating, though, because it seems to me that the industry is a little more competitive than what I suggest, and that people need to be able to learn more from the rating than even supplied there.  Then again, I think that the star rating would really help to identify where a game stands in its respective genre, and perhaps provoke the interested party to take stock in the written review.  It's hard to say, though.  The important thing is that this system wouldn't work with numbers, really.  It wouldn't fit in with what we see now, and it would take some getting used to.  On a side note, I find that this is somewhat similar to NWR's pros and cons section, coincidentally, my favorite part of the reviews here.

I also thought that the percentage needs to be based on a more concrete and less abstract system when it comes to reviews.  Most websites are afraid to give games numbers in failing ranges.  Anywhere buy NWR, do you rarely see a four, and you never see a three.  Perhaps if we considered ratings as more of a ranking.  A 10 would place the game at 100%, which would mean that if you were to take every game released on current platforms (or comparative platforms, at least) that the game described would be in the very top percentile of all games.  A fifty percent would mean the game is better than half the comparative games out there.  A one percent would mean the game is worse than ninety-nine percent of everything that's been released comparably.  This would hopefully encourage raters to use the lower spectrums more, and allow consumers a better idea at what is being looked at.  The error in this system has to be a lack of subjectivity.  If someone loves sports games or football in general, Madden might be in the top five percent, even if it isn't much of a great game.  However, if our reviewer does not understand football rules, concept, strategy, or culture, Madden could wind up ranked below fifty percent.  Since reviewers are typically just names to the consumer, such a discrepancy could wind up being very confusing, perhaps even more confusing than the way things are now.  In essence, each individual's personal taste, as well as the genre of the game, could end up influencing the score, perhaps much more than the actual game's content could.  I think that this type of rating would be most effective when paired with a list of scores that other games received by the reviewer.  Given this, it could be a powerful concept, though it takes a considerably longer time to learn from this method of scoring.

Anyone else have their two cents?

Offline GoldenPhoenix

  • Now it's a party!
  • Score: 42
    • View Profile
RE:How do you consider ratings?
« Reply #1 on: September 14, 2007, 09:30:50 PM »
Very interesting question, personally I like the 10 pt system, but I feel it should be more standarized. One thing I've really respected about Matt at IGN is that his numbers actually seem to mean something. Even if a game gets a 5 it means that the game may still be worth a shot, but don't expect anything more than average. While a 8 or above game means it is good to great. Now if you have a reviewer that sticks to that system (and also adopts a comparison scale like you suggested) I think it would work great. It is when reviewers don't have any standards and while one reviewer may give a game a 5 that could mean they thought the game was terrible, while another could give it a 7 and that is comparable to the 5 of another person.
Switch Friend Code: SW-4185-3173-1144

Offline King of Twitch

  • twitch.tv/zapr2k i live for this
  • Score: 141
    • View Profile
RE: How do you consider ratings?
« Reply #2 on: September 14, 2007, 10:01:07 PM »
I also like the 10-point system, here's one way it could work

10: contains some revolutionary or extraordinary aspects that set it above the average game in recent memory. meticulous detail in every facet of the game. good length and maybe a few extra features; a game you want to show your grandchildren.
9: highly entertaining, well-rounded with a few extremely well-done elements
8: minor graphical flaws, maybe a little short, maybe not appealing to everyone, lacks a few features but still worth $50
7: spotty control and/or plain graphics, average overall. but if you like the genre you may find some things you like. not worth full price though
6: few redeeming qualities.. yet strangely may be spoken fondly of in 10 years.
5-0: Acclaim and Activision
negative points: non-games

With those parameters you can easily look at the scale and tell which zone a game should lie. Even so, I prefer words words words with every review.  
"I deem his stream to be supreme and highly esteem his Fortnite team!" - The Doritos Pope and his Mountain Dew Crew.

Offline that Baby guy

  • He's a real Ei-Ei-Poo!
  • Score: 379
    • View Profile
RE: How do you consider ratings?
« Reply #3 on: September 14, 2007, 10:01:58 PM »
Well, remember that most console titles now retail for $60, too.

Offline King of Twitch

  • twitch.tv/zapr2k i live for this
  • Score: 141
    • View Profile
RE: How do you consider ratings?
« Reply #4 on: September 14, 2007, 10:07:12 PM »
Automatic 7 for being over $50
"I deem his stream to be supreme and highly esteem his Fortnite team!" - The Doritos Pope and his Mountain Dew Crew.

Offline KDR_11k

  • boring person
  • Score: 28
    • View Profile
RE: How do you consider ratings?
« Reply #5 on: September 14, 2007, 11:04:23 PM »
9-10: Great game, there's almost no unfun parts in the game. Many publications refuse to give a 10 (or 100%) on principle so IMO it's meaningless.
8: Good game. Mostly well executed though there may be a few minor unfun parts.
7: Average game (par). Sound concept and all but has a lot of unfun parts. Never pay full price for these!
6 and lower: The unfun is taking up most of the game time. Stay the hell away. 6 may be tolerable for a few minutes but anything below is lulz at best.

This is also consistent with the terminologsy I use (whenever I say good but not great I mean 8/10).

Offline Kairon

  • T_T
  • NWR Staff Pro
  • Score: 48
    • View Profile
RE:How do you consider ratings?
« Reply #6 on: September 15, 2007, 12:39:34 AM »
I actually like 5 point rating scales, or a rating system like the one the San Francisco Chronicle employs for movies: a system where we can have

1. a man in a chair jumping out of his seat clapping
2. a man in his chair clapping
3. a man in his chair watching attentively
4. a man in his chair sleeping
5. a chair with no man in sight

The reason I like that system is because it gives me an emotional base to touch on first, so that I can tell from a glance "this movie will wow me" or "this movie is average" or "this movie will put me to sleep." And it achieves this with infinitely more efficacy than any point or number system!

But aside from liking it for being more emotionally precise a rating system, it also remains vague enough that I can then read the review and get a sense of where the movie succeeds and fails. Reading comprehension is my best tool when tackling reviews, and the best reviewer won't be one whom I agree with, but the one whom I can always understand the viewpoint of. After all, we're all individuals, and I want to know if I will like the movie, not Mick LaSalle (who's awesome, by the way). A good reviewer can explain the experience such that I can easily say after reading a review that "I'll like this movie even though the reviewer didn't and rated it ____" or maybe even say "I won't like this movie even though it was rated _____."

Reviews like that are a joy to read, not because they break movies down into ridiculous areas like "visuals" or "sound" or "storyline" but because they preserve the holistic experience of entertainment, and can be fun just because an active reading of the review is almost like a conversation, the reviewer points out certain things and I get to decide whether I like them or not.  
Carmine Red, Associate Editor

A glooming peace this morning with it brings;
The sun, for sorrow, will not show his head:
Go hence, to have more talk of these sad things;
Some shall be pardon'd, and some punished:
For never was a story of more woe
Than this of Sega and her Mashiro.

Offline that Baby guy

  • He's a real Ei-Ei-Poo!
  • Score: 379
    • View Profile
RE: How do you consider ratings?
« Reply #7 on: September 15, 2007, 04:09:33 AM »
So KDR and MJR, let me ask you this.  What's the point of having a 1-5 when they all mean pretty much the same thing?

Offline SixthAngel

  • Score: 18
    • View Profile
RE:How do you consider ratings?
« Reply #8 on: September 15, 2007, 07:34:56 AM »
A five point or five star system is the way to go.  In a in a 10 point system anything 6 and below tends to be thought of as worthless.  This isn't something that will change through effort of game sites, that is the way things are since middle school so it tends to carry on into these reviews.  Look at MJR.  He suggests a 10 point review system yet made 5 points equal to zero making the system back into a five point system.

5 points or stars differentiates the review from other concepts like grades and allows a far more reasonable approach to reviews.

Offline King of Twitch

  • twitch.tv/zapr2k i live for this
  • Score: 141
    • View Profile
RE: How do you consider ratings?
« Reply #9 on: September 15, 2007, 08:38:07 AM »
Yea it's more of a carryover from grade school and i was too tired to fill the rest in. 5-star would make sense but you couldn't give MK: DS 3.9 stars, it just wouldn't be the same.
"I deem his stream to be supreme and highly esteem his Fortnite team!" - The Doritos Pope and his Mountain Dew Crew.

Offline IceCold

  • I love you Vanilla Ice!
  • Score: 2
    • View Profile
RE:How do you consider ratings?
« Reply #10 on: September 15, 2007, 08:48:53 AM »
SixthAngel is completely right - while a 5 should be an average game which might be worth a shot, in the games industry that number is 7 instead. It's like in high school where the average of the class was always around 70% - that's carried over into people's mentality.

And that's why sites which try to right the problem by themselves, however honourable it may seem, are doomed to fail because the rest of the industry just won't change. And then the reviewed game has a lower score than it deserves when compared to other games.
"I used to sell furniture for a living. The trouble was, it was my own."
---------------------------------------------
"If your parents never had children, chances are you won't either."
----------------------------
"If it weren't for electricity we'd all be watching television by the candlelig

Offline NinGurl69 *huggles

  • HI I'M CRAZY
  • Score: 28
    • View Profile
    • Six Sided Video
RE: How do you consider ratings?
« Reply #11 on: September 15, 2007, 09:48:16 AM »
It's either FIST or it's WEAK.
:: Six Sided Video .com ~ Pietriots.com ::
PRO IS SERIOUS. GET SERIOUS.

Offline that Baby guy

  • He's a real Ei-Ei-Poo!
  • Score: 379
    • View Profile
RE:How do you consider ratings?
« Reply #12 on: September 15, 2007, 01:35:52 PM »
Quote

Originally posted by: IceCold
SixthAngel is completely right - while a 5 should be an average game which might be worth a shot, in the games industry that number is 7 instead. It's like in high school where the average of the class was always around 70% - that's carried over into people's mentality.

And that's why sites which try to right the problem by themselves, however honourable it may seem, are doomed to fail because the rest of the industry just won't change. And then the reviewed game has a lower score than it deserves when compared to other games.


See, that's one reason why I think the star-type system is better.  People don't write off a two-star movie like they do a 5/10 or a 2.5/5, even though each represents half of a full score.

The problem is that there isn't a right and wrong to making a video game.  You can't say a game got 75% of itself correct.  That's more of why I disagree with the high ratings some writers give out.  And still, I must repeat that I hate it when the bottom half of a scale just seems like a placeholder.  For some places, it seems like the scale is from 5-10, which, to me, is just a little sad.

Offline UERD

  • Score: 0
    • View Profile
RE: How do you consider ratings?
« Reply #13 on: September 15, 2007, 01:55:31 PM »
I like Kairon's emotional-response idea. In any case, there should be solid benchmarks attached to each value. I know that not everything can be broken down objectively, but ratings based on comparable criteria can differ greatly between reviewers (or even the same reviewer over a period of time) based on things like changing perceptions and expectations, prior biases, etc.

Also, I find that reviews for niche games are kind of unhelpful. I really enjoyed Baten Kaitos Origins, for example, but that game got scores that were literally all over the board- and all those reviews hinged on the battle system, which was the 'niche' aspect of the game (more specifically, whether the reviewer liked it or not).
"I'm looking for shrunken heads w/ DVD playback options. I figure I can hang them in my car like dice. Will you help me?"
- thatguy

"Can you shoot out customizable fireballs? Then why should your Mii be able to?"
- vudu

Offline bustin98

  • Bustin' out kids
  • Score: 30
    • View Profile
    • Web Design Web Hosting Computer Sales and Service
RE:How do you consider ratings?
« Reply #14 on: September 15, 2007, 07:39:18 PM »
That niche market remark hits on what I think happens with Wii reviews.

I like the 10 point and I'd like to see the average score be a 5, but I still have a carry-over from school grading systems.

Another good point is why even have the lower numbers if they basically mean the same. In the same train of thought, why have a 10 because realistically no game is perfect, though a few games have elicited that response.

How about this idea: group the scores like:

1 : should be burned
2-4 : poor, almost unplayable
5-7 : average
8-9 : recommended
10 : near-perfect must buy

Where the grouped numbers mean about the same but I'd prefer a 9 over an 8, or a 7 over a 5. So it gives freedom for a reviewer to critically judge a game and at the same time give an idea of how they'd rank it amoung the other similarly graded games.

Offline KDR_11k

  • boring person
  • Score: 28
    • View Profile
RE:How do you consider ratings?
« Reply #15 on: September 15, 2007, 10:59:52 PM »
Quote

Originally posted by: thatguy
So KDR and MJR, let me ask you this.  What's the point of having a 1-5 when they all mean pretty much the same thing?


Dealing with expectations, it's automatically assumed that ~75% is par. Personally I'd use a par, above par, below par scale so the point of reference is no longer x/y but >0 and <0. That only creates confusion though as you can see in my user review of Wario Ware: Touched! In a later review I just wrote my thoughts down instead of using any consistent marks for scoring.

Reminds me, I should review something but I have nothing to review right now... Maybe I should get Tingle's Rosy Rupeeland?

Offline Infernal Monkey

  • burly British nanny wrapped in a blender
  • Score: 2
    • View Profile
RE: How do you consider ratings?
« Reply #16 on: September 15, 2007, 11:44:55 PM »
Games shouldn't be given a rating at all. It leads to everyone skipping the review text and just whinging about the little number at the end. But!

10 - Avoid At All Costs, Overrated Rubbish
9 - Klax Wave
8 - Number Eight
7 - Not Enough Skins
6 - Pizza Time
5 - Leftover Pizza Time
4 - Stupid Lisa Garbage Face
3 - Video Game
2 - Developed by Square-Enix
1 - Excellent for parties, recommended

Offline couchmonkey

  • I tye dyed my Wii and I love it
  • Score: 2
    • View Profile
RE: How do you consider ratings?
« Reply #17 on: September 17, 2007, 06:12:52 AM »
The only thing I know for sure is we don't need no 100-point scales.  It just leads to confusion over whether Bald Space Marine 5 is really better than Rainbow Platformer 12 because it got a 9.7 instead of a 9.4.
That's my opinion, not yours.
Now Playing: The Adventures of Link, Super Street Fighter 4, Dragon Quest IX

Offline that Baby guy

  • He's a real Ei-Ei-Poo!
  • Score: 379
    • View Profile
RE:How do you consider ratings?
« Reply #18 on: September 17, 2007, 06:44:14 AM »
Quote

Originally posted by: Infernal Monkey
Games shouldn't be given a rating at all. It leads to everyone skipping the review text and just whinging about the little number at the end. But!

10 - Avoid At All Costs, Overrated Rubbish
9 - Klax Wave
8 - Number Eight
7 - Not Enough Skins
6 - Pizza Time
5 - Leftover Pizza Time
4 - Stupid Lisa Garbage Face
3 - Video Game
2 - Developed by Square-Enix
1 - Excellent for parties, recommended


Now, is six for TMNT games, or is it reserved for the little-known sequel to Burger Time?

Offline Ian Sane

  • Champion for Urban Champion
  • Score: 2
    • View Profile
RE: How do you consider ratings?
« Reply #19 on: September 17, 2007, 07:13:07 AM »
I find it easiest to understand number ratings if they work like school grades.  So above 90% is exceptionally good, 80-89% is quite good, 70-79% is okay but there's a lot of room for improvement and once you start getting below that things start turning to crap.  I don't know what schools the guys who say a 5 out of ten is average went to.  It must have been one full of really stupid kids.  50 percent sucks.

Typically with games I'm following the titles I'm interested in anyway.  I check over the Game Rankings and if some game I didn't expect is getting huge scores I'll check it out.  I don't need to read a full review for games I'm not interested in who get crappy scores.  And for games I really want to get I only need the "this doesn't suck" confirmation.  It gets 8 or higher and I'm interested I don't need to read the review.  I only read if then if it gets a bad score that I didn't expect to find out why.

The only problems I see with reviews is problems with the reviewers themselves.  Some get wrapped up in hype and overrate.  Some have a strong bias and underrate.  Some are blatantly paid off to give a good review.  But having a score out of 10 doesn't change that.  If it was only a write up and we had to read every review that doesn't change IGN getting a bug up their butt about Mario Kart not having skins or overrating Star Fox Adventures.

One sight I visit frequently because it reviews older games is VideoGameCritic.net.  Nearly everything is reviewed by one guy who collects games.  Each review is about a paragraph and he assigns a letter grade like in school.  It works really well.  Not a bad format to copy.  But I don't think all sites should use this method.  The sheer number of titles on his site make the paragraph format ideal but a major site that only covers current games probably should provide more detail.

Offline that Baby guy

  • He's a real Ei-Ei-Poo!
  • Score: 379
    • View Profile
RE: How do you consider ratings?
« Reply #20 on: September 17, 2007, 03:59:29 PM »
I have to disagree, though.  When you grade based on accuracy, sure, a school grade scale works, because you need to know well above fifty percent to be considered knowledgeable on the necessary material.  Game's can't be detailed as accurate.  There's no black or white, right or wrong.  That's why the grade scale system doesn't work.  The good is all lumped into 7-8 points, with too little space to say what's what, and still, not enough people reading the content when they don't know about the game.  If they used a smaller system, I think more would at least skim the review, or if a full ten-point scale were used instead of the abysmal pseudo-ten-point scale, that at best uses the top half, things would be more clear.

Offline Djunknown

  • HEY! HEY! LISTEN!
  • Score: 1
    • View Profile
RE:How do you consider ratings?
« Reply #21 on: September 17, 2007, 04:17:09 PM »
Quote

I don't know what schools the guys who say a 5 out of ten is average went to. It must have been one full of really stupid kids. 50 percent sucks.


I believe a magazine called Game Now or something like that (It was one of EGM's sister magazines) tried something like that. I was confused at first, I thought they were really, really tough on games. Still didn't make sense after a second read-through.

I'm a bit partial to the 10 point scale, with .5 intervals. If a game gets an 8.9, what's significantly wrong with it to merit a .1 difference? An 8.5 to me sounds like a great game, but either a minor artistic (re-used textures) or technical glitch (like long loading times) stops it from getting the coveted 9.0. A 9.0-9.5 are the games that's a thrill from begginning to end. Its a textbook example of its genre done right, but not perfectly.

For me, a 10 is when the developer maxes out the hardware. The art is fantastic. The frame rate is consistent no matter what. The localization (where available) is spot on. The voice acting (where available) is so good you want to listen to it again and again. Finally, its accessible. This title has the ability to gain new converts to the genre, or at a base level, detractors of the genre have to respect it. This number is rarely given out, and if its considered, he/she should run it by their peers and editors after an extensive process.

I think we'll never reach that catch-all scoring system, since we all have different standards and tastes, but as long as the reviewer sticks to their review guide, and can back up their reason for said score in a logical, reasonable manner. it shouldn't be too much of a problem.

Ma ma sa, ma ma coo sa
Ma ma se, ma ma sa,
Ma ma coo sa

Offline vudu

  • You'd probably all be better off if I really were dead.
  • NWR Junior Ranger
  • Score: -19
    • View Profile
RE:How do you consider ratings?
« Reply #22 on: September 18, 2007, 10:29:30 AM »
IGN has both a number and a word rating (e.g. 9.5 & Incredible or 5.5 & Mediocre).  Why not just dump the numerical score and simply go with the word?  Good and Passable tell you more than a number does (which can be ambiguous and asinine at times).  If a game is a good game, say so, rather than give it a 7.5 and force your audience to figure out for themselves whether a 7.5 is a good score or an average score.
Why must all things be so bright? Why can things not appear only in hues of brown! I am so serious about this! Dull colors are the future! The next generation! I will never accept a world with such bright colors! It is far too childish! I will rage against your cheery palette with my last breath!

Offline Kairon

  • T_T
  • NWR Staff Pro
  • Score: 48
    • View Profile
RE: How do you consider ratings?
« Reply #23 on: September 18, 2007, 10:45:50 AM »
Yeah, like I previously stated, I believe far more in emotional benchmarks for reviews instead of numerical ones.
Carmine Red, Associate Editor

A glooming peace this morning with it brings;
The sun, for sorrow, will not show his head:
Go hence, to have more talk of these sad things;
Some shall be pardon'd, and some punished:
For never was a story of more woe
Than this of Sega and her Mashiro.

Offline King of Twitch

  • twitch.tv/zapr2k i live for this
  • Score: 141
    • View Profile
RE: How do you consider ratings?
« Reply #24 on: September 18, 2007, 10:48:01 AM »
You get emotional over everything though..
"I deem his stream to be supreme and highly esteem his Fortnite team!" - The Doritos Pope and his Mountain Dew Crew.