Topic: Making the Review Process Better

[Ian Sane]

I think a site like GameRankings is really good for giving me somewhat of an idea of what games are worth checking out.  You can't just look at the average score though but the range of scores.  If a game gets unanimously bad scores I think it's safe to stay away from it.  If it gets unanimously good scores I think I ought to give it a shot.  If the scores are all over the place then I'm likely going to read the individual reviews and get an idea of whether I would like it or not.

Regarding the letter grade system one of my favourite sites is VideoGameCritic.net.  Everything is reviewed by one guy who has been gaming since the Atari days and has every system and is a serious game collector.  Each game he gives a paragraph about it and a letter grade.  I mostly like his site because it also covers a lot of retro games and when you're collecting for an old system it's a good reference to see what's worthwhile and what isn't (I suppose it would useful for the VC as well).  But the guy has something a lot of reviewers don't: perspective.  I remember years ago someone who worked for IGN's Playstation page was talking about how Final Fantasy VII got him into games.  He was PAID to review Playstation games and his gaming experience didn't even cover the first two years of the console he reviews for?  How often do you read a review where they're crapping on some game because it's 2D or it's a shmup?  That reviewer doesn't remember when games like that were current.  He classifies them as old and calls it a day.  The guy I read has followed games forever so he isn't going to crap on a game with "old school" design because he still plays those games on his old consoles.  At the same time he likes modern game design as well.

I think however the letter grade he uses is arbitrary.  He could just as easly use numbers, I just like his format and feel he has credibility.  One thing he also does is have a guest reviews RPGs because he doesn't care for the genre.  I like that he will admit that.  Not every genre is for everyone so if a reviewer clearly isn't really into a genre they should get someone else to look at it.

On that note, and I'm sure someone will crap on me for this, EGM's review crew is not a bad way to do it.  Do they even still do that?  Anyway, back when I used to read videogame mags they had four guys who each favoured certain genres more than others each give a score out of ten and I think, at least in theory, that's a great way to do it.  You talk about some people liking Phantom Hourglass and others not.  Well that's how you address that.

One thing you should never do though is just weigh a game's value as the sum of its parts.  The score should never just be an indication on how well made the game is.  Some games with no bugs, beautiful graphics and sound, and precise controls still are boring as hell while some slightly imperfect games are a blast to play.  You have to think like a friend giving his opinion of a game and not some beta tester checking off a list.

[NWR_insanolord]

I also liked EGM's way of doing reviews, but it's kind of obsolete now. It was great when all you had to go on was magazines and maybe one or two web sites, it was an easy way of getting multiple opinions on a game in one place. Now, though, you can go to GameRankings or MetaCritic and get a dozen or more reviews on any game just as easily. I also agree with Ian (and that and GP agreeing with Lindy mean the apocalypse is just days away) that games can be better (or worse) than the sum of their parts and the final score has to reflect that and not just be based on the game's technical merits. A review without opinion, if it were even possible, would be useless.

[KDR_11k]

I remember years ago someone who worked for IGN's Playstation page was talking about how Final Fantasy VII got him into games.  He was PAID to review Playstation games and his gaming experience didn't even cover the first two years of the console he reviews for?

Why not? Hegives the perspective of someone who got into gaming with FF7 which I think a lot of people did. Would it be preposterous to recruit someone who started gaming with the Wii in order to review Wii games? Does being used to the idiosyncrasies of an older time really make you a better reviewer? From what I see those old preconceptions tend to get in the way whenever a paradigm shift occurs because the oldies remain locked into their old paradigm and judge everything by the old values (and complain that a Wii game isn't HD). Old gamers have their perspective but new gamers have a different perspective and without reviews from that perspective you're not going to get in touch with the new gamers.

I agree with your final point. Reviewers gave EDF scores in the 70s while still saying the game is an insane amount of fun. I think one review gave a 10/10 (or whatever their scale was) despite all the technical flaws and such because they really don't matter.

[Halbred]

Honestly, I think reviewing games requires some knowledge and appreciation of the medium's history.

[Ian Sane]

Why not? Hegives the perspective of someone who got into gaming with FF7 which I think a lot of people did. Would it be preposterous to recruit someone who started gaming with the Wii in order to review Wii games? Does being used to the idiosyncrasies of an older time really make you a better reviewer?

I think a reviewer should be very knowledgable of what he is reviewing.  The first movie you ever saw was at one point the best movie you ever saw.  Ignorance can have a significant effect on your opinion.  Now videogames are old enough now that it's harder to have lived though all of it.  With books and music and movies you CAN'T have lived that long.  It's impossible to have that experience.  But you should be well informed of what else is out there.

Is it not idiotic when reviewers praise Halo as the greatest FPS they've ever played?  Even if you stick with just consoles Goldeneye and Perfect Dark were doing stuff that Halo "innovated" years before it came out.  But ignorance of the past makes it out to be a better game than it is.  I want an informed opinion.  You have to have some knowledge of the subject matter to give that to me.

Regarding the Wii thing, well maybe if the audience of the site was strict the blue ocean group for which the Wii is their first real intro to videogames.  In that case it would fit.  But a videogame site like IGN?  Totally inappropriate.  The audience is experienced gamers and if your opinion has no perspective it isn't going to be of any use to them.

[KDR_11k]

Careful, just because the features have been done before doesn't meant there's nothing good about a game. A game without new features can still deliver a combination that feels new or just plain better than what came before it. Also I think quite a few people never had an N64.

Of course the audience for most review sites is experienced gamers, anyone else would get only worthless results from them so why would they bother going there?

[GoldenPhoenix]

The first movie you ever saw was at one point the best movie you ever saw.

Doesn't mean it was a good movie. Personally I think in some ways being an older Retro gamer can be a detriment in that you have built up biases from your nostalgia for the "classics" and really cannot connect all that well with a newer generation of gamers, most of which would think most Atari games are terrible and shallow.

[Ian Sane]

Careful, just because the features have been done before doesn't meant there's nothing good about a game. A game without new features can still deliver a combination that feels new or just plain better than what came before it. Also I think quite a few people never had an N64.

I didn't really word my example well.  When Halo came out many reviewers were busting a nut over how it revolutionized console first person shooters and was so innovative when the only person who would ever think that is an ignorant one.  And the N64 isn't some super obscure console like the Jaguar.  A lot of people had it and any reviewer worth a sh!t should have come familiarity with it's biggest games.

Of course the audience for most review sites is experienced gamers, anyone else would get only worthless results from them so why would they bother going there?

I don't see your point.  Is this bad?  Should they change their format to accomodate a completely different audience?  If non-gamers want their videogame review site then they can make it.  Nothing stopping them in theory.

[Halbred]

To be honest, Halo did rework the traditional FPS. It added objective-based gameplay, a conservative two-gun system (some like it, some don't), the genre's first "superstar," and unique multiplayer options. Before Halo, Goldeneye revolutionized a genre that had, until that point, been defined primarily by Doom.

You can argue Halo's overall merits, but I don't think there's much of a question that it gave the FPS genre a shot in the arm in a lot of ways.

I think the FPS genre is conversative ANYWAY. In my mind, there have only been three or four really groundbreaking FPS games: Wolfenstein 3D, Doom, Goldeneye, and Halo. But that's just me.

[Nick DiMola]

Zach, you're definitely forgetting Half Life. That game provided the first objective based gameplay, superstar and even unique multiplayer options. Arguably, the only major innovation Halo brought was feasible vehicle based gameplay, but even Tribes beat it to the punch on that one. This is all beside the point, though.

I see Ian's point overall, and I don't really have a problem with what he is saying, though, I wouldn't want every reviewers to be on the same page in terms of their historical knowledge and experience. Varying opinions give people the ability to identify with particular people to get their ideal review. Without people who are wet behind the ears, a number of younger gamers wouldn't be served particularly well, because us dinosaurs have very different expectations of games.

[Halbred]

Yeah, I totally blanked on Half-Life. I'd argue with you about Tribes, though. The concept was there, but the execution was...questionable at times. Let's say this: Halo brought the FPS to the console gamepad in a way that hadn't been done *as well* before.

And I think I'm just biased against the gamers who were raised on the PS2. :-) I'm that crotchety old gamer on the porch, yelling that kids today have it easy. Back in MY day, we didn't even HAVE save points! Some games didn't even have save SLOTS! You kids with your fancy respawns and online play...

My god...I've become Cranky Kong!

[GoldenPhoenix]

Yeah, I totally blanked on Half-Life. I'd argue with you about Tribes, though. The concept was there, but the execution was...questionable at times. Let's say this: Halo brought the FPS to the console gamepad in a way that hadn't been done *as well* before.

And I think I'm just biased against the gamers who were raised on the PS2. :-) I'm that crotchety old gamer on the porch, yelling that kids today have it easy. Back in MY day, we didn't even HAVE save points! Some games didn't even have save SLOTS! You kids with your fancy respawns and online play...

My god...I've become Cranky Kong!

Halo was archaic compared to PC FPS games when it came out. The game did nothing new or innovative beyond the rechargeable shield (Which may have been done before). Vehicles were done in Codename Eagle as well, the predecessor to the Battlefield series. Not to mention that Goldeneye had a vehicle based mission with the tank where you had full control over it. So Halo is nothing more than a cut and paste job from games before (though it was polished) and since console gamers at the time were not that experienced with the PC they ate it up. It is funny because if Halo had come out on the PC instead of the Xbox you probably would not hear of the game, and if you did it would be a minor title.

To bring this topic back on track, I think this is a good example of how flawed the experience argument is. What if a person is more experienced in FPS games on the PC? Should their opinion matter more than someone who really only has console FPS experience? What about other genres done on consoles and PC? How do you approach them?

[NWR_Lindy]

Halo was archaic compared to PC FPS games when it came out. The game did nothing new or innovative beyond the rechargeable shield (Which may have been done before). Vehicles were done in Codename Eagle as well, the predecessor to the Battlefield series. Not to mention that Goldeneye had a vehicle based mission with the tank where you had full control over it. So Halo is nothing more than a cut and paste job from games before (though it was polished) and since console gamers at the time were not that experienced with the PC they ate it up. It is funny because if Halo had come out on the PC instead of the Xbox you probably would not hear of the game, and if you did it would be a minor title.

This is getting ridiculous.  I actually agree with GP on all of these points.

[Kairon]

Lindy and GP in the same boat? Good god what is the world coming to...

[NinGurl69 *huggles]

It'll probably sink.

[Kairon]

To bring this topic back on track, I think this is a good example of how flawed the experience argument is. What if a person is more experienced in FPS games on the PC? Should their opinion matter more than someone who really only has console FPS experience? What about other genres done on consoles and PC? How do you approach them?

I wouldn't want every reviewers to be on the same page in terms of their historical knowledge and experience. Varying opinions give people the ability to identify with particular people to get their ideal review. Without people who are wet behind the ears, a number of younger gamers wouldn't be served particularly well, because us dinosaurs have very different expectations of games.

I agree with these sentiments. Not to mention, there are TONS of differences even among people with plenty of experience. NWR alone should be proof of how longtime experienced gamers can have so many different takes on what comprises a quality game.

[Halbred]

Geeze, I've never heard of Codename Eagle. Now I feel bad. GP is right, and I agree with Kairon. Don't give me FPS games! Unless they're fun, like Call of Duty 4 (DS).

[GoldenPhoenix]

I always try to make Halbred feel bad.

Another game that was significant in regards to opening up the genre was Deus Ex as well (with a cool protagonist). That is one take on the genre that I hope to see more of, where you have a pretty wide open world, or at least ways of completing various tasks.

Though I am not a fan of the trend, I find it interesting now that it seems the FPS genre's direction is being dictated more and more by consoles. Though in some cases it turned out pretty good (Far Cry 2). So maybe in a few years the PC won't be the benchmark anymore for FPS because developers are focusing on consoles first and then porting it up.

[Plugabugz]

I didn't make any decent suggestioons coz i was hoping for Lord Sir James Jones would review Pokemon Dungeon on the podcast again for me to thoroughly enjoy

[Stogi]

I just read the FP and interestingly enough, Kairon and I had this exact same conversation a couple weeks back. What it amounted to was that we both agreed that unless you have personally reviewed both games, there is no congruency between scores.  Thus, it is basically pointless to give out scores if you have multiple reviewers.

[KDR_11k]

Halo may have been a polished cut-n-paste job. So was WoW. Some genres just need polish.

Ian, telling them to make their own publication is a dangerous thing for a manager of an existing publication. It's like a console maker saying "let them make their own console". The whole goal of a disruption is to grab markets that the competition will willingly surrender and then make them surrender more and more until there's nothing left. The manager of a publication should instead say "hey, there's a ton of people who aren't reading our publication, how can we get to them?".

[GoldenPhoenix]

Halo may have been a polished cut-n-paste job. So was WoW. Some genres just need polish.

That is true except it isn't like Halo was the only polished FPS game, especially compared to the PC. It would be like polishing Tomb  Raider 1 (which they did) even though the genre has advanced beyond it.

[Stogi]

Regarding the letter grade system one of my favourite sites is VideoGameCritic.net.

He gave WiiMusic an F. No wonder you like this guy.

[KDR_11k]

Well, the console market is for all the casual gamers out there, the PC is too hardcore for them .

[Kairon]

unless you have personally reviewed both games, there is no congruency between scores.  Thus, it is basically pointless to give out scores if you have multiple reviewers.

Of course, to have one person review everything is highly inpractical and serves little purpose for those who disagree with their stances on a basic level... would you like...say, Golden Phoenix, to review every single game for NWR Stogi?

Heck, and even then, a person might change their mind or mood or who knows what.

Take, for example, my movies collection. I have a whole range of movies of all different genres. Some of them I will readily admit are better than others. But the "best movies" aren't necessarily the ones I always pull out to watch. If I think a movie is worth four stars, but watch a two star movie way more often, does that make my personal evaluation of their respective qualities invalid? Or is quality of movie a separate issue from how much utility you can get from it?

Should a "quality rating" refer to end-user utility? To artistic value? To a lack of defects? Should it account for innovation? Uniqueness? A comparison to other efforts of its sort? Should it account for mood, taste, temperament, politics, social atmosphere, timeliness? Heck, should contributing factors like packaging or special features play a role? Say yes to more than one of these and already the equation for evaluating something becomes inherently complex and inherently unstable, especially over any extended period of time, even for just one person.