Topic: Making the Review Process Better

[Kairon]

still get lots of fun out of games like Carnival Games, Wii Music, Animal Crossing, and various other more casual focused games.

You are being so MEAN to Animal Crossing by putting it with that company.

Yeah, GP. See? Even Ian agrees. Animal Crossing is a CORE game. &P

BTW your review on PH was the best.

...must...resist...arghing....AARRGGHHHH!!!

[Halbred]

Yup! I remember GP was my sole defender in that Talkback thread. Many thanks, GP.

Screw you, Kairon! Phantom Hourglass can suck it! ;-)

[Justin Nation]

Perhaps it would be best to have the additional criteria/exceptions open-ended. That would allow for Carnival Games to have an exception for people who legitimately enjoy boardwalk/midway games... which my relatives do and that would explain a lot of their affection for it despite its shortcomings. I think what I dislike about it is how poorly it is implemented. It really was made as, or is only generally as well made as, shovelware. It sucks people in easily for its theme and being unique but not because it deserves the attention. Shallowness I could excuse but the control for many of the games is just amazingly wonky and poor. To me that's the shame of it, though I agree for that segment the hate wouldn't likely be enough to dissuade many people.

[NinGurl69 *huggles]

If you're gonna write, figure out who your audience is.  (primary, secondary, tertiary)

[GoldenPhoenix]

I find the controls in Carnival Games to be surprisingly good for the most part. So take that.

[Justin Nation]

Perhaps they're good if you want them to be true to the carny rigged tradition of games...

[Kairon]

Perhaps they're good if you want them to be true to the carny rigged tradition of games...

Does that mean the game should be lauded as a "simulation?" &P

[Justin Nation]

True... so maybe a clarification for "People who enjoy the frustration of skill having little correlation to success" +7!

[Nick DiMola]

I believe more important than coming up with more and more complicated ways to reach different audiences, is writing your review with personal experiences in mind, and discussing what you felt worked and didn't work while playing the game and why.

Ultimately, these experiences will serve as a guideline for someone reading your review. Regardless of your personal opinion of the game, readers will be able to take away tangible information about the game and how it actually plays and determine if those mechanics and concepts suit their preferences.

Of course, the reviewer also needs to avoid falling into the trap of becoming too biased, and downplaying positive aspects of the game simply because one aspect of the game truly bothered them. The final score, and any affected subsections of the game should only illustrate that frustration

[NinGurl69 *huggles]

nice thoughts

^ too bad that sort of stuff just doesn't happen with today's "media."

[KDR_11k]

I think we had an argument like that before where I suggested allowing multiple scores with qualifiers for a review that are picked depending on what the reviewer wants to express but people kept saying we should have a fixed set of scores that all reviews should use instead of each review using its own set.

[Justin Nation]

I think we had an argument like that before where I suggested allowing multiple scores with qualifiers for a review that are picked depending on what the reviewer wants to express but people kept saying we should have a fixed set of scores that all reviews should use instead of each review using its own set.

I think the people who disagreed with you are just limiting themselves unnecessarily and pigeonholing all reviews to only be able to count against one consistent set of potential qualifiers would ultimately weaken the point of having them at all. Sort of like how the Wii has suddenly legitimized the "casual audience" as distinct from mainstream and hardcore... where before it came along you wouldn't have had a need. Things change, times change, context changes.

A good example specific to only a handful of games but that would be an excellent qualifier if someone would offer it (mainly because people could truly find it beneficial): A qualifier on a review of something like Guitar Hero Aerosmith/Metallica/etc for fans of the band specifically. I rented it thinking I would love it because I love a lot of Aerosmith's music. Thing is, the mix of what they offered was mostly stuff that I wasn't stoked to play. So I could even see two qualifiers or more.

Fans of Aerosmith's entire catalog: + 2 (they actually did span a great deal of their catalog
Fans of Aerosmith's more current music: -1 (a lot of older stuff newer fans may not even recognize)
Guitar Hero fans neutral to Aerosmith: -2 (aside from the music itself it wasn't terribly compelling compared to the broader GH3)

Now, these qualifiers would need to be accompanied by a very clear and concise paragraph or two for the sake of really knowing why but I think it is a great example of qualifiers that could certainly be relevant at a game level. Just a matter of people being creative and possibly even... having some fun with it. Nah, reviewers are to be taken seriously, no room for that.

[NWR_Neal]

I finally caught up and read this entire thread.

I really like Justin's ideas of qualifiers because they account for whatever personal biases exist in reviews.

For example, if I reviewed GH:Aerosmith/Metallica, I would hold a bias against both bands because I'm not a huge fan of either band. However, if you were a big fan of either band, the games should be almost instapurchase.

Likewise with my Sonic Unleashed review (ugh, I don't know if I should even bring it up). For big fans of the hedgehog, that game would have a higher score.

I could go on with some more examples, but I'll just end it by saying I really like the idea of qualifiers.

[Halbred]

But again, qualifiers to the score aren't necessary if you're a good writer. Score the game based on its own merits, but use the body of the review to tell the reader about what audience the game does (or does not) play to.

[NWR_Lindy]

That's why I like Reviewer Bios.  Something that tells the background of the reviewer and what their preferences are.  Everybody has a couple of genres that they know front-to-back, and everybody has a couple that they don't know as well.  You can point this out in your review, sure, but if somebody looks at your "favorite games" list and it matches theirs, they know that they'll likely have a similar opinion of the game as you do.

ASIDE - We have bios here at NWR, but they're buried and never updated.  Our future plans include bringing these bios more front-and-center.

[NWR_Neal]

I try to update my bio...

[GoldenPhoenix]

I have a big problem with personality dictating a review's merit, just because someone may prefer a certain genre does NOT mean they can't be objective with another genre. But if a biography is created people will tend to pigeon hole them and anything outside of their preferred genre will be pigeonholed as them not knowing what they are talking about.

[Justin Nation]

Sure the body of the review can fill in perspective of how different groups may feel but still, the failure to quantify how that impacts things makes for likely ambiguity. Take the typical Pro/Con most reviews tend to have. Most of the time the number of each is somewhat the same but assuming by number they even each other out would be a mistake. Truthfully just one con could blow out a laundry list of pros. In the end while people may bemoan quantifying their score it is an exercise in accountability. There is no hiding behind "Well, what I meant was". You're putting it out there and as a matter of fact that score can then be put up against others to paint a picture of where the game falls. Now, this is definitely where aggregators fall on their face. Fact is, my critical 6 could make sense when considered against other reviews and another person's 6 could actually be a big anomaly in terms of how they normally see the world. To the aggregator they're all the same though. Some power in the average of 100 reviews but all reviews aren't created equal.

There is no perfect system. I just like the idea of the work being the responsibility of the reviewer. Go to some lengths to make your feelings plain. Take the time to try to put yourself into another perspective to evaluate the game in a different light, accounting for another person's legitimate perspective.

I think the Aerosmith or Sonic game reviews stand as great examples where the qualifiers help. If you give the game a bad review is it because of the game itself, the license, the lack of originality? Even for fans of the band the game could be terrible if the mechanics of the game are poor. Even fans of the music would be entitled to clearly know this rather than just flatly saying fans of the band will pick it up anyway. No, they'll pick it up anyway because nobody would address the shortcomings of the game in a way that fans of the band wouldn't disregard as someone just not liking the band. Again, I liked the game, I just think the situation it falls into illustrates the problem well.

Trust me, as verbose as I am (as you can see), there is easily a part of me that believes that the work a reviewer puts into their choices of words and turns of phrase should be properly appreciated. Writers put nuance into the reviews to be appreciated and people who short-cut to the score can miss a lot of quality points. Hell, the format of reviews for this very site, where the body of the review only is on the first page and numeric scores need to be clicked through to... it is something I collaborated on. Thing is, as a reviewer the job is to serve everyone, even the chronically lazy or just those in a hurry. So there is something to be said for taking the time to crafting consistent and clear condensed views. Having written in real print I can tell you a full review of a game in 250 words or less is a powerful exercise. I enjoy going the other direction (obviously) but perhaps the best test is to put equal time and effort into both. Qualifiers would just be the direct paragraph and score offset guided right to a legitimate audience, doing them a greater service than the "for the masses" review that addresses both everyone and no one.

[NWR_insanolord]

Giving a game a numerical score is arbitrary enough, giving it a score based on how you think you might feel if you had someone else's opinions is crazy and getting close to Game Informer Paper Mario territory. Halbred's right, you just have to write well enough that someone reading the review could infer from the text if they'd like it more or less than the reviewer.

[KDR_11k]

I suggested qualifiers before because I've seen reviews where the text spelt them out (e.g. "add 10% to the rating if you like X" or in some cases "our scores are based on the OOTB version, with the first patch add x% to the rating").

[Stogi]

I agree.

[Kairon]

Giving a game a numerical score is arbitrary enough, giving it a score based on how you think you might feel if you had someone else's opinions is crazy and getting close to Game Informer Paper Mario territory. Halbred's right, you just have to write well enough that someone reading the review could infer from the text if they'd like it more or less than the reviewer.

Yeah, with numerical scores as arbitrary as they already are, I'm not convinced that we should jump even deeper down the rabit whole with qualifiers.

This is an interesting topic, and I know that for some reason I feel uncomfortable with qualifiers like "add 1 point if X or take away 1 point if Y," but whether that's for a good reason or not, or for any reason at all, I still haven't figured out.

[Justin Nation]

Giving a game a numerical score is arbitrary enough, giving it a score based on how you think you might feel if you had someone else's opinions is crazy and getting close to Game Informer Paper Mario territory. Halbred's right, you just have to write well enough that someone reading the review could infer from the text if they'd like it more or less than the reviewer.

I think part of the reason for your hang-up with qualifiers is at the core of the first statement, that numeric scores are arbitrary. Not given thought perhaps but I would argue that a review without a numeric score on it would be much more arbitrary than one with it. I'd say either as a stand-alone could possibly qualify but if all you have is text for a review without any score whatsoever unless you write War and Peace you will fail to have any guarantee of your message getting through loud a clear. Just the use of the word "infer" gets you into highly treacherous territory. I suppose email is then to you a crystal clear mechanism for the exchange of thought and feeling? Couldn't you detect I was sarcastic or kidding or deadly serious or heartbroken just from a jumble of words I assembled with no other indicators for you to go on?

Perhaps my numeric scoring and qualifiers are then the emoticons on the written review. By simply putting down that arbitrary number you convey a great deal of hard information in a definitive way simple words could never hope to cleanly deliver. By that score you know what company the game keeps. By seeing the breakdown scores you know roughly what to expect in terms of sound, visuals, control, everything. Sure, you could waste a lot of words trying to get it across but words in that case are inefficient and, if anything, even less reliable a means of clearly communicating with everyone than even that arbitrary number that is often given more weight than it deserves.

The score is a necessary evil because print and true nuance do not go hand in hand. I could be a poor deliverer, you could be a poor receiver, we could have different societal or experiential perspectives that alter the meaning of what you or I would think are clear and concrete words. Inference lacking a personal connection, a great deal of previous exposure, or some other more concrete factor is doomed to losing the message.

Realize, the qualifier isn't necessarily just an excuse to tack on more points or scores or grades. It is an opportunity to clarify with a numeric changer complemented by text that explains the change. So rather than someone trying to infer my thoughts from my scoring or my scoring from my thoughts I could provide both and attempt to get something that more consistently approaches clarity.

[Kairon]

So rather than someone trying to infer my thoughts from my scoring or my scoring from my thoughts I could provide both and attempt to get something that more consistently approaches clarity.

Actually, I think this doesn't increase the accuracy (how true their understanding is) of someone viewing your review at all. It may, however, improve the precision (how consistent their understanding is) of their evaluation of your evaluation. Or not. *boggles*

I also agree that scores are necessary. But I also think there's a lot they don't leave said. I believe that the numerical system lacks the room for personal emotional evaluation, which is a very big reason why I believe werespond well to the rent-buy-avoid system (which clearly defines the expected satisfaction of a consumer's monetary investment).

I've said this before and I've said it again, I loved the SF Chronicles way of reviewing movies back when I read the paper:
-Guy jumping out of his seat
-guy clapping
-guy watching
-guy sleeping
-guy not there

That gives not only defined a clear multi-tiered structure for movies of different values, but it also gave clear and immediate emotional touchpoints. It worked even if you disagreed with a reviewer, because we're all familiar with being ready to fall asleep in a movie that someone else is watching with rapt attention. Finally, it gives a little wiggle room to the score too: the movie is roughly this good, roughly the same as another movie, it's up to you to decide which you like better. The truth is that consumers actually translate point scores to that vagueness as well, not caring how much something is over 80 in metacritic, just that it is over 80.

[NWR_insanolord]

You're right that emotions aren't conveyed well in text and that the score does better in that regard. I'm not against game reviews having scores, but you have to admit that review scores are, at least to some extent, arbitrary. Using hypothetical scenarios to modify my review score, guessing what my emotional response would be if I were a different person, is significantly more arbitrary. I think the qualifiers are a good idea, but tying them to the score is a bad idea. Guitar Hero Aerosmith has been brought up as a demonstration for the qualifiers, but if you read NWR's review of the game you'll see that the reviewer pointed out more than once that someone who's a fan of the band will have a better time with it. Making a separate part at the end of the review where the reviewer gives reasons why you might like the game more or less than he or she did isn't a bad idea, but as I said if the reviewer writes well enough and illustrates his or her points well enough you could get that information from the text of the review.