Topic: Making the Review Process Better

[Kairon]

Using hypothetical scenarios to modify my review score, guessing what my emotional response would be if I were a different person, is significantly more arbitrary.

I think this is really what I have problems with in regards to qualifiers. Not only are you telling me whether I'll like the game and why, you're telling me how much I'd like it better if I was X, or if I was Y. It sort of assumes you can categorize and pidgeonhole and predict and commodotize readers. And it assumes that certain groups of people are homogonous and have the same tastes.

Just because I fall into category A doesn't mean I like X, or care less about Y, or care more about Z. And how DARE you make that assumption about me.

I'm not mad. &P I'm just prickling at the thought of how people like Nintendo owners, casual gamers, FEMALE gamers, and everyone who's ever been outside of the strictly defined "jhardcore gaming" group have been the subjects of prejudice for ages, and more and more as of late.

[KDR_11k]

1. The score is only as arbitrary as your oppinion of the game. It's quite possible to say how much you enjoyed a game, just assign certain enjoyment levels to the different scores and if something falls between them use decimals to weight it. Qualifiers are mostly for things like declaring a game a tired rehash, that's of zero consequence for someone who didn't play the original and would like to know which to buy. Generally the qualifiers should be limited to pretty clear cut things, don't throw them on just because you can. E.g. a score for "casual" gamers would be completely misplaced on a review of Ikaruga.
2. I'm against a scale of buy-rent-avoid because it's impossible to shift (e.g. if I have a higher/lower purchase threshold) and doesn't have a "pricedrop to X" entry (which applies to nearly all games for me, another issue with low range scales since what I consider pricedrop territory would be retail for most and there's no differentiation in that range to tell my pricedrop games from my retail games). Also among the games that are worth buying there are still quality differences, some are more must-buy than others and a reader with a limited budget would not be able to determine a priority.

Generally there seems to be this advocating of less accurate or no scores but the quality range for games is there, it is perceptible and usually when someone asks for buying recommendations you will point out a few must-haves before the rest of the good stuff. If you just slap a 50 entry list of games you consider buyable in front of him he'll either be stumped or start with a random one (or the one with the brand he knows). You can sort a list for scores, you cannot sort a list for review impressions.

[NinGurl69 *huggles]

You just jumped the gun and shoved a list in front of him.

You should've first asked what types of games he likes.

[KDR_11k]

What games he likes does not matter on the Wii, everyone gets shown the same list anyway.

[Justin Nation]

Haha, this is why I love this topic. Regardless of where it goes seeing people who are passionate and engaging their minds about what is effective and what isn't.

To tell the truth I think what you would find at the end of all of this is that no matter what system you use or way you implement the scoring/lack of scoring/supplemented scoring much more rides on the quality of the reviewer than any other factor. Any of these systems, with the right reviewer, could work and be of benefit. How many people out there could write at the level to make it work regardless of format? OK, probably not too many. It is some hard stuff. I like to see the passion in here though of people asking the hard questions, thinking through their approach, and wanting it to be a good fit for the readers.

[KDR_11k]

Well, yeah, most reviewers suck, that's why I call the score a necessary crutch.

[Justin Nation]

Yeah, that's where my accountability thought with numeric scores comes in as well, even though I like what the score forces even with good reviewers. Your thoughts are great but in the end people deal in measurements... even if arbitrary. Watch a football game and the way they spot the ball and you definitely see the problem in action. Placement is arbitrary but the decision needs to be made and people have to live with that.

[NWR_insanolord]

If you think the placement of the ball in football is arbitrary you don't understand the game.

[Justin Nation]

Arbitrary in the sense of humans judge the entire thing. A human places the ball (prone to error), humans move and place the yard markers (prone to error), when they measure for first down they place the one end (prone to error), then they pull the other one to the end of the chain (tell me when you see them pulling out the slack, etc that isn't prone to error) to then eyeball whether the tip of the ball is past the marker. Now, the theory is that in the end it all balances out one way or another in terms of luck of placement or that the cumulative error-proneness often ends up making the end result more approximately correct than some of its components may be. Nonetheless, add that all up and ultimately all of the placements, ultimately all subject to human error (lets not even get into dudes throwing their hats away from their body and the like to approximate where something landed) and thus in that way arbitrary.

Technically, with things as they are, they could place something in the ball itself and you could begin to make things painfully precise... or take major steps towards it. Would that make the game better? No. But if accuracy was a concern they would do something about it.

[Stogi]

I think the only way to make the review process better is to write two reviews for the same game. One review would be directed towards the fans of the genre and videogames alike and it would include a score. It would get into the "hardcore" nitty gritty and talk about L33T topics such as framerate. In contrast, in the other review you would only depict your experiences; no scores or qualifiers. It would be a narrative; a slice of life so to speak. This way everyone wins (except you who has to write two reviews). Actually, you could split the reviews and the outcome would be just as beneficial. Still, you would then need two copies of the same game.

[GoldenPhoenix]

Actually I kind of like the IGN approach (along with others) where you have the main review along with another opinion.

[vudu]

Actually I kind of like the IGN approach (along with others) where you have the main review along with another opinion.

I proposed NWR adopt this method years ago and was shot down (I can't remember by who).  Maybe King Lindy will revisit the idea.

[Halbred]

Not sure how that process would work. The vast majority of games we recieve from publishers are sent to ONE reviewer because we just get ONE copy. Now, in situations like Mario Kart Wii, where basically everybody on staff gets a copy, yeah, that's more feasable. But it definately can't be the norm. It's not like we all work in the same office and can just pass a game around.

[vudu]

Why not?  Why can't you send someone else on staff the review copy once you're done with it?

Even if the "another opinion" section is added after the publication of the main review, it's still worthwhile provided it's a different take than on the original review.  And honestly, it doesn't even have to be added to the official review.  It might just be that the other reviewer adds his two cents in the talkback thread.

It just seems like it's something that's worth looking into.  I've seen several reviews on the site comment about disagreeing with certain aspects of a review that they didn't write, but they didn't disagree enough to be worth writing another full-blown review.  I think having an "another opinion" section would be a happy compromise.

[NWR_Neal]

I think we kind of do that anyway in the talkback thread.
That whole "ship to another staffer" process would still take a lot of time, especially since we're doing this volunteer and shipping takes time.

Something doesn't sit well with me adding content to the review after the fact. I'd much rather have some sort of Afterthoughts in the blog or something. Maybe even review our own review or something like that. I recall Jonny saying something in an RFN about how he wouldn't have given Twilight Princess a 10 now. Hell, in my short time period as a staffer, there's some games that I might second guess my score after some months. For example, Madden 09 really grew on me throughout the football year.

[NWR_Lindy]

In an ideal world, having two opinions on every review would be great.  In reality, it's a logistical nightmare.  We just can't do it.

However, I like the idea of going back and revisiting old reviews, and would like to see that incorporated into something we do in the future.

[TheYoungerPlumber]

Multiple reviews for a high-profile or controversial game are great. I have supported the idea of a less thorough, but still formal, second opinion format. Like secondary reviews, it would be used only when warranted. (Two reviews saying largely the same thing is usually unnecessary.)

[Halbred]

See? There's another problem with numbers. Even the same reviewer wouldn't give the same game the same score after the passage of time. So not only are reviews subjective as to the reviewer, they are subjective as to the time during which they are reviewed.

[Stogi]

It's also subjective to the number of games the reviewer has played as well as the class of those games.

[Justin Nation]

See? There's another problem with numbers. Even the same reviewer wouldn't give the same game the same score after the passage of time. So not only are reviews subjective as to the reviewer, they are subjective as to the time during which they are reviewed.

Ah, but to say that you presume that only the numeric score would change over time and not the body of your opinion... which would be entirely an untrue assumption. If you've warmed to the game it would be reflected in your words, same as your numeric scores. If you've become jaded you'd think it would also enter into their review, not just reflected in the score.

[vudu]

Recently (like six minutes ago), nron10 bumped his Madden 2009 review to mention that he still really enjoyed the game.

I argue:  Why stop there?  Since you're still playing the game six months later why not post a few paragraphs about your after-thoughts?  Do you still agree with the score you gave it?  Would you change anything about your review if you were writing it now instead of in August?

[NWR_Lindy]

I like this idea.  I can't guarantee that it will be implemented any time soon (there's other stuff to take care of before this could even happen on the site), but it would be cool for us to have the option of doing this.

[NWR_Neal]

vudu, why do you think I bumped it?
Like Lindy said, we've got other priorities but I still want to see if I can go anywhere with it.

I read over my review before bumping it and I surprised myself by agreeing with my original score and text. Eventhough I enjoy the hell out the game, it's still an 8 in my book.
That and Mario Super Sluggers are fantastic games that just needed a few things added to make them better. The most grevious error Sluggers had was the lack of a season mode; Madden has a lot of control hiccups. I still love playing both games.