Author Topic: Making the Review Process Better  (Read 21703 times)

0 Members and 1 Guest are viewing this topic.

Offline Kairon

  • T_T
  • NWR Staff Pro
  • Score: 47
    • View Profile
Re: Making the Review Process Better
« Reply #100 on: January 12, 2009, 04:04:22 PM »
Using hypothetical scenarios to modify my review score, guessing what my emotional response would be if I were a different person, is significantly more arbitrary.

I think this is really what I have problems with in regards to qualifiers. Not only are you telling me whether I'll like the game and why, you're telling me how much I'd like it better if I was X, or if I was Y. It sort of assumes you can categorize and pidgeonhole and predict and commodotize readers. And it assumes that certain groups of people are homogonous and have the same tastes.

Just because I fall into category A doesn't mean I like X, or care less about Y, or care more about Z. And how DARE you make that assumption about me.

I'm not mad. &P I'm just prickling at the thought of how people like Nintendo owners, casual gamers, FEMALE gamers, and everyone who's ever been outside of the strictly defined "jhardcore gaming" group have been the subjects of prejudice for ages, and more and more as of late.
Carmine Red, Associate Editor

A glooming peace this morning with it brings;
The sun, for sorrow, will not show his head:
Go hence, to have more talk of these sad things;
Some shall be pardon'd, and some punished:
For never was a story of more woe
Than this of Sega and her Mashiro.

Offline KDR_11k

  • boring person
  • Score: 28
    • View Profile
Re: Making the Review Process Better
« Reply #101 on: January 12, 2009, 04:31:17 PM »
1. The score is only as arbitrary as your oppinion of the game. It's quite possible to say how much you enjoyed a game, just assign certain enjoyment levels to the different scores and if something falls between them use decimals to weight it. Qualifiers are mostly for things like declaring a game a tired rehash, that's of zero consequence for someone who didn't play the original and would like to know which to buy. Generally the qualifiers should be limited to pretty clear cut things, don't throw them on just because you can. E.g. a score for "casual" gamers would be completely misplaced on a review of Ikaruga.
2. I'm against a scale of buy-rent-avoid because it's impossible to shift (e.g. if I have a higher/lower purchase threshold) and doesn't have a "pricedrop to X" entry (which applies to nearly all games for me, another issue with low range scales since what I consider pricedrop territory would be retail for most and there's no differentiation in that range to tell my pricedrop games from my retail games). Also among the games that are worth buying there are still quality differences, some are more must-buy than others and a reader with a limited budget would not be able to determine a priority.

Generally there seems to be this advocating of less accurate or no scores but the quality range for games is there, it is perceptible and usually when someone asks for buying recommendations you will point out a few must-haves before the rest of the good stuff. If you just slap a 50 entry list of games you consider buyable in front of him he'll either be stumped or start with a random one (or the one with the brand he knows). You can sort a list for scores, you cannot sort a list for review impressions.

Offline NinGurl69 *huggles

  • HI I'M CRAZY
  • Score: 28
    • View Profile
    • Six Sided Video
Re: Making the Review Process Better
« Reply #102 on: January 12, 2009, 04:37:52 PM »
You just jumped the gun and shoved a list in front of him.

You should've first asked what types of games he likes.
:: Six Sided Video .com ~ Pietriots.com ::
PRO IS SERIOUS. GET SERIOUS.

Offline KDR_11k

  • boring person
  • Score: 28
    • View Profile
Re: Making the Review Process Better
« Reply #103 on: January 12, 2009, 04:54:17 PM »
What games he likes does not matter on the Wii, everyone gets shown the same list anyway.

Offline Justin Nation

  • Programmer, Senior Editor, Daddy
  • Score: 1
    • View Profile
    • MAMEiac Gaming
Re: Making the Review Process Better
« Reply #104 on: January 13, 2009, 12:06:53 PM »
Haha, this is why I love this topic. Regardless of where it goes seeing people who are passionate and engaging their minds about what is effective and what isn't.

To tell the truth I think what you would find at the end of all of this is that no matter what system you use or way you implement the scoring/lack of scoring/supplemented scoring much more rides on the quality of the reviewer than any other factor. Any of these systems, with the right reviewer, could work and be of benefit. How many people out there could write at the level to make it work regardless of format? OK, probably not too many. It is some hard stuff. I like to see the passion in here though of people asking the hard questions, thinking through their approach, and wanting it to be a good fit for the readers.
Justin Nation
Resident Dinosaur, Occasional Contributor, Daddy
MAMEiac Gaming
Administrator, The Nintendo Switch Fan Community (Facebook Group)
"People are stupid"

Offline KDR_11k

  • boring person
  • Score: 28
    • View Profile
Re: Making the Review Process Better
« Reply #105 on: January 13, 2009, 01:31:42 PM »
Well, yeah, most reviewers suck, that's why I call the score a necessary crutch.

Offline Justin Nation

  • Programmer, Senior Editor, Daddy
  • Score: 1
    • View Profile
    • MAMEiac Gaming
Re: Making the Review Process Better
« Reply #106 on: January 14, 2009, 09:43:27 AM »
Yeah, that's where my accountability thought with numeric scores comes in as well, even though I like what the score forces even with good reviewers. Your thoughts are great but in the end people deal in measurements... even if arbitrary. Watch a football game and the way they spot the ball and you definitely see the problem in action. Placement is arbitrary but the decision needs to be made and people have to live with that.
Justin Nation
Resident Dinosaur, Occasional Contributor, Daddy
MAMEiac Gaming
Administrator, The Nintendo Switch Fan Community (Facebook Group)
"People are stupid"

Offline NWR_insanolord

  • Rocket Fuel Malt Liquor....DAMN!
  • NWR Staff Pro
  • Score: -18987
    • View Profile
Re: Making the Review Process Better
« Reply #107 on: January 14, 2009, 10:53:45 AM »
If you think the placement of the ball in football is arbitrary you don't understand the game.
Insanolord is a terrible moderator.

J.P. Corbran
NWR Community Manager and Soccer Correspondent

Offline Justin Nation

  • Programmer, Senior Editor, Daddy
  • Score: 1
    • View Profile
    • MAMEiac Gaming
Re: Making the Review Process Better
« Reply #108 on: January 14, 2009, 07:27:25 PM »
Arbitrary in the sense of humans judge the entire thing. A human places the ball (prone to error), humans move and place the yard markers (prone to error), when they measure for first down they place the one end (prone to error), then they pull the other one to the end of the chain (tell me when you see them pulling out the slack, etc that isn't prone to error) to then eyeball whether the tip of the ball is past the marker. Now, the theory is that in the end it all balances out one way or another in terms of luck of placement or that the cumulative error-proneness often ends up making the end result more approximately correct than some of its components may be. Nonetheless, add that all up and ultimately all of the placements, ultimately all subject to human error (lets not even get into dudes throwing their hats away from their body and the like to approximate where something landed) and thus in that way arbitrary.

Technically, with things as they are, they could place something in the ball itself and you could begin to make things painfully precise... or take major steps towards it. Would that make the game better? No. But if accuracy was a concern they would do something about it.
Justin Nation
Resident Dinosaur, Occasional Contributor, Daddy
MAMEiac Gaming
Administrator, The Nintendo Switch Fan Community (Facebook Group)
"People are stupid"

Offline Stogi

  • The Stratos You Should All Try To Be Like
  • Score: 18
    • View Profile
Re: Making the Review Process Better
« Reply #109 on: January 14, 2009, 08:52:36 PM »
I think the only way to make the review process better is to write two reviews for the same game. One review would be directed towards the fans of the genre and videogames alike and it would include a score. It would get into the "hardcore" nitty gritty and talk about L33T topics such as framerate. In contrast, in the other review you would only depict your experiences; no scores or qualifiers. It would be a narrative; a slice of life so to speak. This way everyone wins (except you who has to write two reviews). Actually, you could split the reviews and the outcome would be just as beneficial. Still, you would then need two copies of the same game.
« Last Edit: January 14, 2009, 08:55:09 PM by Kashogi Y. Stogi »
black fairy tales are better at sports

Offline GoldenPhoenix

  • Now it's a party!
  • Score: 42
    • View Profile
Re: Making the Review Process Better
« Reply #110 on: January 20, 2009, 08:42:05 PM »
Actually I kind of like the IGN approach (along with others) where you have the main review along with another opinion.
Switch Friend Code: SW-4185-3173-1144

Offline vudu

  • You'd probably all be better off if I really were dead.
  • NWR Junior Ranger
  • Score: -19
    • View Profile
Re: Making the Review Process Better
« Reply #111 on: January 21, 2009, 02:12:04 PM »
Actually I kind of like the IGN approach (along with others) where you have the main review along with another opinion.

I proposed NWR adopt this method years ago and was shot down (I can't remember by who).  Maybe King Lindy will revisit the idea.
Why must all things be so bright? Why can things not appear only in hues of brown! I am so serious about this! Dull colors are the future! The next generation! I will never accept a world with such bright colors! It is far too childish! I will rage against your cheery palette with my last breath!

Offline Halbred

  • Staff Paleontologist, Ruiner of Worlds
  • NWR Staff
  • Score: 17
    • View Profile
    • When Pigs Fly Returns
Re: Making the Review Process Better
« Reply #112 on: January 21, 2009, 04:23:46 PM »
Not sure how that process would work. The vast majority of games we recieve from publishers are sent to ONE reviewer because we just get ONE copy. Now, in situations like Mario Kart Wii, where basically everybody on staff gets a copy, yeah, that's more feasable. But it definately can't be the norm. It's not like we all work in the same office and can just pass a game around.
This would be my PSN Trophy Card, but I guess I can't post HTML in my Signature. I'm the pixel spaceship, and I have nine Gold trophies.

Offline vudu

  • You'd probably all be better off if I really were dead.
  • NWR Junior Ranger
  • Score: -19
    • View Profile
Re: Making the Review Process Better
« Reply #113 on: January 21, 2009, 04:41:42 PM »
Why not?  Why can't you send someone else on staff the review copy once you're done with it?

Even if the "another opinion" section is added after the publication of the main review, it's still worthwhile provided it's a different take than on the original review.  And honestly, it doesn't even have to be added to the official review.  It might just be that the other reviewer adds his two cents in the talkback thread.

It just seems like it's something that's worth looking into.  I've seen several reviews on the site comment about disagreeing with certain aspects of a review that they didn't write, but they didn't disagree enough to be worth writing another full-blown review.  I think having an "another opinion" section would be a happy compromise.
Why must all things be so bright? Why can things not appear only in hues of brown! I am so serious about this! Dull colors are the future! The next generation! I will never accept a world with such bright colors! It is far too childish! I will rage against your cheery palette with my last breath!

Offline NWR_Neal

  • NWR Staff Pro
  • Score: 27
    • View Profile
Re: Making the Review Process Better
« Reply #114 on: January 21, 2009, 06:19:44 PM »
I think we kind of do that anyway in the talkback thread.
That whole "ship to another staffer" process would still take a lot of time, especially since we're doing this volunteer and shipping takes time.

Something doesn't sit well with me adding content to the review after the fact. I'd much rather have some sort of Afterthoughts in the blog or something. Maybe even review our own review or something like that. I recall Jonny saying something in an RFN about how he wouldn't have given Twilight Princess a 10 now. Hell, in my short time period as a staffer, there's some games that I might second guess my score after some months. For example, Madden 09 really grew on me throughout the football year.
Neal Ronaghan
Director, NWR

"Fungah! Foiled again!"

Offline NWR_Lindy

  • Famous Rapper
  • NWR Staff Pro
  • Score: 13
    • View Profile
Re: Making the Review Process Better
« Reply #115 on: January 22, 2009, 01:18:04 AM »
In an ideal world, having two opinions on every review would be great.  In reality, it's a logistical nightmare.  We just can't do it.

However, I like the idea of going back and revisiting old reviews, and would like to see that incorporated into something we do in the future.
Jon Lindemann
Contributing Editor, Nintendo World Report

My Game Backlog

Offline TheYoungerPlumber

  • Thy Rod and Staff
  • NWR Staff Pro
  • Score: 10
    • View Profile
    • Nintendo World Report
Re: Making the Review Process Better
« Reply #116 on: January 22, 2009, 04:45:42 AM »
Multiple reviews for a high-profile or controversial game are great. I have supported the idea of a less thorough, but still formal, second opinion format. Like secondary reviews, it would be used only when warranted. (Two reviews saying largely the same thing is usually unnecessary.)
::Michael "TYP" Cole
::Associate Editor
Nintendo World Report

"Only CHEATERS mess up!" -Waluigi

Offline Halbred

  • Staff Paleontologist, Ruiner of Worlds
  • NWR Staff
  • Score: 17
    • View Profile
    • When Pigs Fly Returns
Re: Making the Review Process Better
« Reply #117 on: January 22, 2009, 04:37:30 PM »
See? There's another problem with numbers. Even the same reviewer wouldn't give the same game the same score after the passage of time. So not only are reviews subjective as to the reviewer, they are subjective as to the time during which they are reviewed.
This would be my PSN Trophy Card, but I guess I can't post HTML in my Signature. I'm the pixel spaceship, and I have nine Gold trophies.

Offline Stogi

  • The Stratos You Should All Try To Be Like
  • Score: 18
    • View Profile
Re: Making the Review Process Better
« Reply #118 on: January 22, 2009, 04:57:17 PM »
It's also subjective to the number of games the reviewer has played as well as the class of those games.
black fairy tales are better at sports

Offline Justin Nation

  • Programmer, Senior Editor, Daddy
  • Score: 1
    • View Profile
    • MAMEiac Gaming
Re: Making the Review Process Better
« Reply #119 on: January 22, 2009, 08:44:00 PM »
See? There's another problem with numbers. Even the same reviewer wouldn't give the same game the same score after the passage of time. So not only are reviews subjective as to the reviewer, they are subjective as to the time during which they are reviewed.

Ah, but to say that you presume that only the numeric score would change over time and not the body of your opinion... which would be entirely an untrue assumption. If you've warmed to the game it would be reflected in your words, same as your numeric scores. If you've become jaded you'd think it would also enter into their review, not just reflected in the score.
Justin Nation
Resident Dinosaur, Occasional Contributor, Daddy
MAMEiac Gaming
Administrator, The Nintendo Switch Fan Community (Facebook Group)
"People are stupid"

Offline vudu

  • You'd probably all be better off if I really were dead.
  • NWR Junior Ranger
  • Score: -19
    • View Profile
Re: Making the Review Process Better
« Reply #120 on: January 25, 2009, 06:58:55 PM »
Recently (like six minutes ago), nron10 bumped his Madden 2009 review to mention that he still really enjoyed the game.

I argue:  Why stop there?  Since you're still playing the game six months later why not post a few paragraphs about your after-thoughts?  Do you still agree with the score you gave it?  Would you change anything about your review if you were writing it now instead of in August?
Why must all things be so bright? Why can things not appear only in hues of brown! I am so serious about this! Dull colors are the future! The next generation! I will never accept a world with such bright colors! It is far too childish! I will rage against your cheery palette with my last breath!

Offline NWR_Lindy

  • Famous Rapper
  • NWR Staff Pro
  • Score: 13
    • View Profile
Re: Making the Review Process Better
« Reply #121 on: January 25, 2009, 07:37:26 PM »
I like this idea.  I can't guarantee that it will be implemented any time soon (there's other stuff to take care of before this could even happen on the site), but it would be cool for us to have the option of doing this.
Jon Lindemann
Contributing Editor, Nintendo World Report

My Game Backlog

Offline NWR_Neal

  • NWR Staff Pro
  • Score: 27
    • View Profile
Re: Making the Review Process Better
« Reply #122 on: January 25, 2009, 07:48:01 PM »
vudu, why do you think I bumped it?
Like Lindy said, we've got other priorities but I still want to see if I can go anywhere with it.

I read over my review before bumping it and I surprised myself by agreeing with my original score and text. Eventhough I enjoy the hell out the game, it's still an 8 in my book.
That and Mario Super Sluggers are fantastic games that just needed a few things added to make them better. The most grevious error Sluggers had was the lack of a season mode; Madden has a lot of control hiccups. I still love playing both games.
Neal Ronaghan
Director, NWR

"Fungah! Foiled again!"