Page 1 of 7

Dubious Ratings

Posted: Wed Oct 28, 2009 2:28 am
by Baak
<rant>

I've had two ratings recently from Killery who never replies to my PM's (here or on MariusNet) nor to a post I put on the Magma site. In the PM's I basically asked him to please explain where his bad reviews came from and why he would take the time to post them when other people obviously like the maps (based on # of d/l's).

One he gave 1 star with the review: "five is over rated. just 3 for this" -- sounds like he thinks he's God's gift to critics or something. What an elucidating description of his playing experience.

The other he gave 2 stars with the equally eloquent review: "rofl gothic", even though: (a) the map is a major mod from the original Bungie Cryptic Wightings; and (b) that's the first thing it says in the description of the map! DUH! Apparently Killery never reads anything.


My personal opinion with ratings is:

(1) Unless there's a LOT of traffic like we had on The Mill back in the day there's no point in ratings at all. If people download stuff, they download it. Let those #'s do the talking. People don't download stuff 400+ or 700+ times these days because the maps suck.

(2) Personally I'd make ratings only be 3 or more stars. If you don't like something, no one else cares. Often bad ratings just discourage newbie mapmakers anymore (not something we need). If something gets only a dozen downloads over a long period of time then it probably isn't that good (or it's just obscure).

(3) Most people can't be bothered to take the time to do good reviews/ratings, if they did we'd have a lot more of them on The Tain. It's much easier to give bad reviews/ratings without an informed and rational explanation (e.g. "I hate this", etc.).

(3) I think ratings on maps/plugins should only come from mapmakers and/or people with some kind of respect and reputation in the community. People who fly through and give poorly educated ratings/reviews have absolutely nothing to lose.


This has REALLY irritated me -- especially since this guy just whipped through and gave a couple of crappy ratings in a matter of seconds to things that took literally hundreds of hours to create. This ticks me off. I've been trying to ignore it, but since this second review I've started to think this guy is doing it on purpose. I bet I put 10x the amount of time into the text description on The Tain as this guy took to play and critique it properly. How many other ratings/reviews has he sprinkled through The Tain? You could go on a spree and pollute the place in minutes. Why not? Who's gonna stop you?

I don't mean to bitch/whine but I put waaaaay too much effort into this stuff for someone to come through on a whim and give poor (and obviously uninformed) ratings. Who needs it?

I'm seriously debating yanking these plugins just because this pisses me off. Then if people want to get them they can just d/l them from the OoH Battle Plan.

</rant>


Ok -- now that I've ranted (and thanks for listening)...

So people can say the obvious first response to my rant: "ratings don't mean anything, just ignore it". Well, if that's true then I think we should let the mapmakers remove (but not edit) ratings. Prune out the crap. I'd rather have 400+ d/l's and zero reviews/ratings on something than one B.S. review/rating. Better that than removing the maps methinks.

Another good option would be for people posting plugins to have a "no reviews/ratings allowed" option or: "screen reviews/ratings". If no one gives them good reviews/ratings, then fine. At least you can weed out the wankers.


I hereby open the floor to thoughts and comments on this subject, and thanks again for listening to my rant.

Cheers.

Re: Dubious Ratings

Posted: Wed Oct 28, 2009 6:28 am
by haravikk
It's true that there simply isn't enough volume for the ratings system to work properly. Giving control to the mapper though wouldn't be much better, as they may remove genuine ratings they simply don't agree with.

Really it'd need normal moderation. A report button may be better, which basically just sends a PM or something so it can be reviewed and removed as appropriate.

Re: Dubious Ratings

Posted: Wed Oct 28, 2009 8:56 am
by Khadrelt
I've never liked ratings systems. It's a proven fact that people are more likely to go out of their way to express their opinions about stuff they hate than stuff they like. Plus my opinions seems to differ widely from popular opinion. Lots of the maps that get tons of rave reviews I don't really like much at all, but I don't go in and give it a '1' rating just to be a punk.

Re: Dubious Ratings

Posted: Wed Oct 28, 2009 2:10 pm
by vinylrake
agree that there aren't enough people in the community active enough to make the ratings really effective. add in the midst of the remaining very active members are a handful of feuds/bad-blood/personality-conflicts and ratings are really ripe for abuse.

# of downloads is good measure. tho its not a measure of how much people like something, most maps aren't going to get huge dl #s unless there is some word of mouth buzz. the worst thing that can happen is someone dl's something with high dl#s and thinks it's crap. but unless one is on a dialup it's not a big deal to dl a map. even a big one.

a useful statistic might be # of games played on mariusnet that had a map/plugin active, but i don't imagine that's something that is tracked by mariusnet - don't even know if such a thing were possible, but it would give an idea of what's being played. ideal would be "#of games in the last month, last 90 days, and cumulative total".

Re: Dubious Ratings

Posted: Thu Oct 29, 2009 6:43 am
by Eddaweaver
Don't take it personally Baak, he has gone through a large number of plugins and given them low ratings. These are a sample
crappass
not just bad
rofl. not good.
well. make it better
acking noisy shit map
not acutual castal and worse mesh
there is no good
good effort but no fun
very noisy map 1.5 rate
wtf wtf shit
silly map. cant see head map
this is not bad but too small map
trees are good. color map sucks
overhead map is kewl mesh is horrible
not just bad map
nothing special
stupid shit map.
gay map
water is very noisy except that 4 rating
rofl gimbler and URzooked
five is over rated. just 3 for this
rofl gothic
worst map ever
:mrgreen:

Re: Dubious Ratings

Posted: Thu Oct 29, 2009 11:49 am
by Doobie
yes, as others had said there aren't really enough people rating for the ratings to be terribly effective. That said, i'd be reluctant to do away with them altogether (interesting side note: the original developers didn't want the tain to have ratings in the first place).

I really like the idea of tapping mnet for stats like number of games played and such. That's a very cool idea and very feasible.

As for rogue raters... tough subject, but i don't know if I would even want to bother removing them or their ratings.

I mean, in clearly malicious cases, ok, maybe, but maybe this guy genuinely doesn't like the long list of plugins he's been reviewing. And that's fine, he's entitled to his opinion, even if he's wrong.

I definitely do NOT favour letting map makers cherry pick reviews, that sounds crazy to me. However, being able to mark something as unreviewable might be an interesting option.

just my 2 cents

Re: Dubious Ratings

Posted: Thu Oct 29, 2009 1:53 pm
by Death's Avatar
Indeed, I would be opposed to removing any ratings, or allowing authors (or admins) to affect them in anyway (other than with the original map of course!).

The point of having a review is that it can be made quite clear how serious to take the rating. If the reviews are all craptastic like those are, it is pretty clear the particular rating is meaningless.

Re: Dubious Ratings

Posted: Thu Oct 29, 2009 9:42 pm
by vinylrake
so maybe a useful stat to show on a review would be the reviewer's average rating - useful when a person rates 90% of the maps he reviews 1 or 2 stars. Or how about even a link for 'other reviews by this reviewer' so you could click or rollover a reviewers name and see their average reviews, #of reviews and a link to their other reviews?

Re: Dubious Ratings

Posted: Thu Oct 29, 2009 10:01 pm
by Death's Avatar
I would really vote for the "dont worry about it so much" approach.

While some of the mariusnet stats would be cool in their own right, I don't really feel like any action is needed to "handle" these bad reviews/ratings. If anyone really cares, they will click to read the review and see that the corresponding rating is nonsense.

Actually I have a lot of respect for Baak's approach: if someone rates your poorly, contact them and see what the problem is, and if it can be resolved.

Obviously trolls ruin this but they are also few and far between.

Re: Dubious Ratings

Posted: Thu Oct 29, 2009 11:35 pm
by Baak
Whoops -- clobbered and had to repost (see below)...

Re: Dubious Ratings

Posted: Thu Oct 29, 2009 11:48 pm
by Pyro
Perhaps a rating for the reviewer? Where anyone can review the quality of reviews others give. Like a "agree" or "disagree" with this review option or a variable included with their rating. This variable would modify the rating so that the number of agrees and disagrees alter it. Like if 50% agreed and 50% disagreed, then whatever the review gave it like a 3 stays as a 3. Well agree and disagree wouldn't be the correct words more like "should be higher" and "should be lower".

Or make a post and get a squad of potentional reviewers to take a pass at a bunch of downloads.

Re: Dubious Ratings

Posted: Thu Oct 29, 2009 11:50 pm
by Baak
Wow, best discussion I've seen on a forum in ages :)


I think for me it comes down to the following: I agree with Doobie that people are entitled to their opinion, but I disagree with both the method and lack of accountability. There is no weighting based on any criteria, so people can troll and just mark stuff on a whim. Because there are so few reviews (people can't be bothered to take the time) then trolls' ratings/reviews are automatically weighted too heavily -- a major flaw.

Case in point is my Triskelion map. It had 470+ downloads over a few years before one troll comes along with a 2-star, 2-word review that clearly shows he/she never even read the description. That one review/rating that took how long (seconds? minutes?) is the only rating/review. And this is ok because it's "his/her right to do so"? Come on.

If people can't take the time to review properly, how many people are going to see the 2-star rating and possibly skip the download because of it. This is where it's not enough to say that everyone is going to be as educated as this discussion group (i.e. people who care and are informed) and "rework" a rating in their heads after actually reading through the reviews. Only people like us might do that.

If people are entitled to their opinion even if they can't be bothered to read the description or didn't play the plugin much (i.e. invest almost no time/energy), then I believe the mapmaker is equally entitled to erase their opinion -- why not? They have their rights, so do the mapmakers. It's the mapmaker's work.

What happened to the mapmakers right to say: "My opinion is this rating/review is inappropriate and shouldn't be included". In the current system the mapmaker has no rights and no say in the matter whatsoever. Why not? It's their work.


How about this:

Show the mapmaker-removed rating/review greyed out but do not include it in the overall rating. The mapmaker would also have to include a text reason for why they blanked it.

This way if people are removing what are obviously real reviews, it will be apparent and the educated/informed people (like DA mentioned) will know the difference (because they actually read) and can weigh the ratings/reviews accordingly. If they are not real reviews, no one cares and they can see these bogus reviews if they wish (and the fact that they were wiped).

This provides a feedback mechanism for bad reviews. Maybe the person didn't understand the map because it's meant to be played with a particular plugin and they didn't do that, so then -- if they want to take the time to use the plugin properly -- they can revise their review. If they don't, then the feedback is there for other's to learn -- if they are taking the time to read deeper. If it's a wanker review, this method removes the bogus ratings/reviews automatically.

Self-policing, accountable -- adds depth to the whole system -- and very simple.

I think the same mechanism that allows you to put a reason for blanking a rating/review could also be a comment for all the other ratings/reviews. Sometimes I want to put a little comment/clarification on a point I see in a review, or suggest something else fun people can do -- but I can't. This would be an ideal way, because it's where people who want to take the time to read about a plugin can read additional feedback from the author(s).



What it comes down to for me is I'm not happy that some yahoo can come along and do this kind of thing on a whim (I mean the guy could be 11 or stoned or something) and essentially spit on something I'm proud of. I don't believe anyone has the right to do that.


That's the flaw in the current system: it's too small to work as an "include everyone's rating/review" because there are not enough samples to make the statistics work properly. It's much more like a small art gallery.

If you were hanging your art in a gallery, everyone is definitely entitled to their opinion -- I agree completely -- but you do not post just anyone's opinion on the wall or print it under the critic's column.


I believe the simple method I outline above gives mapmakers the ability to self-police the trolls, educates people who might not understand how something is meant to be played, provides feedback in a fair and accountable way, and encourages mapmakers and players to communicate more freely and effectively.

Re: Dubious Ratings

Posted: Fri Oct 30, 2009 1:53 am
by Death's Avatar
equally entitled to erase their opinion
I disagree fully and completely. By putting your map on the Tain, you put your map to the public, which includes the right of the public to review. If you don't like how the public reviewed your map, or if they were negligent and you just get trolled, that's just the way it goes.

I like the idea of author being able to respond to comments (ebay sellers can respond to negative feedback, Newegg sometimes replies to poor reviews). I think that is a good way to address it, then any curious parties can see why any complaints are not necessarily as the reviewer portrayed.

However, the author should have 0 ability to affect ratings at all. No matter what. That opens the door wide open for trolling in the opposite direction (Not saying you would Baak, but it would happen, and it would happen fast).

But here is the thing: Are these guys even trolls? I would argue not. Any registered user of the Tain is allowed to review maps. If they give crummy ratings with crummy reviews, that is just the way it goes (again, it is quite easy to see which ones are crummy, based on the review itself). The user in question has also given a number of (arguably equally asinine) very positive ratings. His reviews should not be weighted any less only because you do not like where your hard work fell in his rubric.

Re: Dubious Ratings

Posted: Fri Oct 30, 2009 2:13 am
by Baak
So basically you're saying reviewers can do whatever they want with zero accountability and that's ok.

Then I want the ability to have no ratings/reviews on plugins. I would say this comes under my rights as a mapmaker. This way people can download (so no one loses), but I don't have to give people completely free reign to review with no accountability in an environment with statistically insignificant numbers.

Re: Dubious Ratings

Posted: Fri Oct 30, 2009 2:18 am
by Death's Avatar
Baak wrote:So basically you're saying reviewers can do whatever they want with zero accountability and that's ok.
It's not about accountability.

It's about how meaningful their rating is, which is already balanced by the review itself (in the case of Killery, it is clear his ratings are not very meaningful).