Dubious Ratings

[Khadrelt]

Well, would it be possible to allow the 1-click ratings, but be able to also pull up a list of exactly who gave what ratings? That would make it easier to add ratings without having to write a review but still allow everyone to see who gave the ratings so such 'fixing' would not be possible.

[Baak]

It makes sense to tie reviews to ratings for "accountability" -- so you can't just click a bunch of times (especially on your own maps).

Probably tying a rating to the account username would be the way to go, and rule out the mapmaker's username from being able to rate (as is likely the case now).


Again, to me the ratings are the "distill all information down to one number", quickie sound byte that doesn't work when there are so few ratings per map. This is my main gripe about them now -- they're too easily abused, whereas reviews have no such problem.

[haravikk]

If data could be tied into Mariusnet then that would be awesome. I don't know what kind of data can be retrieved or how easy it'd be to tie into the Tain though, but a record of games completed with four or more players would be useful to see how many definite games have been played on a map, or with a particular plugin.

Solo/co-op maps could be reduced to two player minimum, and only counted if they completed normally.

Maps that people mostly play solo though would be difficult, they'd still need ratings.

[vinylrake]

um Baak? I wouldn't worry too much about Killery's review. I just noticed he reviewed one of my maps. While I COMPLETELY agree with the comment part of his "review" (the first half of the sentence) where he says the map is "average", I have absolutely NO IDEA what he is asking in the second half. here's the entire review.

"this map is avarage, but why not compelted head map?"

Why not completed head map indeed?

[Yes, I know there is some language translation issues at work here, but that IS a funny question, you have to admit.]

[Melekor]

Killeryspeak translation: head map = overhead map.

[vinylrake]

Killeryspeak translation: head map = overhead map.

Ah. That actually makes sense in terms of WHAT he was asking, but I still don't know WHY he asked that. Do you know what he means by "completed" (over)head map? Is it supposed to have a border or something? Or does he mean he doesn't like overhead maps that don't show parts of the map you haven't been to yet? Any other ideas?

[Pyro]

My guess was what you just said, VR. About overhead maps not being completely "open" the way many multiplayer maps start out.

[Baak]

My guess was what you just said, VR. About overhead maps not being completely "open" the way many multiplayer maps start out.

Yeah, and you can set them that way -- Wild West is a good example.

I'm surprised that didn't automatically make it a 1-star review.

[vinylrake]

Yeah, and you can set them that way -- Wild West is a good example.

Oh yeah, as a n00bie mapmaker, I am sure I _did_ set it that way probably just to see what it looked like and then either forgot to turn it off, or just left it in for the heck of it.

I'm surprised that didn't automatically make it a 1-star review.

It probably should have been a 1 star to offset the obviously biased positive review.

[Myrd]

Food for thought:

http://zedshaw.com/blog/2009-11-6.html

[vinylrake]

That's an interesting way of looking at stats. I think the author's premise of WHY the binary results occur is inaccurate, and he goes through a lot of work to correct something that isn't necessarily flawed. MY impression is that people generally only go out of their way or make an effort to provide feedback when they love something or they hate it, but rarely when they are just 'satisfied' with something. So if 70% of people who try out a map think "Yeah, that's a decent little map. Not my new favorite or amazingly unique, but it was fun, I'll keep it in my plugins folder" those people are RARELY ever going to bother providing feedback - it's just not important enough to them to bother. It's generally only the people who really like a map or really dislike a map a whole lot who are going to take the time to rate a map and write a review. I believe THAT's why you end up with lots of 5''s and lots of 1's. Not because the general populace is treating the rating system as a binary like/dislike vote, but because the tiny % of the overall populace who are willing to take the time to respond already fall into the like/dislike extremes of the spectrum.

I think I'd prefer a slightly more nuanced rating system than a plain 1-5 stars with maybe 5 categories with ratings of 'Poor / OK-Good / Great' in each category that could be added together to get a more meaningful comprehensive rating. So a reviewer would rate categories like "Map appearance (includes models and scenery)", "Playability (location of starts, terrain passability, etc)", "Units" (rating of any new/modified units, attacks, spells, etc), "Creativity" (would range from "great new idea", "functional variation on a familiar theme", or "a poorly thought out mess"), and an overall completely subjective 'Fun"/replayability / whatever rating. If ratings were given points: 'Poor'(0), Good/OK(1) and Great(2), summing up the ratings would result in a 1-10 rating with most maps somewhere in the middle because even with people who really dislike or really like a map, there are few maps that fail abysmally or succeed wildly in ALL categories. I actually try to do this myself when I rate maps - consider the different aspects of a map/plugin and not just rate it 5 if I loved it or 1 if I didn't. But it's harder when there are only 5 points total - if I break down my review to 5 categories and there are only 5 points/stars to be awarded each of my points is a binary vote on that aspect of a map.

Now that I am writing this I think Captain Pringle used to do something like this in his reviews - lemme check... ok back now - yeah, he had 5 categories which he rated from 1-10 points.

Concept/Idea: 10.0
Design/Creation: 10.0
Layout/Flow: 10.0
Units/Physics: 10.0
Overall: 10.0

[Melekor]

After this latest example of a dubious rating, I think we need to revisit this topic.

Actually, I think dubious is entirely the wrong word to describe these ratings. There's nothing dubious about them, instead they are what I would call "indubitably bogus".

Now, there were a lot of good ideas in this thread for how to fix the system, but the reality is that takes work to implement. If that means that nothing will be done in practice, I would be in favor of just manually going into the DB to delete the "indubitably bogus" ratings, vs doing nothing at all.

[Pyro]

A two and a half year old necro... nice. I wouldn't be against that idea. Though how obvious must a review be to be considered bogus? We have already seen really obvious ones, but is there a gray area where a review/rating could be justified with the correct wording? Then again if someone's review is deleted, that could give them a second chance to make a better one if they cared.

[vinylrake]

the really egregious (indubitably) bogus reviews should be easy to spot. it's the possibly-bogus that can be a tougher call.

the review in question has a 1 star 'rating' but it's not even a review. just a one line comment and a question. i;ve seen people use the review section for comments. would be nice if there were a 'comments' button so people could add comments without having to rate a map. adding a comment could send a ping to a mapmaker's inbox.



i'd be all for a policy that allowed deleting reviews that aren't even reviews so only the real reviews remain, but the question is what is the criteria for determining what a 'real' review is?

i'd say one criteria should be length. definitely no need for excessively long reviews, but (unless it's an incredibly pithy haiku) if a review can be tweeted it's probably not a useful review.

another criteria for elimination would be if the author is grinding unrelated axes or taking points off for things unrelated to the map itself.

[Khadrelt]

I agree with the length requirement. Reviews should go beyond 'best ever!' or 'this sux!' It doesn't have to be very long, just long enough that whoever's writing it has to actually think about what they're saying. That way it'd be easier to spot a reviewer who obviously hadn't actually played the plugin and was just being a dork.