tain.totalcodex.net

Death's Avatar wrote:It's about how meaningful their rating is, which is already balanced by the review itself (in the case of Killery, it is clear his ratings are not very meaningful).

This is where I disagree.

With zero peer type reviews (from other mapmakers, etc.) and such statistically insignificant numbers, you have one person's not-very-meaningful rating/review being the only one on the board -- this gives it 100% weighting.


[Edit:] I'd prefer to have the option of having no ratings/reviews (including good ones) than letting people have free reign where I don't believe they are acting responsibly. This would fall under the rights of the mapmakers imo.

I mean, if you don't like the system...

Actually here is a really simple solution that I think address the issue of a lot of these outliers.

Don't calculate a composite score until there are say...3 rating/reviews.

I think I have the solution:

Drop the ratings(stars) and leave the reviews.


This way you go to a map and see "2 reviews" -- read them and make your own judgement. You can get a feel for what the reviews are telling you.

No ratings (stars) and therefore no bullshit resulting from improper weighting.

People who want to read the reviews can do so and glean whatever information they want from them. Add the ability for mapmakers to comment on the reviews and you have a really nice system imo.

Baak wrote:I think I have the solution:

Drop the ratings(stars) and leave the reviews.

I havent chimed into this debate yet. I think THIS is a win.

Having played many hundreds of betas Baak releases so he can get a map or plugin set "just right" before an official release I can understand Baak's irritation that a 2 sentence 'review' based on 5 minutes of trying a map or plugin can negatively effect other people's impression of something that took months or even years to perfect, but I can also see the reasoning behind mapmakers not having any 'rights' as far as public reviews or comments.

If the Tain has a clear policy about reviews and mapmakers are aware of it, then I don;'t think mapmakers have any 'right' to have negative comments removed, or not to have the negative rating count in the overall rating - BUT I think a very reasonable compromise would be to let the mapmaker reply to/comment on a negative review (ala the Ebay seller feedback rating). If the existing Tain architecture doesn't allow that easily or it would be too difficult to add, perhaps an author could be allowed to post his/her own 'review' which the author could use to address comments or reviews s/he feels are unfair. That way the negative comments could be addressed, and the author's own rating could offset the negative rating.

I think the real solution is to get more people playing myth and the mapmaker's maps. That way, there's bound to be more reviews which will drown out the trolls.

Myrd wrote:I think the real solution is to get more people playing myth and the mapmaker's maps. That way, there's bound to be more reviews which will drown out the trolls.

Can't we just drown the trolls the old-fashioned way? Water-boarding really is underrated as a solution to pretty much any problem.

Myrd wrote:I think the real solution is to get more people playing myth and the mapmaker's maps. That way, there's bound to be more reviews which will drown out the trolls.

I think that'll most likely happen the day we can all go down and buy a good electric car.


Seriously, I think dropping the star ratings and emphasizing reviews instead (which btw eliminates people accidentally putting 1-star ratings when they didn't mean to -- something else I've encountered) -- with the addition of mapmaker comments on the reviews -- would be a much more robust system that would benefit all.

What about putting it to a vote?

So I finally heard back from Killery -- after a nudge from Pyro it seems -- and after reading through his semi-convoluted message I understand what I thought he did originally with giving my StoneHeart RDF map a 1-star rating:

He gave it 1-star because Vinylrake had given it 5-stars (our order has played 500+ games on it so I think we like it) -- because he thought it should only be rated 3-stars overall.

I thought this is what he meant by his original review saying: "five is over rated. just 3 for this" and giving it 1-stars.

If this isn't an obvious example of how this system can be abused, I don't know what is. One person can sway things and this guy is admitting to doing just that.

Even if I went and asked everyone who's loved playing the map in the past to rate/review it, this guy is intentionally low-balling the map to bring it's overall rating closer to what he alone believes is correct -- not to rate/review it with his opinion, like you are supposed to.


I hereby close the case for how ratings(stars) can and are abused, and how just having reviews and feedback provide a much better system.

That's definitely an abuse of the system; regardless of how others have rated a file you should rate it based on what you think it deserves, and comment on why and if needed why you don't think it deserves a 5 rating. The purpose of adding a rating is give your take on a map, not negate someone else's.

haravikk wrote:That's definitely an abuse of the system; regardless of how others have rated a file you should rate it based on what you think it deserves, and comment on why and if needed why you don't think it deserves a 5 rating. The purpose of adding a rating is give your take on a map, not negate someone else's.

Precisely.

It is worth noting that this goes both ways. While this discussion started because someone got a few reviews they did not like, changes to the system would also affect the other spectrum. The other side being where people only post 5 or 4 star ratings because of a few specific things rather than say the overall or something.

We should change the system to encourage more ratings. The ratio of ratings to downloads is absolutely dismal under the current system - 1000:1 or more is not uncommon. With so little information being collected, it's no surprise that our ratings are poor quality. To get more ratings, we need to make it easy - single click and anonymous. (although still requiring an account, of course) For this reason, ratings and reviews should be separate things.

Another benefit of more ratings is that it becomes practical to delay computing a score until a minimum number of ratings (10 or so) is achieved. That wouldn't work now, since, afaics the median number of ratings is either zero or one.

Melekor wrote:We should change the system to encourage more ratings. The ratio of ratings to downloads is absolutely dismal under the current system - 1000:1 or more is not uncommon. With so little information being collected, it's no surprise that our ratings are poor quality. To get more ratings, we need to make it easy - single click and anonymous. (although still requiring an account, of course) For this reason, ratings and reviews should be separate things.

Another benefit of more ratings is that it becomes practical to delay computing a score until a minimum number of ratings (10 or so) is achieved. That wouldn't work now, since, afaics the median number of ratings is either zero or one.

Agree.

FYI: The reason we decided on requiring reviews for ratings was to prevent the abuse that was on the Mill - where the admins were suspected of "fixing" ratings. By having people have to write reviews with ratings - this would allow people to see exactly why something has the rating it does.

Whether this decision needs to be revisited is of course under debate - I'm just explaining the original choice why we didn't add 1-click ratings without reviews.