Rating the rating system 9 0

Or we could rework the result algorithm to handle the differences in the way people vote. .-.

Thanks for the review! It's interesting how it shows every person has his own approach to rating games.

I like your first suggestion for labeling grades, since it might help people have a more consistent rating scale (this could resolve in part @CMLSC 's issue). I'd also be alright with introducing a usable "zero" grade, to help 5 feel like a real middle ground.

I'm less a fan of the other suggestions, and the main reason why is the problem of granularity. With the Ludum Dare/itch.io 5-star system, most people rarely use the too harsh 1 star grade, and reserve 5 stars to few excellent entries. For the bulk of the games, that leaves us with 3 options, i.e. "Bad"/"Average"/"Good". The first problem is, in an event as large as LD, this was hardly enough to get reliable rankings (i.e. rating 1000 games with most within a range of 25x3=75 stars means a lot of ties).

Even if we decide not too care about the reliability of rankings (which admittedly can never be perfect), there's a lot of wasted potential, because for two games a user has rated "good", there's probably one that he could have said "I liked it more". With a 5 star system, you don't get that information. With 10 stars, you get it a little bit more (see this rating distribution). (Note: I think Ludum Dare is actually planning for half-stars for the next jam)

So, the idea is that finer grades lets people express more accurately which game they preferred. This idea is also the reason why we have that "Manage ratings" page, which we'd actually like to expand to let you edit ratings directly in it (by reordering games with drag'n'drop, or with manual entry). We're even considering letting people finetune ratings as decimal numbers to let the bigger nerds make their own "definitive rankings". What do you think about the idea?

Oh and to react on the 7-grade/0-centered idea, I see the interest it, but feel like it would be less intuitive than scales we knew at school (5/10/20/100 depending on the country…). Letting your friends know that your game rated a "+1.51" average in graphics would just make them raise eyebrows. Though maybe other people would like the idea? I'm open to more dicussion.

@CMLSC
And making it transparent. Nothing worse than an obscure algoritm "correcting" data. In other words: if done right, then worth a shot.

@Wan
Yeah, I thought the other suggestions might not fly that well (I like the 7 grader exactly for its oddity, it makes people think!) I prefer punchy grades as a reviewer, although I did find use for all the 10-grades when calculating overall scores. If low granularity results in many ties in the top entries then, sure, it is not proper. As long as the system is clear about the difference between grades then we are golden.

I did use the Manage ratings page a few times to checks ratings I'd given in the past to keep things consistent. Didn't change any ratings afterwards as that would not have worked well with my visible ratings policy. I can see it being a useful tool for people who review and rate differently though, so it is a good addition.

Decimal ratings on the other hand… I reckon simply ordering entries in the rating manager (drag and drop sounds nice!) would be good enough. Basically a backend preference value is stored when the scores are the same. Then if the end result scores for two or more entries are the exact same, an unlikely event, the backend preference values can be used as a tie breaker (transparently, of course).

If you do push for a visible decimal system, then it has to be a relatively hidden feature indeed. Otherwise everyone and their dog is going to start using it from the get-go resulting in a world of hurt! 100-grade systems… shudders

Way back in elementary we had a weirdo 4 to 10 system. It was never explained why. (Apparently for historical reasons only, go figure).

Thanks for responding.

that would not have worked well with my visible ratings policy

I like this policy, if I wasn't too lazy I would probably do the same. This makes me think… we could probably set up an opt-in feature to let users make their votes public. Hmmm.

Also we're on the same page for the decimal ratings, if present it should not make things too complicated. Reading you inspired me possible implementations, like making drag'n'drop apply hidden, super small decimal changes just to break ties.

@Wan
A public ratings opt-in would be great!

Glad the idea inspired you. Applying small changes to the grades themselves sure would work. Having thought about it some more, a separate order value used only in the case of a tie would simply create silly problems. It sounded good on paper; time to shred that paper.

I'm not convinced public ratings is a great idea. While I get the idea, since a lot seem to rate publicly anyway; It doesn't solve anything and I always feel bad knowing how other people rated me. It may also add unplanned baises, "they rated me well" will lead to higher than actual scores. Say normally you'd have given a two, that thought may push you into a three, just to spare them. (With or without your own ratings being opted in for public).

I really don't like the public rating idea.
And for the rating, I don't have the same signification, for me a 5 doesn't mean that you are on the average of the other entries but that you made just the minimum of what was expected. So every one can have more than 5 if they have a good game.

@timbeaudet That's a good point. After some thinking I felt that public ratings should not be revealed during the voting phase anyway, because if the practice becomes widespread 1. it could influence other's people ratings a lot 2. it would spoil approximate game rankings. Holding the detailed ratings until the end of the jam would limit the impact of the "rub my back I'll rub yours" thing, but not completely I admit.

@thrainsa Each person has a different scale in mind when rating games, so I'm not that surprised to see Huvaa's differ a lot from yours. You know I'm often even nicer than you ;) Everyone still gives better grades to better games so I tend to think all approaches are valid… But the luck aspect of people getting more or less harsh voters is indeed a concern. Maybe giving labels to grades (Huvaa's first proposal) would help?

@Thrainsa

I really don't like the public rating idea.

As an option no-one is forcing you to use it. The potential for bias is there, if you do have ratings on your own entry of course. I only talk from the perspective of someone making unranked entries.

for me a 5 doesn't mean that you are on the average of the other entries

In any big jam it is probable due to the massive amount of entries that most of them are average. Obviously if I only played, say, 3 great entries, then all of them would receive great grades too. Middle of the road projects get average ratings, regardless of the other entries in the specific jam. Sure enough, I've played thousands of titles so maybe that is easy for me to say.

In any case, people having different signification for each grade is a big problem as ratings between 0 and 10 then lose meaning. You might get lucky and receive a bunch of ratings from people who never rate below 5, or you might not get so lucky. If everyone did use 5 as an average then those in-between ratings would be more specific accross the board as a result.