A
AstroTibs
I noticed that the formulas for various leaderboard elements were available. I have two suggestions that I think will really improve their robustness.
=== 1: Low ratings should suppress leaderboard position, not raise it. ===
In the case of model generators, you have the term (averageRating × ratingCount × 10).
Say that someone has produced an awful model. If this model got 100 1-star ratings, then this term's influence on the leaderboards would be equivalent to a fairly good model: say, one that got 20 5-star ratings.
Since the medium score is 3, I think this term should be changed to
((averageRating - 3) × ratingCount × 10)
This way, 100 1-star and 20 5-star votes will now separate model rankings rather then put them together.
Alternately, to address the point Maxfield made about ratings, you could instead replace "3" with the median value of all voted on models.
=== 2: Counts should be square-rooted. ===
This is inspired by signal-to-noise ratio calculations. In astrophysics, for example, counts of photons entering detector pixels N are uncertain by Poisson error: √N due to various influences.
If my above formula is changed to:
((averageRating - 3) × √(ratingCount) × 10)
then what this does is still award the highest boost influence to models with more ratings, but it also suppresses the runaway leader/Pareto effect. It'll be harder for the highest-ranked models/etc to hopelessly covet their positions from contenders due to, for example, lucky head starts.
Ideally, this square-root would be applied to all counting elements (e.g. download number, reaction count) done on all leaderboard values.