To start some conversations. I can highlight some thoughts.
Primary reasons on why I prefer weighted averages:
• If Top X songs are set to define skill rating (after many discussions on what X should be) and if skill rating is consistently used as a comparative tool to measure performance between two files demanding different skills and requirements, then the X-th and (X+1)th songs should hold very similar weights and (X+1)th song is weighted at 0 by definition of Top X.
• Although weighted mechanism rewards you for "biases" encoded in the performance of files in your Top N more than the unweighted mechanism, a regulated (key word here) solution should still capture many benefits that the unweighted solution provides: specifically and most importantly, a stronger representation of lower ranked files in your Top X is needed to determine a user's skill rating.
• If we want to reward users for activity, why shouldn't the season's ratings be used there? Whether skill rating vs. seasons rating is assigned as the more "official" metric can be reserved as another conversation. Point is, there is a solution aimed to reward players who consistently play the game.
I get it. Our current weighted system does not do it well, but this doesn't necessarily translate to "any weighted solution cannot do that". It's a tradeoff between "improving representation of lower ranked files" and "rewarding performance for songs subjectively more challenging than what your current skill rating suggests", and in my opinion, that should be respected.
I've written a first iteration of what a weighted setting would look like. Attached is a
Colab notebook for reference. There's a pandas dataframe containing the new projected rankings, the username, their projected weighted skill rating, and their current rank in game.
For fun: If you want to determine your projected skill rating under a weighted mechanism I designed, scroll up to the "Determine your skill rating" section, replace my username with yours, then scroll all the way to the top, and click on the play buttons for the first seven cells.
-