Skill Rating & Leaderboards Discussion [Updated Oct 16]

[TheSaxRunner05]

Originally Posted by PrawnSkunk

What is Skill Rating?

Skill Rating is a new skill-based ranking system that assigns a rating to each player based on their performance on public songs. The following graph describes the relationship between the number of goods in a score and how well the difficulty of the song represents your skill:



This formula was created using data accumulated by the Official Tournament division placement team. Using this information, each qualifying score with less than 28 goods can be changed into a AAA-equivalent score, giving you the difficulty of the song on which you have essentially played with 0 goods. This difficulty is multiplied by one hundred to yield your rating.

I think this curve falls off too steeply to accurately gauge a player's skill.

A) Why have it decrease all the way to 0? Why not cut it off at 75% at the 28g mark? Someone could get 28g on RATO and it be beaten out by a Free Space AAA.

Then again, someone could get 20g on RATO and have it beaten out by an 80 AAA

B) The curve simply drops off too steeply, adding 1 point of difficulty adds more than 1g to your score. This will underrate players who don't whore for AAA's and overrate players than do.

Ex. 16-18g on an 80 is about equal with a 64 AAA - just no way in hell those are in the same ballpark.

For instance in my own, I would consider the more difficult song here the better score -

#7 63 None Would Escape 6,243 1372 0 0 0 2 59
#8 67 Oops 6,237 1394 5 0 0 4 13

Adding 4 points of diffuculty should theoretically add more than 5-0-0-2 to your score on any given song.

C) This system overrates a player's strengths and completely neglects weaknesses. Then again, everyone has strengths and weaknesses. However this system really shows how good someone is at their strengths, and not a measure of all around skill. For example, my jumpstreaming ability is just plain bad compared to my scores on non-JS charts in the FMO range.

D) Suggestions - Fix the difficulties on many errant charts, especially in the lower range. The D2-D3 range is very off because of just a few charts (please fix party4u now that you can record a score without a background). Also because of this new skill system, I think there should be a mixed consideration for difficulty to AAA and Difficulty to SDG. I know OWA currently has spoken against this, but if our skill ratings are going to show this balance, why wouldn't the song difficulties to some extent? ABCDeath is a perfect example, and if it's showing up on many, multiple players top 4 songs, it's clearly overrated.

I like the system, but it can certainly use some tweaking. It doesn't need a major slope change as the difficulty/PA drops, but it needs to be propped up some at least.

[TheSaxRunner05]

My Paint skills are lacking, but you get the idea

[FF_rules]

Originally Posted by Red Blaster

And yet you're still somehow D2 in tournaments.
Get outta here.

I still haven't AAA'd a song in D40, let alone D50. We can be D2 rivals 4lyfe (I have a problem with consistency; but I can generally play in Division 3 I suppose)

Also my skill rating is getting dangerously close to D4 atm according to just P4u and ABCDE lol.

[mrpreggers]

WHY DON'T TOKENS CONTRIBUTE TO LEADERBOARD RANKING FUCK YOU !!!! ok

[popsicle_3000]

mfw my #5 rank is my 7-0-0-1 on dead & direct I got when i was high d3/low d4
skeletor as my #1 FTW

how does the ranking work for same lvl songs? ex i've AAA'd UC, fast asleep and eternal drain, all 67, yet ordered as UC > fast asleep > eternal drain.

edit oops i noticed same rating on all 4 of those

Originally Posted by rushyrulz

Suggestions:
Add skill points section on results screen in R^3 for every score.
Upon searching, display a player's rank within their division as well to assist with tournament placements.

+1

Originally Posted by ThunderFlip

...

Kappa

except you fail to recognize that you didn't actually play any song between your two rank changes. thus meaning it was either people passed you, or (more likely) the formula was tweaked.

[icontrolyourworld]

I suggest a readjustment to some of the divisional cutoffs for skill ratings.
Right now it looks like it's set at
8650 for d7
7600 for d6
6450 for d5
5550 for d4
4650 for d3
2300 for d2
0001 for d1

I think D1-D3 is awkwardly grouped right now. You have to AAA or better level 23 to get to D2, and 47 or better to get to D3. I might be a bit out of the loop on the lower divisions actual skill but maybe something a bit more linear leading up to D4 would be nice. I suggest the cutoffs at:

8650 for d7
7600 for d6
6450 for d5
5550 for d4
3700 for d3
1800 for d2
0001 for d1

what do you guys think?

[PrawnSkunk]

Did a small alternate account purge today. 2 D6 alts, 13 D5 alts.

Originally Posted by icontrolyourworld

I suggest a readjustment to some of the divisional cutoffs for skill ratings.
Right now it looks like it's set at
8650 for d7
7600 for d6
6450 for d5
5550 for d4
4650 for d3
2300 for d2
0001 for d1

I think D1-D3 is awkwardly grouped right now. You have to AAA or better level 23 to get to D2, and 47 or better to get to D3. I might be a bit out of the loop on the lower divisions actual skill but maybe something a bit more linear leading up to D4 would be nice. I suggest the cutoffs at:

8650 for d7
7600 for d6
6450 for d5
5550 for d4
3700 for d3
1800 for d2
0001 for d1

what do you guys think?

Thank you for your suggestion! I agree that an average AAA-equivalency of Difficulty-23 across four songs is too steep for D2. I will lower the cutoff for Division 2 tonight. However, I feel an average AAA-equivalency of Difficulty-37 is far too low for Division 3. As a general rule of thumb, a player should be able to meet two of the following criteria to qualify for that division:

Division 3
46 AAA
47 Flag
53 SDG
58 10-14g
60 15-19g

Division 2
23 AAA
24 Flag
29 SDG
34 10-14g
36 15-19g

[PhantomPuppy]

hmmm, thats actually pretty smart. it did seem like the d2 range was way too large how it was before, with some being barely able to keep up with the songs with their 15-20 AAA's and others AAA'ing all the way up to like, 45 lol. nice fix guys

edit: on another note, its scary how close im getting to being last place in d5 xD everyone is improving and im here staying stagnant atm.

[mrpreggers]

tokens need to count for rank ok look at the 88s there are 5 public ones and 7 tokens alright and look at the public ones like music (for kirby) and xanthystrauma OK i'm at the point where i have to play these files to improve where's my motivation : - (

[_Zenith_]

So it has just dawned on me about something. The new leaderboard system makes it slightly easier to sandbag in divisions.

Case in point: I have several low D7 scores; some that could easily be in my top 4 and bump me up to around that area. Specifically my 5-0-1-1 You Time score, had I not missed, it would be in my top 4. I want to say same to my 11-0-1-3 but probably top 10 rather than top 4.

Players who intentionally miss or boo on purposely such as players like aarrgghh are easily capable of sandbagging when people use the new skill rating system to place people (This would never be the problem in the OT but it's slightly a problem in the current user tournaments).

I know that a change to this would require raw scoring instead of combo scoring which lolwon'thappenforayear because of obvious asinine coding involved into combo scoring and I already know staff is somewhat working on unravelling that code.

Any thoughts?

[Deidara837]

lol aarrgghh

[PrawnSkunk]

Another leaderboard update for you all~

Skill Rating now includes token songs! One Winged Angel has moved any token songs containing zero framers, gimmicks, or short (< 30 note) intense difficulty spikes to Difficulty-0. The following token songs will not affect skill rating: Crowdpleaser, OMGWTFT0k3N, party 4u "holy nite mix", The Wises Were Wrong, and PWND!.

[PhantomPuppy]

oooh, tokens affect rank? i might actually try to unlock some now lol

[FF_rules]

Originally Posted by PrawnSkunk

Crowdpleaser, OMGWTFT0k3N, party 4u "holy nite mix", The Wises Were Wrong, and PWND!.

#1 63 party 4u "holy nite mix" v2 5,622 657 9 0 0 2 60

what about v2?

Originally Posted by PrawnSkunk

Division 3
46 AAA
47 Flag
53 SDG
58 10-14g
60 15-19g

Wasn't D3 supposed to AAA a D50 as a minimum? filling just 2 of the criteria is also pretty vague; 10-14 on D58 and 15-19 on D60 for example (These type of songs have varying range of difficulty; Party 4u v2 has 10 seconds of "d60~ difficulty" and the rest falls very short). What i'm trying to say is Song length isn't factored in at all.

Note the difficulty system was designed to rate a song's AAA difficulty. Using that number to determine it's "~20g margin of error" is overlooking this key issue.

-I'm just putting in my many cents into this issue. Also according to this system I should be nearing Division 4 but I can't even play certain D50 songs and get a SDG. (Sandbagging? Maybe; but my top scores are just easy songs and stuff with inflated difficulty because of jacks).

----

Edit3: Also; I feel there has always been a gap between D2 and D3. As I've skated in between these two divisions and I feel the introduction of D2 and D2.5 in the double tourney was a step in the right direction. There are players who can get up to high D30s~ with a AAA and there are players who can get to D50 with SDG in the Division2 description atm; this is a huge gap of skill. We could probably do with expanding the amount of divisions + altering the formula for weighted scores on top 4 (or more).
-weighted score suggestion: Change the formula for how scores are weighed; the 4th highest shouldn't be worth a fraction of your highest, it should drag your overall score down a little harder. Possibly add more than just top 4; this should change how the system rates scores from people who can score high in a few outliers vs very consistent players. Alternatively; add a system to diversify difficulty of each song by labeling them as part of a play style.
(Streams)
(Jacks)
(Jumps/Hands)
(Rhythm) -mini streams with pauses, 123 break 123 break, etc
(Free Style) -a very diverse song with all of the possible patterns while maintaining difficulty throughout.
The system will categorize songs by scoring each difficulty as 2 different sets (The standard difficulty and each of the 5 difficulties it scores in the categories stated earlier.) Let's say a song is all jacks/jumps and sparse streams (hellbeat); when someone scores 5,000 points on it, they'll fill the category of Jacks/Jump with 5000 point score ~This process continues for songs that are stream heavy (AAA for example) and such until we have a diversified list of at least 10 songs that all fill different categories as their highest scores. Scores will be weighted evenly between their #1song in each category and can be viewed by everyone (Let's say a player has almost all of their scores in Streams and a big gap compared to anything with just jumps or jacks; this can be easier to distinguish what needs work -for the player- too).

[_Zenith_]

Originally Posted by FF_rules

#1 63 party 4u "holy nite mix" v2 5,622 657 9 0 0 2 60

what about v2?

iirc v2 doesn't contain 0 framers.

The token songs he is talking about either contain 0 framers, intensely fast trill sections (CP in particular), and songs that even the best players have a 1/100 chance of actually getting a decent score.

[TheSaxRunner05]

Party 4u's difficulty was based on the background being in the way, now that no background scores record it needs to be nerfed to somewhere in the 50's.

I think what FF_Rules is saying is that the system puts all its weight on our strengths and ignores weaknesses. For example, I get D5 level scores on songs like Boot, Going on Spring Wind, Gynandromorph (overrated file), Goatstep (why is this not a 64), Oops, ETERNAL DRAIN [Heavy], The Divine Suicide of K [Heavy] (How is this still an FMO) and so on, but when it comes to endurance jump stream charts I still score like a mid D4 player.

Also, because the difficulty is based on difficulty to AAA, factors like song length and difficulty density aren't factored either. The best example of this is the song 300. The Jumpstream becomes manageable as a high D3 player and the slop comes from the jack transitions. Sure it may be a 70 in AAA difficulty, but even low-mid D4 players can teens the chart without too much problem, while getting over 100+ goods on most other FMOs. Not factoring for this can overrate players. The solution is simply adjusting some of the difficulties to at least factor difficulty to SDG/Teen instead of just difficulty to AAA.

[One Winged Angel]

I have no intentions of changing the way difficulties are determined, but if SDGs are producing skill rating scores that most players don't deem to be entirely accurate, I'm assuming the formula can easily be tweaked to reflect this.

There's no way to say with absolute certainty that 7g on a 60 is worth specifically the equivalent of a 50.XXX whatever AAA, they're just approximations (someone like prawn or trumpet could elaborate more on how these numbers were achieved). Just know that in terms of official tourney placements and what not, players will still be placed on a case by case basis with individual combing through levelranks. I wouldn't worry a great deal about being misplaced because of one or two outlier scores that are outside of your general skill range.

[TheSaxRunner05]

That makes sense, and if anyone sees a problem with a chart's difficulty, it needs to be taken up in the [Contested Chart Difficulties] thread. I agree it's important to run through everyone's level ranks for placement, only using the ratings as a guideline, but as I've said above, I think the formula for the rating needs to be tweaked slightly.

In terms of difficulties, the mid FMO+ Range has been vetted quite thouroughly, but imo the low FMO/VC/C range has a lot to be desired in fixes.

Is there any way to run a query to see which songs are showing up the most in player's top 4 or top 10? If certain charts are appearing waaay to often, we can use that as a starting point for difficulty discussions? I really would like to see data on that.

Edit:// Sorry it keeps double posting because for some reason my internet connection is crap today

[mrpreggers]

Originally Posted by PrawnSkunk

Another leaderboard update for you all~

Skill Rating now includes token songs! One Winged Angel has moved any token songs containing zero framers, gimmicks, or short (< 30 note) intense difficulty spikes to Difficulty-0. The following token songs will not affect skill rating: Crowdpleaser, OMGWTFT0k3N, party 4u "holy nite mix", The Wises Were Wrong, and PWND!.

ty

[Rapta]

Rank 9 is currently Velocity, and: "Velocity has insufficient games played (< 250 games played and < 50 unique songs passed)" Is this right?