Skeptical of Difficulty Evaluations
Posted on: August 4, 2014, at 01:38:54am

Operating with what I believe I know about the difficulty level assignments, that is, that each of them is assigned a difficulty by a person or group of people, as opposed to using some formula, I have come to question this practice, or at least I question whether enough effort has been put into developing a working formula, thereby improving the game, or perhaps the genre as a whole.

Taking a sample of 358 songs (only those which I have played, and therefore, have easy access to the number of notes in each song), I have made a formula just within excel that has an R squared of 0.922 which seemingly accurately correlates the difficulty of the song to the number of notes per second. Namely, it's the following power expression:

y = 0.98912934x^0.50763381

Where y is the notes per second, and x is the difficulty. This is shockingly close to simply taking the square root of the difficulty to find the notes per second, or conversely squaring the notes per second to find the difficulty. In a way it makes me wonder whether my initial assumption was correct, because even with a sample this large, I really was not expecting such a high R squared. I should however note that the variability of the nps increased with the difficulty, and the fact that I could not include any 75-99 songs in my sample would suggest that higher nps will be come less and less good at indicating difficulty. What is also interesting to me is something that agrees with an inference I made a while ago about there being a theoretical "max" nps, which also relates to the average fastest "clicking speed" that I've noticed, which is about 10/second. My personal top clicking speed (on a mouse, to be clear) is about 14 per second, but I can only comfortably maintain 10 per second. I have a feeling that this physical limit combined with a similar mental limit contributes to this max of 10 nps (this agrees with the max FFR difficulty being 99).

After looking at the forums for a bit, it seems that using nps for the difficulty is not exactly popular, which I understand from the perspective of the D4+ individual, playing at a difficulty level at which there is an empirically high nps variation, increasing as you go up. At the same time, avoiding a serious nps discussion does little to improve the experience of our little community here. I for one honestly do think there exists a system that could take into account all the elements of difficulty within a rhythm game, or at least those that exist within the note chart, and I really think keeping the whole "difficulty is subjective" argument alive is useless (comfortable, easy) to the rhythm gaming genre as a whole.

Call me naive, but since this is my personal favorite rhythm game, I feel like the game has a duty to continue to perfect itself. Are we not all here just to enjoy this "perfect", highly polished experience? I am convinced that more can be done.

1. 1: TheSaxRunner05
   Posted at 10:35pm on September 13th, 2014

   How would the formula handle songs like Pandora or Purple, where all of the note difficulty is in a small section of the song? Also, what if the note density comes from an easy pattern, like (34)(12), would be the same NPS as 32nd note streams, which may have an FFR difficulty difference of about 20 points?
   For, probably, Mid D4 and lower, NPS pretty much decides the difficulty, but as you hinted at, it gets more complicated once you hit about 60 in difficulty.
2. 2: Trumpet63
   Posted at 7:47pm on September 14th, 2014

   You're right, nps does completely miss those factors. I would love to make a formula that somehow accounts for those problems, but I don't know much about step files at all.
   To account for the things you mentioned, I would try the following:
   #1 increase the difficulty based on the variance of the nps, or do something with the max and mins. Or if that doesn't work that well, I could take a weighted average with the squares of the nps (which I think is essentially what variance does).
   #2 go through and reduce the weight of double/triple/quad notes because they are easier to read than 2/3/4 notes in a stream (right?)
   I've given a lot of thought to this, because I'm pretty convinced it can be done. I also anticipate problems with:
   #4 "jackhammers", basically anything above 10 nps of the same note gets really hard, or could require a fingering change. In general I think repeated notes can be tougher than a repeating pattern, so the formula could take any repeated notes
3. 3: Trumpet63
   Posted at 7:48pm on September 14th, 2014

   into account.
   #5 pattern recognition could be used to take into account trills, or simple patterns like 123, 4321 (reducing the difficulty), and to a lesser extent maybe something like 123 124 123 124 patterns.
   #6 if I wanted to go really in-depth, I could increase the weight of notes that are off-beat.
   #7 and for any other kind of esoteric, unmeasurable difficulty, I would dynamically alter the difficulties based on player performance, i.e. % of players that AAA'd compared to other songs of the same difficulty. (Note: if not many people bother to AAA a particularly unpopular song, it could spike in difficulty, but on an active site this would be self-correcting if you give players an incentive to play songs with higher difficulty)