Ingame Song Information

[jimerax]

Originally Posted by qqwref

I'll outline my idea again. You compare two files by simply taking pairs of scores, the best score from the same player on each file, and comparing them by how far they are from the AAA (that is, PA). After you discard pairs that don't tell you anything useful (people who get over 100 boos on either file, people who haven't completed both files, people who have the same score on both, etc.), the file that has a worse score in >50% of the pairs is more difficult. A technique like this will produce a useful measure of difficulty, even in the worst case scenario, because it automatically gives a higher rating to files most players in the community have difficulty with.

You have to consider player's advantage(there should be some bias for picking some players too), unlock requirement, # of players, release date, etc. how you define? If you can make a formula that makes sence for difficulties, it will be one of useful objective method. It's more difficult to determine the difficulties of easy files.

Also, it's almost impossible to make a formula for pattern difficulties but I still want to consider technical factors as a measure of difficulties (subjectivity etc but still).

[iironiic]

Is it possible to incorporate both ideas into one? I see pros and cons in the two ideas mentioned.

EDIT: If possible, can someone update the difficulties of the files with ?? in the 1-100 scale? I'm trying to figure out something using these values along with the ideas mentioned in this thread. Thanks!

[qqwref]

Originally Posted by jimerax

player's advantage(there should be some bias for picking some players too)

Some people are better at one type of chart, some at another. Compare, say, GEP and D&D. If they are really equally difficult, the numbers should roughly cancel out. If most people are better at one chart than another, I'd say it really is more difficult.

Originally Posted by jimerax

unlock requirement, # of players, release date, etc.

Completely irrelevant to my approach. Players who haven't played both of the files don't get counted. That's where it differs substantially from the "count AAAs/SDGs/FCs" idea - you're not comparing statistics, but individual scores.

[emulord]

The only problem with that is when n is small. For something like Token Whore, nearly everyone who unlocks it will AAA. When someone doesn't AAA it, the difficulty might jump from 1 to 7 or something.

[jimerax]

No there're many problems other than that.

- It's still not easy to determine actual difficulty criteria, if you want to this completely objectively.
- If we can't get enough players, we need to get some trustworthy and active players with various skill level as control. They always need to do their best on their scores (if scores aren't serious, this method is quite useless).
- Basically we can't determine initial difficulties with that method.

[qqwref]

Originally Posted by jimerax

- It's still not easy to determine actual difficulty criteria, if you want to this completely objectively.

What do you mean "criteria"? Isn't it reasonable to say that all VCs (for instance) should be at least as hard as some specific given song? That way you set a specific bar, and people know what to expect. You play a 6, you can expect that for most people it will be between the highest-5 cutoff song and the highest-6 cutoff song in difficulty.

Originally Posted by jimerax

- If we can't get enough players, we need to get some trustworthy and active players with various skill level as control. They always need to do their best on their scores (if scores aren't serious, this method is quite useless).

Sure, in the beginning a borderline song might be rated slightly higher than it should. But the big gain of a totally automated system is that it literally doesn't matter. If people end up getting better scores on the file in (say) a month or two, it will settle back down to the proper difficulty value without any intervention on your part. And if it doesn't, it was really that hard to begin with.

Originally Posted by jimerax

- Basically we can't determine initial difficulties with that method.

I think experience has shown that it's easy to determine an initial difficulty, plus or minus one, just by playing the song and roughly comparing it to other ones you know. And if you make a mistake on that, it again won't matter, because the script will automatically fix the difficulty in the future anyway. This is in sharp contrast to the situation now, where difficulties are regularly listed wrong, and then only the most blatant ones are ever fixed.

I would actually say that you cannot usefully place borderline files just by subjectively guessing how hard people will find the patterns. You have to actually have people play them.

[jimerax]

Originally Posted by qqwref

What do you mean "criteria"? Isn't it reasonable to say that all VCs (for instance) should be at least as hard as some specific given song? That way you set a specific bar, and people know what to expect. You play a 6, you can expect that for most people it will be between the highest-5 cutoff song and the highest-6 cutoff song in difficulty.

well, I just want to see the actual process.

1. pick all people (or some people) who have positive scores on both A and B?
2. compare average PA?

Also, how do you deal with non-pass, non-FC, mash-FC scores (100 boo cutoff ?)?

[qqwref]

Originally Posted by jimerax

well, I just want to see the actual process.

1. pick all people (or some people) who have positive scores on both A and B?
2. compare average PA?

Also, how do you deal with non-pass, non-FC, mash-FC scores (100 boo cutoff ?)?

I've posted the details at least once, but it goes like this:
- choose all players who have played file A and file B;
- throw out players who haven't passed both files, or who have at least 100 boos on one (mashing is not representative of real difficulty);
- for each remaining player, give one point to the file they have a better raw score on, in terms of points missing from the AAA (or do nothing if the two raw scores are equally close);
- the file with more points is easier.

[rushyrulz]

emulord brings up a good point with Token Whore. Under qqwref's system, wouldn't TW be rated at a 1 since everyone who has unlocked it clearly has the potential to AAA, even though it's obviously harder than all the other 1s?

EDIT: lol there really needs to be a scorewipe on token songs for people who haven't earned the token (excluding vrofl of course because that was public for a day).

[iironiic]

Originally Posted by qqwref

I've posted the details at least once, but it goes like this:
- choose all players who have played file A and file B;
- throw out players who haven't passed both files, or who have at least 100 boos on one (mashing is not representative of real difficulty);
- for each remaining player, give one point to the file they have a better raw score on, in terms of points missing from the AAA (or do nothing if the two raw scores are equally close);
- the file with more points is easier.

The problem I see by doing this is this: what if you are considering a user's score that doesn't reflect his/her actual skill? For instance, I still have 18-0-0-11 on He's a Radical Rat, but I haven't gotten a better score yet in spite of laziness and frusteration, but I know I can improve that.

Also, how are you going to consider raw scores that are something like 4-1-1-0, when that score is obviously not going to be recorded on the high score tables? Where will you find scores like that?

Lastly, hypothetically speaking, if everyone were to improve overall, by what you are saying, will that lower the difficulty of a given file?

EDIT: I also don't like the idea of determining the difficulty by the mechanics of the file. It brings to the same problem that people have different skills and weaknesses. Plus, what constitutes a file that's considered an 8 vs. a 9?

I think the best way to determine the difficulty is to have a fixed file which marks the bounderies between two different difficulties, and relatively compare which one seems harder or not. However there are a few problems to consider with this approach. Again, we need to consider differences with skills and weaknesses, but I think this could be remedied if we have more input from the community claiming which file is harder. Another problem is, which files should be the "fixed file?" That is something we need to figure out, but it seems that we have a general idea on this already.

EDIT2: I also see advantages in both approaches, so if necessary, we can consider both ideas and integrate them into one.

[qqwref]

Originally Posted by rushyrulz

emulord brings up a good point with Token Whore. Under qqwref's system, wouldn't TW be rated at a 1 since everyone who has unlocked it clearly has the potential to AAA, even though it's obviously harder than all the other 1s?

Yeah, I guess we'd have to do that one by hand, since we have no people of the appropriate skill level playing that file. (Another question: if this is the case for a file, does its difficulty rating actually matter?)

Originally Posted by iironiic

The problem I see by doing this is this: what if you are considering a user's score that doesn't reflect his/her actual skill?

Yep, this definitely happens. I think this will even itself out though, because given two files that are equally tricky, about the same number of people will get a good score on A and a bad score on B as the other way around. If lots of people improve and have no problem getting a good score on A but can't do one on B, then maybe B really is harder to score on.

Originally Posted by iironiic

Also, how are you going to consider raw scores that are something like 4-1-1-0, when that score is obviously not going to be recorded on the high score tables? Where will you find scores like that?

Those scores aren't (and can't, really) be considered. Difficulty FCing should be a component of difficulty, though. If everyone can get 30ish goods on song A but has trouble FCing with good PA, and everyone can get 30ish goods on song B and has no problem FCing it, then I'd say A actually is harder.

Originally Posted by iironiic

Lastly, hypothetically speaking, if everyone were to improve overall, by what you are saying, will that lower the difficulty of a given file?

Only if they don't also get improved scores on the boundary files.

Originally Posted by iironiic

I think the best way to determine the difficulty is to have a fixed file which marks the bounderies between two different difficulties, and relatively compare which one seems harder or not.

I agree. There are definitely some considerations to picking these files, though; they have to be popular and old (so we get lots of data) and they shouldn't be too biased towards one specific type of skill. So, we wouldn't want something like Starlight for the lowest-VC cutoff, because doing well on that song is entirely contingent on being able to do the jacks.

[bballa48]

I think we need to move far away from this objective formula.

I don't think the current system of subjective rating is bad at all. The only thing I think it needs is some sort of panel of "experts" (not necessarily all the best players, but rather players of varying skill levels who play the game enough to give consistent feedback). Having 1 or 2 people rate the files, while relatively effective, I think is kinda giving the cold shoulder to the community.

For example, Club is an 87 for me and a 75 for others. If I and someone else who sucks at jacking rate it, is would cause an outrage.


Tldr; Same system, just with more judges.

[rushyrulz]

How is 101 Kittens a 9 and Mute City a 10?
Something is wrong with this system lol

[iironiic]

Originally Posted by rushyrulz

How is 101 Kittens a 9 and Mute City a 10?

This is why I don't like our current system. We are basing two different difficulty scales on two different files, and are now getting absurb ratings like this. That's why I think it's best to fix a file emphasizing a variety of patterns (like what qqwref is saying) to determine whether something is a 9 or a 10. Without looking at the songs list, I am willing to bet that Mute City Remix was an older file than 101 Kittens.

EDIT: 101 Kittens was released on 02-24-08; Mute City Remix was released on 08-24-07. Clearly, we see that between these two dates, people have been improving to the point that something that used to be a 10 should now be a 9. We still observe this today.

[rushyrulz]

Not only are the patterns similar, but in 101 Kittens the 24ths are faster(?) and trickier(!), the jump sections are more difficult, and the song is longer.
IMO, 101K needs to be moved up, not Mute City moved down. The whole difficulty of n migrating into n+1 territory thing never should have happened. Back before I got oni I kept saying that I was never going to get it because the difficulties of songs kept getting bumped up and 11s were becoming harder and harder.

[jimerax]

criticisms without showing suggestion for improvement (like "this system is wrong lol") won't help anything.

obviously we aren't comparing difficulties of all 1000+ songs there might be flaw on this system. so this thread exists for reporting suspicious ones.

101 kitttens is long/hard to PA, yet doesn't require speed/stamina.
10 worthy overall, though.

[jimerax]

Originally Posted by qqwref

- choose all players who have played file A and file B;
- throw out players who haven't passed both files, or who have at least 100 boos on one (mashing is not representative of real difficulty);
- for each remaining player, give one point to the file they have a better raw score on, in terms of points missing from the AAA (or do nothing if the two raw scores are equally close);
- the file with more points is easier.

process 2 - on high 12-13 songs, most people have 100+ boos (also in general, 100 boo FC on 13 is better than FMO AAA skill-wise). needs some adjustment?

process 3 - which parameters do you use for the calculation (total score or just good/ave)? I think calculation like this is technically possible, we need to do this for either "all players (statistically, requires large # of n)" or "selected players (can be biased but consistent)".

[igotrhythm]

JX actually raises a valid point for process 3. Goods and avs impact raw and total score the same as percentage of points scored for getting a perfect (good is 50%, av is 10%). This is before combo is factored in, of course. However, while 11 boos = 0.8 goods in total score, 5 boos = 1 good in raw score.

Still, I like this idea, but to reduce server load and to exclude fluke runs, perhaps the sample should be all players who have played both songs at least 10 times each. Not good enough to give feedback on process 2, though.

[rushyrulz]

Originally Posted by jimerax

criticisms without showing suggestion for improvement (like "this system is wrong lol") won't help anything.

I said that and then I went on to say

Originally Posted by me

The whole difficulty of n migrating into n+1 territory thing never should have happened. Back before I got oni I kept saying that I was never going to get it because the difficulties of songs kept getting bumped up and 11s were becoming harder and harder.

Solution: Quit putting such a high standard on n and just call it n+1. I can see why there needed to be a difficulty rearrangement after the avmiss fix/introduction of speedmods, but the difficulties of songs have continuously become lower than they should be where there's no justification aside from the players getting better IMO.

If the players get better, make harder songs that still fit into the set difficulty system, don't call an obvoius 10 a 9 (Yes Gene Gadget Zone, I'm looking at you) with the idea that you're making the game harder because in fact, the difficulty placement of some songs are just wrong (and I know this is still a w.i.p. and everyone's trying their hardest, especially jx). Or just let the players get better without tampering with the difficulty system.

What's happening is that new songs are being introduced at a lower difficulty than they should be and they become the new high standard for that difficulty. After that, more difficult songs get added to that difficulty and they become the standard. After repeating this process a few times, n+1 is the new n which just discourages players who are trying to get better and anyone who uses the difficulty system as a reference.

--------------------THAT BEING SAID------------------

I didn't really post much of a suggestion to the solution of the problem because I forgot to mention that I support qqwref's idea assuming all the problems and exceptions can be worked out. He seems fairly confident and educated on the topic and I think he believes that his system can be successful.
+1 qq.

EDIT:

Originally Posted by jimerax

101 kitttens is long/hard to PA, yet doesn't require speed/stamina.

subjective. It's a 1700+ note song, that requires stamina especially when the majority of it comes from 16th and 24th stream. It may not require a lot of stamina if you're used to playing longer songs like that, but compared to something like Mute City, it's clearly more challenging PA wise and stamina-wise (length).

Also, Mute City is listed as a 9 in the OP lol.

[qqwref]

Originally Posted by jimerax

process 2 - on high 12-13 songs, most people have 100+ boos (also in general, 100 boo FC on 13 is better than FMO AAA skill-wise). needs some adjustment?

Remember, FC isn't necessary. Anyway, on a 13 most players will have 100+ boos, and you'll only get a few data points, but each one will be enough to prove that the file is harder than the hardest FGO (probably Revo). Even on very high FGOs like Grist, many players will have <100 boos. You will get the correct difficulty as long as enough people have played to be able to compare to borderline files - even one person getting a decent run on RATO will show that it's harder to score on than the FGO cutoff. Having a relatively tight requirement lets us eliminate people who mash the entire thing (which generally gives goods proportional to number of notes, and might rate a file like RWOB harder than a file like RATO).

Don't forget, this method is not for microdifficulties (1-99 scale), it's designed for the integer 1-13 scale. It won't give an exact value for how tough a chart is, but it will group files into the 13 difficulty bins in a consistent, objective, and useful way.

Originally Posted by jimerax

process 3 - which parameters do you use for the calculation (total score or just good/ave)? I think calculation like this is technically possible, we need to do this for either "all players (statistically, requires large # of n)" or "selected players (can be biased but consistent)".

I was thinking raw score (1.0*good + 1.8*avg + 2.4*miss + 0.2*boo) mainly because it gives cleaner numbers. Total score in goods would be something like (1.0*good + 1.8*avg + 3.1272...*miss + 0.07272...*boo), not taking into account the combo bonus. And I really do think "all players who have played both songs properly" is good enough.


As for 101 kittens: I could definitely see this as VC, in terms of PA and AAA difficulty. It doesn't require a huge amount of physical stamina, but it definitely requires a lot of mental stamina to do well on because of the length and trickiness, compared to other 9s.