Old 08-26-2019, 11:16 PM   #21
xXOpkillerXx
Forever OP
D7 Elite KeysmasherFFR Veteran
 
xXOpkillerXx's Avatar
 
Join Date: Dec 2008
Location: Canada,Quebec
Posts: 3,800
Default Re: Scoring Data in Brutal Difficulty Range

Your idea of variable decay is basically something I brought up in chitchat discord but less extreme; the ideal formula would be one that varies for each chart.

Let f(c, s) = a
Where c is a chart, s is the raw goods score and a is the resulting AAA equivalency.
This gives much flexibility over various chart structures and can be implemented without too much trouble. Of course, that requires some computed difficulty, hence my work on that some time ago. I'll see if I can extract some useful stats for this thread this week.

I dont think the current system is "fixable" without big changes, it simply does not account for chart structure, which it should.
xXOpkillerXx is offline   Reply With Quote
Old 08-26-2019, 11:27 PM   #22
RenegadeLucien
FFR Veteran
Skill Rating Designer
DeveloperFFR Veteran
 
RenegadeLucien's Avatar
 
Join Date: Jan 2016
Age: 23
Posts: 256
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by xXOpkillerXx View Post
I dont think the current system is "fixable" without big changes, it simply does not account for chart structure, which it should.
Well, the current system relies entirely on the manually-assigned difficulty rating to sum up all the factors like chart structure. To do anything different, we'd basically need an automatic difficulty calculator like Etterna's.
__________________


Zenith Ultimate Struggle: 3rd (D2)

All public 1-36s AAA'd
RenegadeLucien is offline   Reply With Quote
Old 08-26-2019, 11:37 PM   #23
xXOpkillerXx
Forever OP
D7 Elite KeysmasherFFR Veteran
 
xXOpkillerXx's Avatar
 
Join Date: Dec 2008
Location: Canada,Quebec
Posts: 3,800
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by RenegadeLucien View Post
Well, the current system relies entirely on the manually-assigned difficulty rating to sum up all the factors like chart structure. To do anything different, we'd basically need an automatic difficulty calculator like Etterna's.
Well, yes.

Although Rob's idea would be a patch for the apparent edge cases, at some point if we want accurate difficulty we need better maths, that's just how it is. I got so much shit for trying to make a calc, yet it's clearly the way to go, especially for a game like this with so many charts.

Edit:
The manual difficulty can only be the AAA difficulty. The AAA equivalency formula computes a score for any raw goods count. It is evident that the mapping makes 0 sense.

Last edited by xXOpkillerXx; 08-26-2019 at 11:42 PM..
xXOpkillerXx is offline   Reply With Quote
Old 08-26-2019, 11:39 PM   #24
One Winged Angel
Anime Avatars ( ◜◡^)っ✂╰⋃╯
Retired StaffFFR Simfile AuthorD7 Elite KeysmasherFFR Veteran
 
One Winged Angel's Avatar
 
Join Date: Mar 2007
Location: Squat Rack
Age: 29
Posts: 10,720
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by RenegadeLucien View Post
Well, the current system relies entirely on the manually-assigned difficulty rating to sum up all the factors like chart structure. To do anything different, we'd basically need an automatic difficulty calculator like Etterna's.
The current system needed more information to work anywhere near accurately to begin with. Charts have a single number attached to them to approximate the value of a near perfect or perfect score, but the growth rates approaching that point can be widely different depending on chart structure and skills tested, and this had been publically acknowledged. The current system erroneously slapped on identical decay formulas to every chart within a given subtier using a few inputs and comparisons from surveying the event team staff and making modifications as necessary until it looked 'nice'. Much more work needed to be done to capture accurate equivalencies and I voiced that prior to the system's release but no one seemed to care until the problem became much more evident several years later.

Scores on RATO/DP from D6/D7 players were spitting out equivalencies in the FMO or lower range for years but no one paid any mind because it was just a couple charts and sure whatever that's fine I guess. But now that there's gonna be more and an entire division is going to be reliant on most of that range for what comprises their skill rating, that definitely needs to change.
__________________


Quote:
Originally Posted by ilikexd View Post
i want to be cucked by cirno

Last edited by One Winged Angel; 08-26-2019 at 11:43 PM..
One Winged Angel is offline   Reply With Quote
Old 08-26-2019, 11:54 PM   #25
RenegadeLucien
FFR Veteran
Skill Rating Designer
DeveloperFFR Veteran
 
RenegadeLucien's Avatar
 
Join Date: Jan 2016
Age: 23
Posts: 256
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by xXOpkillerXx View Post
I got so much shit for trying to make a calc.
Who gave you shit aside from Mina? I don't think anyone here would actively oppose you or anyone else trying to make a difficulty calc.

Quote:
Originally Posted by One Winged Angel View Post
Scores on RATO/DP from D6/D7 players were spitting out equivalencies in the FMO or lower range for years but no one paid any mind because it was just a couple charts and sure whatever that's fine I guess.
It's been stated over and over and over again by so many people that the current system's accuracy drops very fast as you go higher up in good count; I don't think this was ever an issue unique to DP/RATO or even high difficulty songs in general.
__________________


Zenith Ultimate Struggle: 3rd (D2)

All public 1-36s AAA'd
RenegadeLucien is offline   Reply With Quote
Old 08-27-2019, 12:15 AM   #26
One Winged Angel
Anime Avatars ( ◜◡^)っ✂╰⋃╯
Retired StaffFFR Simfile AuthorD7 Elite KeysmasherFFR Veteran
 
One Winged Angel's Avatar
 
Join Date: Mar 2007
Location: Squat Rack
Age: 29
Posts: 10,720
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by RenegadeLucien View Post
It's been stated over and over and over again by so many people that the current system's accuracy drops very fast as you go higher up in good count; I don't think this was ever an issue unique to DP/RATO or even high difficulty songs in general.
And yet the system remains the same. I'm aware it's evident elsewhere, it was just most glaring on those charts.

I don't see an issue with trying to hammer out any and all issues in an effort to create a more accurate system. I feel like you take these comments as personal attacks and are quick to displace blame elsewhere, such as on the difficulties having needed to account for this when this was a system assumptive of numerous chart qualities being meticulously considered and represented by a single number so as to treat them identically when extrapolating a score's worth.
__________________


Quote:
Originally Posted by ilikexd View Post
i want to be cucked by cirno
One Winged Angel is offline   Reply With Quote
Old 08-27-2019, 12:55 AM   #27
RenegadeLucien
FFR Veteran
Skill Rating Designer
DeveloperFFR Veteran
 
RenegadeLucien's Avatar
 
Join Date: Jan 2016
Age: 23
Posts: 256
Default Re: Scoring Data in Brutal Difficulty Range

I'm not sure where I gave off the impression that I'm taking any of this personally, but if that's what you're getting, that wasn't my intention, I'm sorry for giving off that impression. The current system isn't even mine.

I'm pretty sure we're on the same side here. The system needs to change. My point is that the only way we're really going to get something that's truly accurate is with a calculator. Yeah, we could have some sort of variable decay rating for each song that tunes the base skill rating formula for it, but to do that we'd need a calculator anyway, unless someone wants to manually go through all 2000+ songs and give all of them another number.
__________________


Zenith Ultimate Struggle: 3rd (D2)

All public 1-36s AAA'd
RenegadeLucien is offline   Reply With Quote
Old 08-27-2019, 08:10 AM   #28
xXOpkillerXx
Forever OP
D7 Elite KeysmasherFFR Veteran
 
xXOpkillerXx's Avatar
 
Join Date: Dec 2008
Location: Canada,Quebec
Posts: 3,800
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by RenegadeLucien View Post
Who gave you shit aside from Mina? I don't think anyone here would actively oppose you or anyone else trying to make a difficulty calc.
Mina was the most direct about it but many in discord would keep saying that I'm wasting my time and be pretty passive-aggressive. Mostly people just bandwaggoning with Mina. I admit that I have a problem with people who say something is not possible without being able to provide a thorough proof of their claim. Many just spew some pseudo maths arguments but cant get into details.

Anyway, if interest for a calc goes up for real now, I might have some motivation to help.

Last edited by xXOpkillerXx; 08-27-2019 at 08:24 AM..
xXOpkillerXx is offline   Reply With Quote
Old 08-27-2019, 08:54 AM   #29
xXOpkillerXx
Forever OP
D7 Elite KeysmasherFFR Veteran
 
xXOpkillerXx's Avatar
 
Join Date: Dec 2008
Location: Canada,Quebec
Posts: 3,800
Default Re: Scoring Data in Brutal Difficulty Range

Here's my take on the difficulty factors of individual notes:

-Local one-hand complexity: how difficult it is to hit that note given the past and future X notes on the same hand (future notes are necessary to account for readability). That is something I havent finalized, but generally the difficulty goes up first with the spacing of notes (the less frames between the notes, the harder it is in a non-linear fashion so that 1-framers are much harder than 2-framers but 30 and 31 frames are pretty similar), then by transition (at the same speed, a jump to a single note is always harder than a minijack or a jumpjack, and single-to-single like 12 or 34 or 21 or 43 have a special weight for being easily hit as a jump or not). Some time-based gaussian window over each note gave decent results with a window of about 1 second (30 frames) or less on each side and a low std dev (< 1.0).

-Global 2-hands complexity: a distribution of the two 1-hand complexities at each timestep. For example, a very hard section on one hand with a very simple one on the other hand could be easier than medium difficulty on both hands at same time. This would need refinement for polys and a more well-defined explanation (with general and edge cases).

-Note time: just a factor of where the note is in time. This has to be picked/formulated so that a note after 5 minutes with low complexity cannot be harder than a note 1 minute in with high complexity. It is easily defined once the complexities are defined. Accounts for focus loss and partly for stamina.

-Note stamina: a large time-based past-only window over the aggregated factors above. Essentially accounts for breaks in a song; it's easier to hit a hard section after a break than in the middle of some stream or whatever.


This gives a difficulty number to each note of a file. It then becomes possible to compute a different AAA equiv formula for each file. The overall difficulty of a file would then not be a single number, but rather a distribution over raw goods count. Nothing forbids us to compute and show difficulty for a specific count (AAA difficulty, 10g difficulty, 20g difficulty, etc).

I have a pretty good setup to compute these already, so I'm saying it here to gather more opinions on the aggregation part and various factors (gaussians parameters, complexity, etc).

@rob if you prefer this to be in another thread let me know

Last edited by xXOpkillerXx; 08-27-2019 at 09:10 AM..
xXOpkillerXx is offline   Reply With Quote
Old 08-27-2019, 11:37 AM   #30
Dynam0
The Dominator
D7 Elite KeysmasherFFR Veteran
 
Dynam0's Avatar
 
Join Date: Sep 2005
Location: North Bay, ON
Age: 30
Posts: 8,669
Default Re: Scoring Data in Brutal Difficulty Range

My main concern with that approach is pattern manipulation and stamina. Getting those right would be a tough ask imo. I still think it is far less tedious to have subjective difficulty assignments and as Rob said we just need to get the decay part correct. It's not incredibly far off at this point.
Dynam0 is online now   Reply With Quote
Old 08-27-2019, 11:42 AM   #31
xXOpkillerXx
Forever OP
D7 Elite KeysmasherFFR Veteran
 
xXOpkillerXx's Avatar
 
Join Date: Dec 2008
Location: Canada,Quebec
Posts: 3,800
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by Dynam0 View Post
My main concern with that approach is pattern manipulation and stamina. Getting those right would be a tough ask imo. I still think it is far less tedious to have subjective difficulty assignments and as Rob said we just need to get the decay part correct. It's not incredibly far off at this point.
Can you give me specific examples ? Either from actual files or just made up sections ? It's hard to get into details without concrete examples.

PS: I already have computed the manipulatable sections mostly. That's dealt with by accounting for the number of frames you have to hit singles as jumps (1 frame being harder to manip, 2 frames is most likely a jump). As for stamina, I explained a basic framework for it; what would you disagree with ?

Edit: I also disagree with "it's not incredibly far off"; far off what ? The only thing the current system can hope to reach is optimal subjective consensus (as in the most people who agree with the difficulties). While that isn't a bad metric per se, it is inevitably flawed and biased, and doesnt solve the problem: variable decay is still just a decay, it doesnt fully compensate for chart structure, it just covers cases where the distribution of ordered note difficulty ressembles one unique function with a modifyable decay. The truth is there can be many more shapes to that distribution.

Last edited by xXOpkillerXx; 08-27-2019 at 12:03 PM..
xXOpkillerXx is offline   Reply With Quote
Old 08-27-2019, 12:28 PM   #32
Dynam0
The Dominator
D7 Elite KeysmasherFFR Veteran
 
Dynam0's Avatar
 
Join Date: Sep 2005
Location: North Bay, ON
Age: 30
Posts: 8,669
Default Re: Scoring Data in Brutal Difficulty Range

Idk man I just think taking this granulated an approach to difficulty is not worth the effort and it's always going to be prone to outliers reliant on fudge-factoring. Who determines how difficult a trill is a 240bpm? Someone who is good at them? The average player? Who is an average player? You see how silly the concept of automating this is? It's entirely subjective. There will still be 97s that feel like 96s and so on based on individual player strengths.
Dynam0 is online now   Reply With Quote
Old 08-27-2019, 12:40 PM   #33
xXOpkillerXx
Forever OP
D7 Elite KeysmasherFFR Veteran
 
xXOpkillerXx's Avatar
 
Join Date: Dec 2008
Location: Canada,Quebec
Posts: 3,800
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by Dynam0 View Post
Idk man I just think taking this granulated an approach to difficulty is not worth the effort and it's always going to be prone to outliers reliant on fudge-factoring. Who determines how difficult a trill is a 240bpm? Someone who is good at them? The average player? Who is an average player? You see how silly the concept of automating this is? It's entirely subjective. There will still be 97s that feel like 96s and so on based on individual player strengths.
That pattern approach never worked for me. The questions you're asking are legit ones, but that's like surface level. I'd really appreciate to debate more on the points I mentionned but you have to be willing to talk details, otherwise this discussion goes nowhere (which is often the case). I'm not saying a calculator will achieve 100% agreement from everybody on every difficulty, but it would give a much better framework to argue on.
xXOpkillerXx is offline   Reply With Quote
Old 08-27-2019, 12:45 PM   #34
Dynam0
The Dominator
D7 Elite KeysmasherFFR Veteran
 
Dynam0's Avatar
 
Join Date: Sep 2005
Location: North Bay, ON
Age: 30
Posts: 8,669
Default Re: Scoring Data in Brutal Difficulty Range

Well considering my prior post stated that going into said details is not worth the effort, I am out of this discussion :P
Best of luck though, if you do figure out a way to do it then I'll be proven wrong and then some
Dynam0 is online now   Reply With Quote
Old 08-27-2019, 03:29 PM   #35
SputnikOwns
The Frog
FFR Veteran
 
SputnikOwns's Avatar
 
Join Date: Sep 2007
Posts: 106
Default Re: Scoring Data in Brutal Difficulty Range

Correct me if I'm wrong, but isn't it quite rare to have an AAA equal to or above one's rating? The algorithm is excellent as far as I'm concerned -- so long as the song difficulties are correct.
__________________

Last edited by SputnikOwns; 08-27-2019 at 03:30 PM..
SputnikOwns is offline   Reply With Quote
Old 08-27-2019, 03:44 PM   #36
Matthia
A W A K E N E D
FFR Veteran
 
Matthia's Avatar
 
Join Date: Nov 2017
Location: Pacific Timezone, USA Age: 17.8
Posts: 166
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by SputnikOwns View Post
Correct me if I'm wrong, but isn't it quite rare to have an AAA equal to or above one's rating? The algorithm is excellent as far as I'm concerned -- so long as the song difficulties are correct.



shit like this happens but only because of how absurdly low the difficulty of this file outside of two or three main hard parts is compared to what it is at currently

Edit: It is very likely that White Walls Part 2 is also seeking a nerf not much in AAA difficulty but rather the How-Easy-This-Can-Be-Abused factor which is the overall discussion we are having at the moment

Last edited by Matthia; 08-27-2019 at 03:48 PM..
Matthia is offline   Reply With Quote
Old 08-27-2019, 04:07 PM   #37
xXOpkillerXx
Forever OP
D7 Elite KeysmasherFFR Veteran
 
xXOpkillerXx's Avatar
 
Join Date: Dec 2008
Location: Canada,Quebec
Posts: 3,800
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by SputnikOwns View Post
Correct me if I'm wrong, but isn't it quite rare to have an AAA equal to or above one's rating? The algorithm is excellent as far as I'm concerned -- so long as the song difficulties are correct.
The thing is they're not, which is why this thread exists
xXOpkillerXx is offline   Reply With Quote
Old 08-27-2019, 04:40 PM   #38
SputnikOwns
The Frog
FFR Veteran
 
SputnikOwns's Avatar
 
Join Date: Sep 2007
Posts: 106
Default Re: Scoring Data in Brutal Difficulty Range

That's totally fine of course. They should be ordered correctly according to top players. No need to adjust the algorithm for 100+ though.
__________________
SputnikOwns is offline   Reply With Quote
Old 08-27-2019, 04:45 PM   #39
xXOpkillerXx
Forever OP
D7 Elite KeysmasherFFR Veteran
 
xXOpkillerXx's Avatar
 
Join Date: Dec 2008
Location: Canada,Quebec
Posts: 3,800
Default Re: Scoring Data in Brutal Difficulty Range

Quote:
Originally Posted by SputnikOwns View Post
That's totally fine of course. They should be ordered correctly according to top players. No need to adjust the algorithm for 100+ though.
This is the second problem. As it was said a few times, there are a bunch of files which are hard AAAs but easy SDGs, and that breaks the current system.
xXOpkillerXx is offline   Reply With Quote
Old 08-28-2019, 05:40 PM   #40
One Winged Angel
Anime Avatars ( ◜◡^)っ✂╰⋃╯
Retired StaffFFR Simfile AuthorD7 Elite KeysmasherFFR Veteran
 
One Winged Angel's Avatar
 
Join Date: Mar 2007
Location: Squat Rack
Age: 29
Posts: 10,720
Default Re: Scoring Data in Brutal Difficulty Range

I agree with Dynamo, honestly there are far too many variables to deal with to ensure a calculator will work effectively. Manually establishing separate decay formulas for charts that are more accessible to less skilled players will reach higher levels of accuracy and communal consensus without the hassle of dealing with all the intricacies needed to be taken into account for a calculator in a game that encodes charts at 30fps. A 16th roll at 300bpm starting on a quarter note can effectively be jumptrilled because the one frame gaps all coincide between arrows hit with the same hand (1/2 and 3/4), but offset the roll by a 16th and now all the one frame gaps appear at 2/3 and 4/1. Having to account for quite literally the same pattern being variable in difficulty depending on the starting note, on top of everything else that needs to be taken into consideration...I personally wouldn't want to invest any time in that.

What Dynamo was fudging around with in Excel looks interesting and I might mess around with that later.

Tangential to that, I've looked at suggestions and scoreboard data and am interested in feedback for this structure of 98+:

98

A Quick Death
Extratone Pirates (Drunk Optimus Remix)
J'ai Envie de Plaisir X2
Playing Super Mario World While Taking Mushrooms
Punkture (sadly)
Schmollbluk

99

A Dichroic Glass Snafu
Almost There*
Call me it. (500 Tortures)
La Campanella
oppIrish (GYPSYCOREremix)
Stress Free Style

100

/a/
Epileptic Crisis
Jamais Vu
Magical 8bit Tour*
Rave 7
Unconnected.
where is my balls
Winter Wind Etude*

101

cold (Kurorak's Bootleg)
DISCHARGE RUSH
do i smile?
Heterochromia Iridis
Husigi Usagi Milk Tei*
Metro
Music -corrupted mix-
OWA Raged at Kurorak for Making OWA Raged at Skeletor
OWA Raged in Vegas (Fuck Bees)
System Doctor*
The Bridgeport Run
White Walls, Part 2*

102

Revolutionary Etude
Staple on Smile
The Disappearance of Hatsune Miku -Theatrical Version-

103

Make the Fire Burn
Odd-22

104

Powerflux
Violent Arcade
You Universe

105+ TBD

A few notes:

* denotes charts that would benefit from separate equivalency decays. I'm not suggesting they should share identical formulas, but a manipulation of the current formula is required.

La Camp's scoring data after a decade in game suggests it can hang with the 99s. Structurally I may not entirely agree with that shift but it fills out the 99s a bit better for the time being. I know there's more charts queued that will end up in this range so stuff will probably move a bit.

I'd rather move M8BT up than AT down. I don't think a gap of two difficulty points exists between those two charts which is why AT didn't also move down.

Punkture is Punkture. There's players that think it shouldn't even be a 95 but an overwhelming majority thinks otherwise. Scoring data is vomit and nothing else exists to compare it to. 97 at the min works, 98 is likely okay.

_.Pulse isn't here because I'm projecting a shift to 97. If people think 98 is more appropriate then sure.

I wanted to bring some 101s up to lessen how many charts exist in that tier but none of them feel as difficult as what's included in the tier above. I feel there's a hard line between the 101s and 102s. Also considered breaking up the 101s into two tiers and pushing 102+ to 103+ but the 101s truly feel quite similar in difficulty to me, and I don't think there's a two point gap between anything that would be considered a lower end 101 and the current 102s. Think we might just need more charts in the 102/103 range (DZ resubmit Apocynthion..)

Feedback appreciated.
__________________


Quote:
Originally Posted by ilikexd View Post
i want to be cucked by cirno

Last edited by One Winged Angel; 08-28-2019 at 05:57 PM..
One Winged Angel is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump



All times are GMT -5. The time now is 06:47 AM.


Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Copyright FlashFlashRevolution