Entropy Gain for per-receptor NPS

Re: Entropy Gain for per-receptor NPS

This makes no sense if you are using those as ground truth in the first place.

Anyway, I don't think it's worth to break down what you currently have if you haven't built a model in the first place. I guess it's fine to test around with some data and see what happens but it'll make more sense to decide what data to extract after you decide what you are modeling in the first place.

On the neural networks topic, lack of useful data sucks but I think convolutional networks could work well to build difficulty curve graphs.

Re: Entropy Gain for per-receptor NPS

So I didn't read this convo but what do you think of showing % of players who played the file that passed/AA'd/AAA'd/etc it, SDVX style

Re: Entropy Gain for per-receptor NPS

What do you mean by "if you are going to use those as ground truth" ? I said I'm going the unsupervised way, there is no ground truth in that afaik. I plan on doing any estimation based on difficulties, not scores. Sorry if I misunderstood your point.

The rest is all true. The goal of this thread was never to talk modeling so much but rather discuss primitives.

Re: Entropy Gain for per-receptor NPS

I currently only have rights to provide stats on the songs/files, I have nothing on the users.

If you meant it as some kind of attribute to predict difficulty could you please explain your reasoning ? Otherwise I'm sorry I can't do that.

Re: Entropy Gain for per-receptor NPS

It gives a rough estimation of difficulty through general performances on the chart
Low % = Hard
High % = Easy
But you need a server to log all the user scores, users have to be online, and the chart needs a good enough number of players
Using neural network is like assigning one person to judge all the difficulties (since it's supposed to map human brains and what not), but what if you disagree with that person

Re: Entropy Gain for per-receptor NPS

Yeah just the fact that you need a massive amount of plays from various players in the lvl spectrum makes that unviable. Plus the difficulty should be predicted before any plays are made (or else it's a bit pointless).

As for the neural net, I have no clue why people are all on it, I dont recall mentionning it in this thread. That being said, you wonder what happens if people dont agree with a neural net's output ? Well, if the vast majority agrees with the net, then those who disagree should try to see if they're biaised because of their skillset and understand what led to that output. If only a minority agree with the output, then the possibility of it being wrong is greater and the chart would need closer inspection to see why that is the case. It's just how things goes when you have no predefined output class or labeled input.

Re: Entropy Gain for per-receptor NPS

Just for the record, I've tried to produce a difficulty algorithm primarily based on "distance to last note on each arrow/hand".

I don't know if there's something inherently wrong with this approach or I was just too inexperienced at programming to see it through to a satisfactory completion but I was unable to get to a result that was deemed usable by myself and the difficulty consultants whom I discussed the results with.

On the subject of neural nets, both myself and Trumpet63 have attempted to use neural nets on FFR's song difficulties using extended level stats. Trumpet got his neural net closer than mine (his had a mean difference of 2.4 points from the actual value whereas mine was 4-5 IIRC) but his used several features (such as note color) that could be cheesed by a clever stepfile artist to over/underrepresent the difficulty of their file (ex. if white notes = high diff, throw in a lot of white grace notes that function identically to jumps in practice.)

Re: Entropy Gain for per-receptor NPS

Thanks for the information !

What metrics did you use in relation with that distance ? Was it min/max/avg/distribution/... ? Because just like nps, it sounds like a solution that needs quite a few statistical values.

Yes note color is arbitrary.

Re: Entropy Gain for per-receptor NPS

It was not pure NPS (NPS was included in the algorithm but only a small factor). It was more like "give each note a value based on how close it is to the next one on the next arrow/hand/overall, then sum everything and take the highest consecutive X notes, add factors for stamina/consistency/NPS"

I did a bunch of playing around with the factors and scales but I would always end up with either long streamy files being rated way too high or big spiky files being rated way too high (or both.)

Re: Entropy Gain for per-receptor NPS

What do you think of the rate at which that distance value changes ? Maybe also confined to a certain timeframe and averaging on that ? If the rate is high, then you have a spiky/bursty section, if low then the difficulty is pretty constant. Then with the actual min/max distance you can get a better idea of how drastic the spikes are or how fast is the constant section.

Re: Entropy Gain for per-receptor NPS

I'd need to experiment with it to get a definitive answer. I can see the value in having something like that, but it would be difficult to separate actual spikes/bursts from just natural variance in patterns (take a staircase for example: there are gaps of 5 notes between every left arrow, but only 1 between (some) down or up arrows, so the down/up arrows look much harder than the left/right arrows, and this could produce odd results for a difficulty change rate value. Would probably have to look at average difficulty over a short period of notes and use that to determine the difficulty change rate.

Re: Entropy Gain for per-receptor NPS

Sorry, I misread your original statement I quoted before. You can ignore that part.

Re: Entropy Gain for per-receptor NPS

I'm not sure, why would you want to seperate those ? It would help me understand if you defined spikes as opposed to natural variance. Lets say we use the distance metric and focus on the up arrow of a long staircase: that receptor is now essentially receiving minijacks of 2 notes (seperated by 1 note on right) every 4 notes. The rate of change can then be computed like this:

up, ,up, , , ,up, ,up
__, ,0 , , , ,-2, ,2

vs

ri, , , ,ri, , , ,ri, , , ,ri
_, , , ,0, , , ,0, , , ,0

(changes between 0 and 1 have been normalized to the opposite of their inverse: 0.5 => 2 => -2)

Takes a minimum of 3 notes to have a variation in distance. While it's true that the average is the same (0), you could maybe take the range between the minimum negative value (biggest deceleration) and the maximum positive value (biggest acceleration).

Deceleration doesn't affect difficulty, don't forget that this is a per-receptor metric. A file starts at 0 difficulty with 0 notes. If you put a jack at x speed, then after a few notes its speed changes to x/2, the only problem is going from 0 speed to x speed, not from x to x/2. Gradual acceleration/deceleration aren't considered in this but you can get a primitive for it using this same concept. So, for the example of the staircase, if we discard the negative values, we get a max range of 2 on up and down, and a max range of 0 on left and right. And you dont aggregate those in any way because the min/max on each receptor is important.

Does that cover the type of example you had in mind, Renegade ?

Re: Entropy Gain for per-receptor NPS

So, on spikes vs natural variance: what I mean by "spike", at least in the context of saying that my old algorithms would rate spiky files way too high, was files such as ABCDEath or TTE which have one disproportionately note-heavy section that overshadows everything else in the file. When I say "natural variance", I mean that some arrows in a long pattern like a stream, jumpstream, or staircase will be harder to hit than others.

What I'm trying to avoid is to see a staircase, get a max range of 2 on up or down as you described, and falsely claim that the staircase is a spike when in reality, it's just a staircase. Whatever metric that is used to determine the rate would have to be able to tell the difference.

Re: Entropy Gain for per-receptor NPS

There are a few different points in this so I'll try to dissect it as clearly as possible.

You have to have some mathematical definition of your concepts if not used for visualisation only. For example, a spike would be a sudden high density x of notes, at least to my understanding of your description. In a more formal way you could say it's any section with high acceleration (lets use a trivial number like 4). Also btw my metric isn't totally correct for another reason, I'll post a fix to it.

So you then have a trivial definition of a spike. With that, you want to avoid cases where the spike is short (i.e. in a staircase, the two ups or two downs) and constant (the staircase goes on for some time like 2 measures). The reason it's trivial is because first of all there is a trivial threshold to set and also because the length of said spike is not well bounded.

You mention TTE. Take TTE's fastest spot (a rolly burst like 123412341234) and remove everything before it. The acceleration from nothing to that is equal for each receptor, so min = max = x. Now take a staircase 123432123432 with the distance between two up arrows being equal in this and the roll (from a per-receptor perspective, that is most definitely fair). From nothing to it, 2*min = max = x. It would seem that both are identical, however for the comparison to hold the total nps of the spike will be lower on the staircase than on the roll (the amount of notes bewteen the fastest consecutive notes per-receptor being 1 for the staircase and 3 for the roll). Therefore, a distinction Should be made naturally but the spikiness (again, per-receptor!) will be the same according to the trivial definition.

EDIT:
Just to be extra clear, I'll point out that what you refer to as a spike as we all know it is easily defined when using all notes (not per-receptor). There's a quick increase and decrease in the nps of the section and that's it. That metric can be useful, but it's not what I was explaining/arguing in the previous few posts.