A new beginning for our SM community.

[Reincarnate]

Re: A new beginning for our SM community.

@Reincarnate - Everything you said about the system is exactly what it should be. All though you can't have it be automatic. At this point in time, most people could do more difficult songs relatively easier than some simpler songs for whatever reasons. I don't see the problem with manual rating any songs that come in after the system is in place.

On another note, is the SM for this going to be a build off of 3.9, 4.0, or what... And as for a theme, is there anything in talks? Or is it all still talk of having the site up etc... first?

I don't understand what you mean when you say it can't be automatic. I'm operating under the assumption of having difficulties stored external to the stepfile itself. As people played files more and more, difficulty ratings would adjust in real time based on performance. Bayesian weighting will remove initial variance-swing factors as well.

If a "simple song" is still tough to do, then it's that much tougher to do and the weighting will take it into account. The beauty of an objective rating system is that it doesn't matter what people "perceive" a file to be in terms of difficulty.

[Oni-Paranoia]

Re: A new beginning for our SM community.

I don't understand what you mean when you say it can't be automatic. I'm operating under the assumption of having difficulties stored external to the stepfile itself. As people played files more and more, difficulty ratings would adjust in real time based on performance. Bayesian weighting will remove initial variance-swing factors as well.

If a "simple song" is still tough to do, then it's that much tougher to do and the weighting will take it into account. The beauty of an objective rating system is that it doesn't matter what people "perceive" a file to be in terms of difficulty.

Automatic was probably the wrong word to use, but yeah. I don't think a files intended difficulty should change for any reason if it's rating is graded correctly in the first place. If a song is a 78, then it's a 78 even if 2 years from now two thousand more people can pass the song, it's still should be a 78.

By the way, let me see if I'm getting this right. 10 beginners fail a song on beginner, song get's rated higher? OML get's AAAed by 50 experts and it's rating get's lowered... The easier song seems harder to pass because no experts play it?

[Reincarnate]

Re: A new beginning for our SM community.

Automatic was probably the wrong word to use, but yeah. I don't think a files intended difficulty should change for any reason if it's rating is graded correctly in the first place. If a song is a 78, then it's a 78 even if 2 years from now two thousand more people can pass the song, it's still should be a 78.

By the way, let me see if I'm getting this right. 10 beginners fail a song on beginner, song get's rated higher? OML get's AAAed by 50 experts and it's rating get's lowered... The easier song seems harder to pass because no experts play it?

Ah, I see what you mean. I technically think that difficulty should change with the times. What was considered hard two years ago is quite a lot easier now. If a song is a 78, I think it doesn't mean much for it to stay a 78 even if people start demolishing it -- otherwise you have to start layering on more numbers. I don't like systems where we initially say "10 is the hardest. Wait, make that 11. JK it's 12. Ah hell, this beast is a 22." Past that point it gets too messy and difficulty metrics lose their meaning.

In other words, pretty much this:


Re: Your last point: No, that's what the system is actually meant to guard against. It takes into account who plays the song and how well they do on it. The end result is a market-weighted metric that says "On average, given a completely randomly-drawn player, this is how hard you can expect this file to be, and thusly it will tell you something about what kind of score you should expect for yourself" which is what you want an ideal difficulty ranking to convey.

[Charlo]

Re: A new beginning for our SM community.

I really like this idea. I had just started playing SM as the "glory days" were waning, so something like this is exciting.

[who_cares973]

Re: A new beginning for our SM community.

Thirdstyle difficulty rating

[Reincarnate]

Re: A new beginning for our SM community.

TS doesn't implement it properly

[BeatD0wn]

Re: A new beginning for our SM community.

I'd be up for this! :O

[Bluearrowll]

Re: A new beginning for our SM community.

So just to clarify on what Reincarnate and Oni have stated;

Basically set up a default difficulty 1, and 100, and then for the songs in the middle, set up some form of program algorithm where, based on SMO scores the mean score will dictate what the difficulty is?

Example: Say there's 100 songs played, and AOAO happens to have the 12th lowest raw score. This would mean it is a difficulty 88, because the program will weigh it out vs the other songs.

Would this be difficult to create? It sounds like it would work great if it could be done, but it would have to be embedded in the new client, and it would also make the work that I would be doing (manually changing difficulties) obselete.

[Reincarnate]

Re: A new beginning for our SM community.

There are problems to all sorts of rating systems. Subjective ratings are obviously biased because they may not be accurate and are prone to judgment errors (FFR tries to use a subjective system, and if you actually crunch all the data, you see all sorts of problems where people are dominating, say, a low 7 but sucking on a high 6).

Going by pure performance alone is misleading due to selection bias. Sometimes only sucky players will play a particular song. Sometimes only good players will pick a particular song.

Counting failures in your scores will lead to errors where people are just quitting early because they get bored (or want to play something else) and so on. Best to count passes-only.

This is why you also need to account for *who* is playing your file. So how do you assess who's playing your files? You need a skill metric that objectively defines who owns and who sucks.

So how do you tell who's good and who isn't? % performance is limited as it doesn't tell you WHAT songs you're good at -- so we need to know the difficulty of the files being played.

So you've got a bit of a circular problem. You need to know how hard the files are, but to do this, you need to know how good the players are playing the files -- but to assess how good they are, you need to take into account how hard the files are they've owned, etc.

Which is where Bayesian weighting and/or bootstrapping comes in. NPS would be decent for this. In other words, "load up" each file with a difficulty ranking based on its NPS alone. Now you have something to get a decent estimate for how good a player is (so even if you play only easy files, your performance metric will indicate that you're mainly getting your high % from low-NPS files).

Although NPS would merely be a bootstrap for loading initial values. After that you'd let the skill+performance feedback loop run its course and allow difficulties to shift to their natural places (which ideally wouldn't be far for most files that scale true difficulty from NPS, but perhaps far for files that don't follow the gaussian distribution well).

[who_cares973]

Re: A new beginning for our SM community.

TS doesn't implement it properly

Same concept though. Something like this would be pretty sweet. Hopefully all goes well with it

[lukestepwalker]

Re: A new beginning for our SM community.

are you trying to make Stepmania Like FFr or something?

[Shikari]

Re: A new beginning for our SM community.

are you trying to make Stepmania Like FFr or something?

Nope. Trying to implement features, like scores database and TP system, to give players some motivation and competition.

[Oni-Paranoia]

Re: A new beginning for our SM community.

@Reincarnate - In earlier posts, I mentioned it would be best to use an impossible song as the hardest song in the system. What we thought was impossible years ago has become easy in some cases. So at least for now, I don't see why a couple of extremely (impossible) difficult songs go at the top.

At least this way if we do have to add numbers, it wouldn't be anytime soon (hopefully).

New Idea (criticize please)

Similar to the groove radar, what if we implemented a small graph. This graph would look similar to the results screen of IIDX. Pretty much, the graph would be....

|
|
|
| <- NPS
|
|
|__________________
Song Length

This way you can see the MAX. NPS in the song. You can see how long the song is. I guess rather than just a number representation, we could definitely and easily use a small visual representation. You would be able to identify streams, bursts etc... by the increase in spikes. on the graph. Just a thought, wouldn't be difficult to implement.

[TC_Halogen]

Re: A new beginning for our SM community.

I don't know how something like this would be implemented -- it seems really complicated. However, it's definitely something that is worth thinking about. It eliminates the need for an arbitrary rating system and gives a clear visual of when a section gets harder. However, for songs that are technically difficult and rely on difficult sections without increasing NPS very much (see a song with stream and then using that same 16th stream as jacks, or something), it doesn't prove to be entirely accurate.

[Oni-Paranoia]

Re: A new beginning for our SM community.

I don't know how something like this would be implemented -- it seems really complicated. However, it's definitely something that is worth thinking about. It eliminates the need for an arbitrary rating system and gives a clear visual of when a section gets harder. However, for songs that are technically difficult and rely on difficult sections without increasing NPS very much (see a song with stream and then using that same 16th stream as jacks, or something), it doesn't prove to be entirely accurate.

Of course there is going to be flaws and we can either...
- Deal with the flaws
- Fix the flaws
I figure we'd be able to label maybe a spot that has jacks above the heightened section. Maybe basing a system off the .sm file would be best where as 0001 0001 0001 represents a jack.

P.s. Don't get me started with mini jacks like Guillame tell or whatever. It's still a thought, mwuahaha.