Serious Judging System Suggestion

[ilikexd]

Re: Serious Judging System Suggestion

It is still incredibly subjective. I'm sure for the most part nearly anyone who has played the game significantly can determine what a good file is when they see it.

This is demonstrably so far from the truth. What happens a lot is people putting down (but also praising, mind you) charts just based on their own personal experiences, frustrations, and, at times, whimsical opinions. It's especially bad when they have no experiencing making charts. It really takes a lot of experience and a good mindset to be able to judge charts.

What needs to happen is the community as a whole should form some method of objectifying the specifications of what kind of things are good or bad in a simfile.

Although actually something like this would be really cool to see.

[Izzy]

Re: Serious Judging System Suggestion

An approval process similar to osu!'s would be a practical approach.

Which is what? I'm not familiar with this game.

[Rebirth0]

Re: Serious Judging System Suggestion

Care to say which ones, AND explain why? I would say that the quality has been increasing as people have been getting better and more experienced.

If you seriously want me to, I could list some files. I haven't played a ton of the newest files, but I know some of them felt dumb. I could spend some time later to go over a lot of the files from the last year. It'll have to be later today or something since I'm about to leave

Yes, some did. And some went the extra mile. The problem isn't with the judges or system as some whole, it's that there was a coinciding of numerous personal failures and the fact that a lot of people had to put stuff off for final exams; a few people were behind, which motivated others to not worry about being on time because they thought they wouldn't be the last ones, and a couple just straight out did nothing -- it's their personal fault and it's being dealt with... it's not the system or the workload, as can be demonstrated by the fact that other judges did everything well and within the timeframe.

I know some of the judges did the job like they should, and it is the personal failings of certain judges that blew this up. But it is also true that lowering the amount of submissions makes it even less likely for that to happen.

[i love you]

Re: Serious Judging System Suggestion

"lol"

That smugness is really appealing to no one as it's known that you were a significant cause of the troubles with last batch.

Just the miscommunication part and the part of me losing my notes and not saying anything about it until two weeks later. Not the batch as a whole. Every judge but one submitted their notes late. I just so happened to be called out for something I did wrong on my part.

Also, I had a good laugh at your "smugness" comment.

[cedolad]

Re: Serious Judging System Suggestion

Which is what? I'm not familiar with this game.

http://osu.ppy.sh/wiki/Beatmap_Approval

I'll try my best to explain how it would work for FFR. Since we don't have a client-based game and instead flash/browser based, it's harder for us to do this method. My idea was to have all submissions uploaded to an "Approval Engine", where the community goes and play them; giving notes, suggestions, comments etc. Then when a chart has enough of whatever (thumbs up's, stars or whatever), it's sent to a judge where s/he then approves it for ranking.

It gives the community a chance to have a say in what they find acceptable, and includes them in the process. It also takes a huge work load off of the judging crew, because by the time they see a chart, it will have gone through numerous amounts of changes and all they would need to do is fine polish it and approve it for ranking.

[Izzy]

Re: Serious Judging System Suggestion

Yea, that sounds like a great idea to me. The community really needs to be a bigger part of what is good or bad and not the judges.

It would be cool if you could upload any of your own simfiles to your profile and have people play them outside of ranking. If there were enough votes or plays on a simfile there then maybe it could just get auto pushed to ranking.

Honestly anything to remove judges from the equation seems like a good idea to me.

[cedolad]

Re: Serious Judging System Suggestion

Of course revamping the current system to this system would require a lot of work. The approval process in itself would require a revamp on the main site as well. Such as a "pending charts" page, where it would be similar to say the old Simfile DB, or the Video DB, with a star ranking and comments section.

[ItsOnlyDanO]

Re: Serious Judging System Suggestion

The osu idea would be good, but a problem could occur that some newer players may not understand what charts are good or bad, thus affecting ratings. This could be fixed by only allowing experienced players to rate the charts however.

[Charu]

Re: Serious Judging System Suggestion

The osu idea would be good, but a problem could occur that some newer players may not understand what charts are good or bad, thus affecting ratings. This could be fixed by only allowing experienced players to rate the charts however.

This is deja-vu...

I still think we should do a system where someone submits, puts it up somewhere so Judges can take a look at it at their own pace and have some non-judge feedback as well.

idk tho

^ going off of your idea Mr. Snivy

Here is a random new judging system idea:

- Have your full judging team of 12 or whatever
- They can judge files whenever they want at their own pace
- Every time a file is submitted it is put in the list and available for download via cloud for judges only.
- A file needs 4(ish) judges to be passed / accepted or whatever. The files might need to be judged in order of submission?


File Name Here (4/4) - 8/7/7.5/5
Other File Here (4/4) - 6/5/5/5
Here is another file (3/4)
One more file (3/4)
Files lol (2/4)
Yet Another Fiiiile (0/4)
Really crappy file by Fission (0/4)


*(#/4) represents judges who have judged the file so far

- Final ratings will be posted in OP only after all ratings are in
- There could be two different groups, but I'd say no more then that.
- That way, you can have the lazy judges judge whenever they feel like it, and the judges who actually do their job keep on trucking like the badasses they are



(Tell me why this wont work, or why it would. I'm actually curious to know how badly this sucks, or how it might be something to work off of lol)


TL;DR: Chart by chart judging system

[cedolad]

Re: Serious Judging System Suggestion

It would be cool if you could upload any of your own simfiles to your profile and have people play them outside of ranking. If there were enough votes or plays on a simfile there then maybe it could just get auto pushed to ranking.

Having the ability to upload your own file to the "Approval Engine" would be a really great idea, unfortunately I think that would take a ton of work and huge overhaul would be in order to do something like this.

Honestly anything to remove judges from the equation seems like a good idea to me.

The implication of the new system wouldn't necessarily disband the judging crew. It would just, in a sense, lessen their work load. Instead of playing through tons of files looking for technical errors, they can instead spend time subjectively looking at a file for playability, and it's a replay value. And that's only if a file has met certain criteria that the community as a whole has set.

[ItsOnlyDanO]

Re: Serious Judging System Suggestion

But the experienced player doesn't have to be a judge, you don't have to be a judge to know what a good simfile is.

[Izzy]

Re: Serious Judging System Suggestion

The implication of the new system wouldn't necessarily disband the judging crew.

Right that is how it would likely be, but that isn't my personal philosophy on the way it should be.

[i love you]

Re: Serious Judging System Suggestion

Well, keep in mind how ridiculously huge and active osu! is in comparison to FFR. The osu idea on here would not work so well on FFR because we are such a small community and most of us probably does not have the time or patience to keep up with checking every single simfile that gets published within the site. We also have limited bandwidth (from what I have heard) so it is not like we could constantly have new threads being made with majority of them being published files. We simply will not be able to keep up with the pace. Hell, most of the judges recently (myself included at the time) had problems meeting the judge deadline.

We could probably apply certain aspects of the way osu! does their judging system but it would have to be carefully done and tested.

[TC_Halogen]

Re: Serious Judging System Suggestion

Yea, that sounds like a great idea to me. The community really needs to be a bigger part of what is good or bad and not the judges.

It would be cool if you could upload any of your own simfiles to your profile and have people play them outside of ranking. If there were enough votes or plays on a simfile there then maybe it could just get auto pushed to ranking.

Honestly anything to remove judges from the equation seems like a good idea to me.

I've been a huge fan of how osu! managed their game; when a file was approved by the osu! panel, it became ranked and people could compete for good scores on it and whatnot, but files that were not ranked were still easily downloadable and easy to share. It would be absolutely incredible if there were a three-step process where a user would upload a file, get it QC-checked (assure that the basics of sync/following the music appropriately were followed), and then get it judged for actual ranking (similar to the judgment process now). Even if a file weren't to make the cut, it could still be played by other people.

The unfortunate thing about it is that implementation of this system is extremely difficult given the extremely high amount of server space that would be required to even put a majority of these files through. osu! literally allows you to submit a file meeting their guidelines through the game, regardless of the quality of the map itself if I remember correctly. Manpower is also an issue.

EDIT: pretty much ninja'd.

[cedolad]

Re: Serious Judging System Suggestion

But the experienced player doesn't have to be a judge, you don't have to be a judge to know what a good simfile is.

Well yeah essentially. The point of keeping the judges around wouldn't be to say "oh hey this a good file" or "oh hey this is a terrible file", but instead give useful criticisms (if any at all), on how to make a file more fun, or help apply theories where the average step artist didn't exactly apply them or apply them correctly. All in all the judges would be more for bringing out more quality from a file instead of looking at technical errors and off-sync files.

The average experienced player would be the ones who say that a pattern didn't exactly portray a certain aspect of a chart correctly, or that there is a misrhythm. Or the file is off-sync.

Right that is how it would likely be, but that isn't my personal philosophy on the way it should be.

To each their own, I understand why you wouldn't a judging crew, but it's inevitable and they're really helpful for up and coming stepartists.