Re: Queue/Batch Discussion Thread v2
 

I've been trying to address the issue for a while now. It was the reason why I had the idea of resubmissions being re-judged by the same people. But this ultimately ended up being a failure, because few judges actually reviewed those resubmissions. An ideal solution, but keeping track of resubmissions this way was inconvenient*. So now it's reverted back to the system where resubmissions can be reviewed by any judge, whether they've played it before or not. It's a trade-off between handling the batch more easily vs consistent judging of resubmitted files.

To make up for this, I came up with appeals (which apparently isn't recognized as an official idea yet) if one has been rejected multiple times due to contrasting opinions of the judges (as opposed to technical errors).


*I might take a look at the idea again. I don't think the idea was executed well (resubmits weren't brought to the attention of judges very much), but a revision might make this work.

Re: Queue/Batch Discussion Thread v2
 

Ok I made a poll for that issue.

- Previous judges should judge the resubmission files again (but only when all judge members are the same with the previous ones.).

Y/N

If majority of judges/step artists here support this, I'll change the system.

Re: Queue/Batch Discussion Thread v2
 

I have mixed feelings about that.

Re: Queue/Batch Discussion Thread v2
 

I don't think it's necessary imo


oh nuts i get to judge two of my songs lmao. hi wyde

Re: Queue/Batch Discussion Thread v2
 

Shouldn't this be 'replaced with the new one'? Should probably fix this. Just throwing that out there.

Re: Queue/Batch Discussion Thread v2
 

fixed.

Re: Queue/Batch Discussion Thread v2
 

More importantly, I'd like to hear from the people who are complaining about their files being judged with highly variable results and possible contradictions in opinions. qqwref? rushy? ichlie? Etc. It's the only reason why I made such a proposal. Read the previous pages to hear their concerns.

Re: Queue/Batch Discussion Thread v2
 

Not an important thing or anything; just wondering why they are not just [<] and [>].

Re: Queue/Batch Discussion Thread v2
 

'sup

Yeah funny you got landed with that lol. hope u liek

Re: Queue/Batch Discussion Thread v2
 

I'm not sure if I agree that previous judges should judge the resubmission files again. On one hand, it would ensure that any fixes that got pointed out actually get fixed; on the other hand, if one person thinks the song or chart style is total crud, they are not going to improve their opinion much when they see that a few minor problems have been changed. I guess you can put me down for a tentative yes, but I'd still like to see the worst grade thrown out in cases of extreme subjectivemania (+ + + -, ++ + + ?, that kind of stuff).

Re: Queue/Batch Discussion Thread v2
 

Throwing out grades is a bad idea in the fact that a file might be objectively incorrect, but will have passing grades for multiple judges anyway for various reasons (judging fatigue, extreme bias toward song choice when chart is passable but not worthy), etc.

tbh, Arch's ratings were a bit all over the place. Hell, even I thought +. was a bit ridiculous for my chart Cheeseburger last batch -- that file shouldn't have gotten anything higher than a +?, lol

If the song has a poor rating, it should only be considered thrown away if the judge can't give legitimate reasons as to why. Obviously if a judge gives a bad grade, they've most likely done it for a reason and if they can prove it, then their grade shouldn't be thrown out because it's rude to them for judging it.

Take me for example in my last batch, and Altair (which I believe I gave it a [-]). What if it received good marks from all of the other judges? My notes get thrown away for no reason?

It's just a thought; it's not a bad idea, but it comes with a lot of issues behind it.

As for resubmissions: pass them around to different judges, because there might be an inevitable bias towards/against songs.

Re: Queue/Batch Discussion Thread v2
 

I think we should all go play o2jam. Just sayin

Re: Queue/Batch Discussion Thread v2
 

It's possible, but I think it's a lot more likely that one person messes up than that three people mess up. Anyway I don't think the notes should always be thrown out, or the judge booted, just that the lowest rating shouldn't be counted when we determine how well songs did.

This is a good idea - to ask the lowest judge (if that rating is the lowest by a lot) to explain in more detail why they gave that ranking and what could be fixed. Then we could throw out their result only if they can't give a reasonable explanation.

The thing which makes me want to throw out the lowest rating is that the current system can potentially give a judge too much power. If one person gives a file [?] or worse, there is nothing the other judges can do - it's basically getting flat out rejected. The purpose of team judging is to give each file multiple perspectives, but it's important to have a system where a judge can't personally prevent a file from getting ingame.

Re: Queue/Batch Discussion Thread v2
 

Divide the judging team into two groups (instead of three), and for each file, omit the highest and the lowest score

Re: Queue/Batch Discussion Thread v2
 

You pretty much said it for me in this, but

This should be considered too. If the judge cannot give good solid reasons as to why it received a lower rating, then it really shouldn't be considered. Pretty much means that they didn't do their job [in the sense that they should be pointing out mistakes or flaws].

Put me down for a yes like qqwref.

Re: Queue/Batch Discussion Thread v2
 

Does it surprise you that someone whose position is that simfiles cannot be objectively evaluated would not have an objective method of judging? It is objective with respect to my charting style, but that's as far as anyone can go. A method is not objective just because it is systematic.

Re: Queue/Batch Discussion Thread v2
 

I would have +'d cheeseburger if I was judging it. idg why you hate it. I guess the <3 of exotic swing is lost when you transfer it into ffr though

Re: Queue/Batch Discussion Thread v2
 

+100 for this.

Re: Queue/Batch Discussion Thread v2
 

increasing judge's load by a lot = no

Re: Queue/Batch Discussion Thread v2
 

I guess I can say it does surprise me a little bit. Truthfully, there's nothing really erroneous about the chart rhythm wise, but the overall structure is a bit off.

It really is lost...you can AAA the entire song just using the 96th swing as 16th runs; kinda sad really.

Backstory: The chart was actually rushed for a tournament that happened on FFR, as was Strikebeam, AND The Crow's Newspaper (although TCN was the first chart I worked on; as a result, it came out the best out of the three). I created the final rounds for every division and put them on the DF engine, and then pulled them off once the tournament was over. I don't hate my chart, it's just not properly structured -- more freeform than anything. But hey, it's cool to see that you liked it :)