Queue/Batch Discussion Thread

[AlexDest]

I need the Chik Habit file that was judged, I changed the patterns after I sent it right before the deadline.

[bmah]

I'm also requesting a bit of leeway time before the next deadline for submissions, whenever you find a date for that.

[i love you]

Originally Posted by TC_Halogen

http://www.flashflashrevolution.com/vbz/showpost.php?p=3372794&postcount=17026

Yeah, i knew you were going to say something about this since it's the only file your girlfriend sent into FFR. Anyway, I'm not going to explain everything in this thread since it'll just be a waste of my time so perhaps answering them on AIM some other time would help. Also, just to get things straight, what the judges said about her file was based on the judges thoughts, you're not going to agree with everything. There are others trying to get their first file in FFR as well not just her.

[jimerax]

Just relooked at the list, and I think 17 or above for acceptance line works pretty fine this time.

Also, need to do something about resubmissions..

Originally Posted by AlexDest

I need the Chik Habit file that was judged, I changed the patterns after I sent it right before the deadline.

I put the fixed one in the batch.

[dore]

just suggesting: files with two +s could be borderline and reviewed? This comes from a particular example from my group (Feux D'Artifice) which I think has no room for improvement and only didn't make it because of one outlier judge (+ + +. -). I only see one other file that this would apply to, so whatever.

[TC_Halogen]

I didn't say anything because it was her file, I said it because I disagreed with some aspects, especially since I actually helped with certain parts of the file. I also fixed things that I did agree with, so there's no need to make hasty assumptions when all I'm trying to do is help.

I too also said that I disagreed, just like how judge opinions don't line up with anyone else's, which is why I asked each individual judge to get input from everyone. If I was extremely desperate to get Kayla's file in game, I would have [++]'d it. But guess what, I'm actually honest and looking for her to get improvements.

It's also not a waste of time considering it's the batch discussion thread, and I took apart everything for the reason of discussion, not for argument.

I also think it's about time I make a thread that I've been wanting to make for a couple of weeks now.

just suggesting: files with two +s could be borderline and reviewed? This comes from a particular example from my group (Feux D'Artifice) which I think has no room for improvement and only didn't make it because of one outlier judge (+ + +. -). I only see one other file that this would apply to, so whatever.

Files like these should be passed around to other judges for a larger sample of rating data.

[bmah]

Wow, The Final Hour is the most hotly contested file ever. lol

[i love you]

Originally Posted by bmah

Wow, The Final Hour is the most hotly contested file ever. lol

^^

[jimerax]

Originally Posted by dore

just suggesting: files with two +s could be borderline and reviewed? This comes from a particular example from my group (Feux D'Artifice) which I think has no room for improvement and only didn't make it because of one outlier judge (+ + +. -). I only see one other file that this would apply to, so whatever.

files that 2 +'d and 3 +.'d.

Sing for Absolution (Patashu) - 16 (4/2/5/5)
Feux d'artifice (Xandertrax) - 15 (4/1/5/5)

You (judges) can discuss about the reasonability of the ratings (lower ones).

[bmah]

Couldn't I say the same about other files as well? I mean, I'm looking at several files here that were borderline from all groups, and any of them could've EASILY got in if one or two reviews from each group wasn't so picky about minor things (i.e. +, +., and +? really do make a difference here, nevermind other ratings). If you actually find an error on the judge's part (which shouldn't happen too often) which made the difference between an accept or a reject, please report it. It's your file after all.

[psychoangel691]

Originally Posted by bmah

Couldn't I say the same about other files as well? I mean, I'm looking at several files here that were borderline from all groups, and any of them could've EASILY got in if one or two reviews from each group wasn't so picky about minor things (i.e. + and +. really do make a difference here).

That was the whole point of his post. That's what I'm trying to point out he said at the end of it

Originally Posted by TC_Halogen

The point that I'm trying to make is, people are judging way too harshly on extremely subjective errors and not enough on objective errors like missed sounds, progression errors, etc.

The final hour was an easy example for him to use because he had gone through it with me before I sent it in to help look for errors. He felt the same way about some judgements in the last batch.

[dore]

I kinda feel like the best solution might be to throw out the top and bottom scores, but we'd have to have 5 judges per team at that point, and there's no way to do that without either more qualified people (which we don't really have), or increasing everyone's workload (which shouldn't be necessary).

[i love you]

Also, what was the point of deleting your post and then reposting the same thing psychoangel?

Just curious, it was rather random lol

[bmah]

Originally Posted by psychoangel691

That was the whole point of his post. That's what I'm trying to point out he said at the end of it

The final hour was an easy example for him to use because he had gone through it with me before I sent it in to help look for errors. He felt the same way about some judgements in the last batch.

Then what we're looking at here are conditionally queued files. Right now, it appears that we have no physical definition for determining what a conditionally queued file is - the exception at the moment simply appears to be only offsync files. In the past, conditional queue was based whatever Tass felt was conditional. Here, I think we need to more concretely define "conditional".

Once defined better, I hope such files will be included in a conditional list. Take note of this jx.

[psychoangel691]

Originally Posted by i love you

Wait wtf, what was the point of deleting your post and then reposting the same thing...

I didn't mean to the comp fed up and screwed something up.

Originally Posted by dore

I kinda feel like the best solution might be to throw out the top and bottom scores, but we'd have to have 5 judges per team at that point, and there's no way to do that without either more qualified people (which we don't really have), or increasing everyone's workload (which shouldn't be necessary).

The problem is there's just too vast a difference. There's no real guidelines on what should be errors and knock points off and what's not. So you'll have one judge looking at something in a subjective way knocking a bunch of points off for it and another only looking for "actual" errors.

I think maybe if there were some sort of guidelines it might help a little bit. I mean like you shouldn't knock off for song choice, layering and pattern choices (if they are correct)

It just seems there's no general standard as to what's actually an error and what's not. Which causes such a vast difference in the judging making things more difficult. I mean look at poor kjwkjw one of his files got a + but also got a - that's pretty confusing if you ask me. From a newer stepper pov it's really confusing to understand when you have 4 different judges telling you different things to know what's actually wrong and what's not. I went through everything that was said and was confused as hell on some of it. And I tried some of the changes and it made parts of the file feel awful.

[qqwref]

I have an idea about judging with 4 judges:
- Out of the four, figure out acceptance by looking at the TOP THREE scores. This way, one judge cannot ruin a file.
- Figure out whether a file makes queue or conditional queue by looking at the LOWEST score (or lowest two). This way, files that can be considered to have significant mistakes can be fixed, and files that nobody has a problem with can be used as is.

Or maybe I'm just annoyed that Famicom Selecta isn't even getting conditional queue'd... haha

[bmah]

haha, and I thought we already fixed the "one judge's discrepency ruins a file" issue

guess not ?_?

[jimerax]

If one rating is totally different from others the rating needs adjustment, but I still don't want to disregard bottom(or top) ratings completely (depends on the reasonability of the comment).

also, 4 judges/team is a lot better than 3 judges/team in terms of judgement accuracy, so we'll go with this system if possible.

[TC_Halogen]

The amount of points needed to get by needs to be dropped. 4 [+.]'s should not get rejected by any means.

[bmah]

@qqrwef: I'm not afraid to admit it, when I complain it's often due to my own files haha, don't worry about that

@jx: I figure that if we can't closely discriminate a specific guideline, which I think is quite the possibility due to the ridiculous amount of subjectivity involved, a file can be contested, and you would have to provide the appropriate reasoning for rejection. If not, there may be negotiations to change that particular file. This can lead to a lot of inquiries, but it does deal with each unique case.

p.s. didn't I mention a complaint form before? yup


p.s.2: @jx: Similarly, a drawing line can be contested, such as what Halogen just did. You can drop or raise the drawing line by however many points as negotiated. This achieves a similar result to what I've already suggested, except it's less accurate. You wouldn't be reviewing each individual file that is debated.