Queue/Batch Discussion Thread

[bmah]

P.S. Regarding files for this upcoming batch, I believe the file limit determined a while ago has not yet been set. In which case, this may be your only chance to send in more than 4 new files. Subsequent batches will have a limit of a maximum of 4 new files per simfiler, excluding resent files.

Also, I know people are already resending files, but from now on, we'd like to ask you to add "[resubmit]" as an extension to the title of song folders, for simfiles that are resubmissions from previous batches.
Since the same judge may not necessarily review the same file you've resubmitted, a list of resubmitted files will help us keep track of files actively undergoing improvement. This should help bring your hard work under more consideration from different judges. It's only fair to do so.

edit: and please, submit in a zip or rar folder...avoid using an external file-hosting site like mediafire rofl
(exception will be made for this batch only)

[i love you]

Originally Posted by bmah

P.S. Regarding files for this upcoming batch, I believe the file limit determined a while ago has not yet been set. In which case, this may be your only chance to send in more than 4 new files. Subsequent batches will have a limit of a maximum of 4 new files per simfiler, excluding resent files.

Also, I know people are already resending files, but from now on, we'd like to ask you to add "[resubmit]" as an extension to the title of song folders, for simfiles that are resubmissions from previous batches.
Since the same judge may not necessarily review the same file you've resubmitted, a list of resubmitted files will help us keep track of files actively undergoing improvement. This should help bring your hard work under more consideration from different judges. It's only fair to do so.

edit: and please, submit in a zip or rar folder...avoid using an external file-hosting site like mediafire rofl
(exception will be made for this batch only)

I support this. This would make it a lot easier for the judges to keep track of resubmitted files for future batches.

[Coolboyrulez0]

What happens to borderline files, are they played by a "supreme judge" and determined if they are accepted or denied or are they just borderline pass/fail?

Btw 2/4 accepts
1/4 borderline
1/4 reject - due to wc being subjective, and quite frankly being a dick, thats all.

[jimerax]

Just looked at the halogen's list, and yeah acceptance rates among groups are so various.

I'll investigate the judgement strictness of every judge (and quality of every part) and need to determine the final cutoff (since currently 40% acceptance rate, which is considered to be a bit too high).

also there're opinions that psychoangel691 should be backup due to the lack of experience here, so who's considered to be fine with that spot?

ps will set "submission number limit" and "resubmission rule" too.

[dore]

lol I'm gonna resend Elements of a Party. Without Tass it might actually get through

[bmah]

Originally Posted by Coolboyrulez0

What happens to borderline files, are they played by a "supreme judge" and determined if they are accepted or denied or are they just borderline pass/fail?

I've discussed this with jx not long ago, and to make it simple, we're just going to let numbers decide for us (i.e. refer to the point system) until we've figured out how to add additional variables to address finer points like these. Also, there's no supreme judge parading on the system.

@jx: my first choice would be blu, but the chances are probably slim
next suggestion would be i love you (rebound) for the spot

thoughts?

[jimerax]

Currently we have around 40 submissions (without applying submission cap).

When the number of submissions reaches around 60-70, I'll post batch close annoucement here (batch will be closed 24 hours after that).

[Plan_Bsk81127]

Originally Posted by bmah

@jx: my first choice would be blu, but the chances are probably slim
next suggestion would be i love you (rebound) for the spot

thoughts?

rebound for sure

[Silvuh]

NOTE: I've been working on this post and another analysis of the point system for about two hours ... I was going to post all the information in one post, but because there have been 12 pots since I've started typing this up... Well, ... I'll just post this part now.

The post:

By JX's suggestions (excluding the accepted + + -) all files with (11) points will be borerline. + +. - is the only borderline (10). So I will associate (15)–(12) as "accepted" and (11) as "borderline".
With everyone fussed on the low acceptance ratio, I will discuss files that may have a higher true rating. My use of "reasonable" refers to the sum altered by the standard deviation.

It's reasonable to assume that + +? ? (10), +. +. ? (10) and + +? - (9) may have a true rating of (11). However, it is unreasonable to assume the same for +. +? +? (10). Out of these three ratings, because (9) is too low, we may could include scores of + +? ? and +. +. ? as borderline.

Increasing the scope of which files are borderline is not a bad idea, because not all of these files will be conditionally queued. Reanalyzing the borderline files will create the "conditionally queued" category and a category corresponding to the old "Played by Tass; not in Queue" category.


But, the same logic can be applied to which files are accepted: if we believe a (12) should be accepted, the borderline scores of + +. ? (11) + +? +? (11) + +. - (10) (three of the current four) could be acceptable. Because we're looking for occurrences of one judge's disagreement effecting a lower score, + +. ? would be the rating most likely for a true acceptance. With + +? +?, the judge's disagreement effects a higher score, and a sum of (10) is too low for acceptance. And ... now I'm just kind of rambling.


Files in the batch with a rating of:
+ +. ?
None
+ +? ?
(2) Experiment
+.+. ?
(1) Black and White
(2) Death Traveller, Disco 80's
(3) Metropolis, Kirby 64

Responses:

Originally Posted by bmah

While this may be true, my suspicions regard the level of detail and scrutiny between various judges. That's what's fishy. Beyond quality of the files themselves, you have to inspect it this way - how files were evaluated.

I'm blaming myself here for the fishiness of my group's results... I wasn't clear a lot and I had a hard time putting into words why I thought things didn't feel right and ... yeah.

Originally Posted by bmah

Nothing but experience. Which means that let's hope that simfilers AND judges learn from each batch judgment.

I know I've learned quiite a bit already, haha...

Originally Posted by bmah

Also, I know people are already resending files, but from now on, we'd like to ask you to add "[resubmit]" as an extension to the title of song folders, for simfiles that are resubmissions from previous batches.

Alright; def. will do this from now on. I'd type which files were resubmits in the e-mail with the attachment, but being able to see the label is less work than having to read an e-mail again each time you forget details about a file...


The other stuff:
I'll also be editing this stuff into my post with my other other stuff.

"Balanced" rating:
(If one judge's score effects a lower sum, I add the standard deviation, and vice versa)
15 + + +
14 + + +., + + +?
13 + + ? , + + -
12 + +.? , + +.+., + +.- , + +.+?= +.+.+.
11 +.+.+?, +.+.?
10 +.+.-
09 + +?+?, +.+?- , +.+?+?, + +?- = +.+?? = +?+?+?
08 +?+?? , + +?? , +?+?-
07 + ? ?
06 +.? ? , +?? ? , +?? - = ? ? ?
05 + ? - , ? ? - , +.? -
04 + - - , +.- -
03 +?- - , ? - - , - - -

It may seem odd that ? ? - (5) has a higher balanced rating than +. ? - (7), but this is because the standard deviation for the latter is much greater.
To minimize this effect, I curved the standard devation.
crvstdev = stdev * (-(1.079 ^ (stdev^2)) + 2)
With this equation, the greatest standard deviation (2.309) becomes halved.

Curved "balanced" rating:
15 + + +
14 + + +., + + +?
13 + + ?
12 + +.+., + +.?, + + -, + +.+?= +.+.+.
11 +.+.+?, + +.-, +.+.?
10 +.+.-
09 + +?+?, +.+?+?, +.+?- , + +?- = +.+?? = +?+?+?
08 + +?? , +.+.? , +?+?-
07 + ? ?
06 +.? ? , + ? - , +?? ? , +?? - = ? ? ?
05 + - - , +.? - , ? ? -
04 +.- -
03 +?- - , ? - - , - - -

This shows + +. ? in the "Accepted" range and +. +. ? in the "borderline" range, which agrees with what I said earlier. So I guess I'm not so sure about + +? ?.
+ +? +? is the exception here. Because one judge is effecting a higher score, it's true rating is more likely less than 11. (Its curved "balanced" rating is 9.97, which is basically 10 ... but I truncated, so, yeah.)

[Korygon]

If we have 40 already and we just posted the email, I'd say nsane's email had to have had at LEAST 70 or 80. After everyone reads about the new email and resubmits, I figure there should be enough for the batch to close

[who_cares973]

Originally Posted by Coolboyrulez0

1/4 reject - due to wc being subjective, and quite frankly being a dick, thats all.

not everyone is going to like your file deal with it

[TC_Halogen]

Seems like you only rejected for song choice though. In this kind of judging system, giving a 1/5 just because -you- don't like the song pretty much kills any chance of the song going in.

If they want a different opinion, I can judge it; I don't know the song or chart so it'd be an entirely unbiased review.

If you feel like a hatred/pure dislike is going to get into the way of your rating and cause you to massively change your rating, it might be more worthwhile for everyone to have the song passed on. Basically, do your review and then mark the song saying that it could use a readjustment.

[AsphyxZero]

Where's GD when you need some subjectivemanias.

[Gundam-Dude]

hello

P.S. Next batch, I'll keep my notes short and brief since I'm like the only person other than wc with a brick wall of text for reviews, lmao.

[psychoangel691]

Originally Posted by Gundam-Dude

hello

P.S. Next batch, I'll keep my notes short and brief since I'm like the only person other than wc with a brick wall of text for reviews, lmao.

But brick walls are good ^_^ actually I do feel bad because I rushed through this to get it done and didn't really go into too much detail at all. At least on files that need work I think the long notes from judges are good because it helps the person learn especially if they are newer to it.

[TC_Halogen]

Originally Posted by Gundam-Dude

hello

P.S. Next batch, I'll keep my notes short and brief since I'm like the only person other than wc with a brick wall of text for reviews, lmao.

Only reason why I didn't put up walls of text is because I was trying to speed through this batch since this was from say...a year ago?

[jimerax]

Well, my comments about judgement distribution and strictness for every judge.

psychoangel - about right, "+?" and "?" could be used a bit more?
patashu - a bit too lenient in "+". "+." and "+?" could be used more.
kory - a bit too lenient in "+". "+?" and "?" could be used more.

wc - about right but should use "+?" more (corresponding to "Maybe").
v4n - ok
gdude - ok

silvuh - seems slightly strict overall, but about right.
halogen - ok
jx - ok

bmah - about right, "-" could be used more for bad files maybe?
ilu - looks like all or nothing, "+?" and "?" could be used more.
kommi - same as above, "+?" and "?" could be used more.

[Dr Tran]

totally forgot about +? lmao yeah i wouldve used that for a good portion of the ?'s

also aj i like nainai as a song but the file was just really eh for me. could definitely get at least a +. from me with enough fixes and a cut


also im permabanned on my main account so replace with this one

[jimerax]

Another news:

I just got files from Nsane, and it was 39 files.
I'll verify those with files in current e-mail and count again.



ps lifted wc from p-ban.

[who_cares973]

please note that i added one more submission to the resubmission than to what i sent to nsane