Queue/Batch Discussion Thread

[TC_Halogen]

Sent in:
The Crow's Newspaper / dBu (10)
Strikebeam / Waterflame (8)

[customstuff]

Glad to see people are using the Waterflame permission I got
I stepped half of -Radioactive- so I might finish that for either this batch or next batch. Most likely next batch because I'm lazy

[qqwref]

Some ideas for judge system reform... The general stuff:
- Get rid of [<] completely, it has no use in a purely points-based system. If one song is clearly better than another, the points will show that. If the songs end up tied pointswise (and are both good enough to get in), we can have the judges vote on which one gets ingame, or else have the two stepauthors combine their efforts and put the file in as a collab.
- If a song has multiple difficulties, have whoever's in charge of the email fix it (for instance by asking the stepauthor which one they want to have judged) before the judges get hold of the batch.
- If you want fewer files in the queue, don't accept fewer files, but instead have batches spread further apart. The files that were going to be submitted anyway will slowly grow over time, but the files that stepartists make specifically for the batch (to have higher chance of getting a song ingame) won't.

More specific stuff on judging:
- Have judges give TWO point ratings for each song. Same 5-point system, but there should be the straight up rating (how good this chart is NOW) and the edited rating (how good this chart could be if this stepauthor listened to the judges and made minor changes). Many charts would get a + for the edited rating, and that's fine - it just means that any problems you see are easily fixed. EDIT: Keep in mind, these ratings shouldn't be too far apart, unless the judge thinks that a few small changes would make the file WAY better.
- Alright, so, the EDITED rating should determine acceptance. Since we only want good files ingame, we should require at least 5/5/4/0 or even 5/5/5/0 for four judges. The 0 means the lowest judge is ignored (possible subjectivemania issues).
- Now, if those top 3 judges (top 4 if there is no clearly lowest judge) gave the same straight up and edited ratings to this chart, it means it's perfect and can be put straight into the queue. Everything else can be easily improved (to the point of being a clear good chart), and thus should be dumped into the conditional queue.
- For files in the conditional queue, the stepauthor can submit revisions, and when they do the judges for that file should (eventually) look at them and see if the minor errors have all been fixed. When the judges for that file agree that they can't see any more minor errors (not counting those that the stepauthor disagrees with), then the file is good enough for public release and can be put in the queue.

[condoct]

Sent in Wadatumi/島白 (Shima Jiro)

[Patashu]

seconding getting rid of < especially since aperson wants to move towards fully integrated multiple-charts-per-song in the engine, and anyone with some brains should too

[who_cares973]

I don't think I've ever used <

Sending in fixes for guitar sounds and resubmitting everything went numb and submitting connect ~speedy mix~ maybe now it won't be so boring

[i love you]

Originally Posted by qqwref

http://www.flashflashrevolution.com/vbz/showpost.php?p=3383831&postcount=17403

Yeah, i definitely agree with the removal of [<] simply because of the things qqwref mentioned. I also think that some of these suggestions he mentioned should be thought over rather than ignored. I'm really liking the idea of the two point rating system seeing that it would most likely encourage people to resubmit their files if it has something minor/major wrong with it.

[who_cares973]

oh **** ignore connect i sent the wrong one

[dore]

It's still nice to have [<] in case two files receive the same rating but one file is still superior to the other

[ChessWizard]

Sent in -The Show- by Waterflame

[bmah]

Originally Posted by qqwref

- Get rid of [<] completely, it has no use in a purely points-based system. If one song is clearly better than another, the points will show that. If the songs end up tied pointswise (and are both good enough to get in), we can have the judges vote on which one gets ingame, or else have the two stepauthors combine their efforts and put the file in as a collab.

Absolutely. [<]/[>] can be used to illustrate a comparison in your notes, but I don't think it should be an actual parameter for the file's overall grade.

Originally Posted by qqwref

- If a song has multiple difficulties, have whoever's in charge of the email fix it (for instance by asking the stepauthor which one they want to have judged) before the judges get hold of the batch.

I've already been discussing with jx modifications to this rule, although not quite in the way you're describing it. What I've been thinking of was that multiple difficulties in one file should be only allowed if a) it was that simfiler's intention to do so, and b) it is noted. Sometimes we might want to make two or more difficulties for the same song.

However, without notification, or if multiple difficulties were accidentally left in, the file will still be rejected. The original intention of this rule was to cull the list of submissions, pretty much.

Originally Posted by qqwref

- If you want fewer files in the queue, don't accept fewer files, but instead have batches spread further apart. The files that were going to be submitted anyway will slowly grow over time, but the files that stepartists make specifically for the batch (to have higher chance of getting a song ingame) won't.

This is all really dependent on the number of submissions in a given batch, plus the time it takes to finish one batch. I do agree that spacing out batches a bit more will spread the rate of files pouring in - although each batch might grow larger as a result of longer waiting times between each batch.

Originally Posted by qqwref

- Have judges give TWO point ratings for each song. Same 5-point system, but there should be the straight up rating (how good this chart is NOW) and the edited rating (how good this chart could be if this stepauthor listened to the judges and made minor changes). Many charts would get a + for the edited rating, and that's fine - it just means that any problems you see are easily fixed. EDIT: Keep in mind, these ratings shouldn't be too far apart, unless the judge thinks that a few small changes would make the file WAY better.
- Alright, so, the EDITED rating should determine acceptance. Since we only want good files ingame, we should require at least 5/5/4/0 or even 5/5/5/0 for four judges. The 0 means the lowest judge is ignored (possible subjectivemania issues).
- Now, if those top 3 judges (top 4 if there is no clearly lowest judge) gave the same straight up and edited ratings to this chart, it means it's perfect and can be put straight into the queue. Everything else can be easily improved (to the point of being a clear good chart), and thus should be dumped into the conditional queue.
- For files in the conditional queue, the stepauthor can submit revisions, and when they do the judges for that file should (eventually) look at them and see if the minor errors have all been fixed. When the judges for that file agree that they can't see any more minor errors (not counting those that the stepauthor disagrees with), then the file is good enough for public release and can be put in the queue.

You have an interesting idea, but I'm not sure how this will turn out.

I think making a second judgment on one file can get quite murky. I might just be a more visual person, but it's not as easy to review a file you're envisioning the corrections (furthermore, it's easier to be lazy with very general ballpark guesses for the second rating). As a result, it's rather easy to provide a second rating that's not well thought-out.

Your idea is a huge shift in the judging system that resubmissions would probably be unnecessary. Partitioning files to queue/resubmissions/conditional queue (or even just two of these) can become complex, as I've found out with the previous batch. For this idea, I assume you want the same judges who reviewed a file to look at the same one in the conditional queue. It becomes a mess to keep track of which judges review what file for past submissions. How much one has to keep track of files also depends on the number of submissions we get.

-----

On a side note, I attempted to revise the details on resubmissions, because having specific judges keeping track of specific files did not end well last time.

1) Resubmissions should be labelled in the song folder.
2) If you have an issue with judging on a resubmit, you must let us know. We will not look at any file if you do not let us know you have an issue.
3) We will not consider complaints on judging with the point equivalent below +.,+.,+?,+? (I believe that's 14 pts)

- resubmissions will not necessarily go through the same judges (this eliminates keeping track of specific judges to review specific files)
- after 2-3 (dunno yet) rejections, contact me or jx and we'll sort out the file (this would hopefully stop a potentially vicious cycle of a set of judges wanting certain changes, and then another set of judges disliking previous changes, resulting in neverending rejection)

[Ziergdsx18]

Sent in Cest What - Terminal 11

[customstuff]

If I resubmitted Phalanx a while ago, without the Resubmit tags... would that be a problem? :s

[bmah]

How long ago since your previous submission of that song? Right now, resubmissions refer to files that were sent from the previous batch, and not any earlier than that.

If you forgot to send the file with the resubmit tag on the song folder, just resend it with proper labelling.

[customstuff]

Originally Posted by bmah

How long ago since your previous submission of that song? Right now, resubmissions refer to files that were sent from the previous batch, and not any earlier than that.

If you forgot to send the file with the resubmit tag on the song folder, just resend it with proper labelling.

It was from the last batch, but I sent it directly after before it was said to put resumbision tags. I'll resend it then.

[NocturneAunamic]

sent in rising action - docmo

[qqwref]

Originally Posted by bmah

However, without notification, or if multiple difficulties were accidentally left in, the file will still be rejected. The original intention of this rule was to cull the list of submissions, pretty much.

I judged a file in the last batch with two difficulties. I'm not so strict that I'd auto-reject something because of that, so I just chose one and reviewed that. But I think if we had something in place to fix this it would be less confusing.

Originally Posted by bmah

I think making a second judgment on one file can get quite murky. I might just be a more visual person, but it's not as easy to review a file you're envisioning the corrections (furthermore, it's easier to be lazy with very general ballpark guesses for the second rating). As a result, it's rather easy to provide a second rating that's not well thought-out.

Right, but I don't think it really needs to be all that seriously thought out (remember this is a 5-point system, so not very accurate anyway). I guess what I just mean is that the judge should think about whether they are giving a low score because of systematic problems throughout the file, or because small sections or technical errors hurt the grade a lot. It would be a little more complex for sure.

Originally Posted by bmah

Your idea is a huge shift in the judging system that resubmissions would probably be unnecessary. Partitioning files to queue/resubmissions/conditional queue (or even just two of these) can become complex, as I've found out with the previous batch. For this idea, I assume you want the same judges who reviewed a file to look at the same one in the conditional queue. It becomes a mess to keep track of which judges review what file for past submissions. How much one has to keep track of files also depends on the number of submissions we get.

Yes. It would be less work for the judges, no doubt about that. (The same judges don't NEED to review, but it's important that we don't end up with different judges wanting totally different things.) And keeping track of files is no problem with a little spreadsheet or text file. Thing is, I think the current system, including the new submission cap system, is biased towards the good stepartists (and towards their STYLES of chart, more importantly) because any file that doesn't get in pretty much has to wait a few months to get fixed, which a lot of people won't want to bother with. And almost all files that get accepted (and the bar is set very high for that) get put ingame without changes, which is a problem when judges are actually seeing errors in the file. My idea should lead to a greater diversity of songs, stepartists, and styles of chart, without actually letting bad charts into the game.

[i love you]

Originally Posted by qqwref

Yes. It would be less work for the judges, no doubt about that. (The same judges don't NEED to review, but it's important that we don't end up with different judges wanting totally different things.) And keeping track of files is no problem with a little spreadsheet or text file. Thing is, I think the current system, including the new submission cap system, is biased towards the good stepartists (and towards their STYLES of chart, more importantly) because any file that doesn't get in pretty much has to wait a few months to get fixed, which a lot of people won't want to bother with. And almost all files that get accepted (and the bar is set very high for that) get put ingame without changes, which is a problem when judges are actually seeing errors in the file. My idea should lead to a greater diversity of songs, stepartists, and styles of chart, without actually letting bad charts into the game.

This is definitely an issue that needs to be fixed. It would require a bit more work but at least it would reduce a lot of the issues we are having right now. 1+ Support

[kommisar]

Well files that need fixes are obviously going to wait longer to be actually placed in the game, since they pretty much need to be judged twice. It's equal chance for everyone imo.

[iironiic]

Just wondering. Can we fix a file that's currently in queue?