Queue/Batch Discussion Thread

[Coolboyrulez0]

Originally Posted by Plan_Bsk81127

FFR needs more jumpstream and chill out on the bursty dnb songs for a bit. lol.

Agreed. We also need more songs in VD-VC range imo. (and in the extremely easy range)

[Izzy]

Originally Posted by Coolboyrulez0

Agreed. We also need more songs in VD-VC range imo. (and in the extremely easy range)

I would have to agree as well. I find 7-10's a lot of fun. Those are the kind of songs I like playing in multiplayer.

I think one of the problems is that most of the people stepping songs are stepmania veterens who are really good at the game. A song isn't stepped right unless you have a note to every noise in the song.

[TC_Halogen]

Originally Posted by bmah

The only reason I can think of is that perhaps decisions may have been finalized based off of Halogen's earlier post, which in this case I do notice a discrepency (noted as +.,+.,+. = 12pts). I'd like to have jx confirm this.

Human error sucks, I know.

Did I do something wrong with how the ratings were accumulated? From what I see here, everything looks fine. =/

[bmah]

Read 0's post. Patashu rated the song as a [+]. So it should be 13 pts.
@Izzy: I disagree with that, and Patashu has already made some deliberately understepped easy files.

[jimerax]

I think majority of files in the game are 8-10s already.

[dore]

Originally Posted by bmah

@dore: the inconsistency due to the level of judging experience (or lack of) is something I'm trying to tackle...it's one reason why I made a progress report, as well as providing resubmitted files an extra sort of distinction amongst the others. I have a number of files that weren't accepted that I suspect was due to this issue. If I actually can't find anything to fix, I'm going to send them completely unchanged, because it's not an issue on my end if it's due to judging incompetency. You can't completely eliminate the problem with this many judges, but I'm trying to reduce it.

I see where you're coming from. I think fundamentally where I disagree is the numerical approach, but that's probably more because I'm used to the Tass system. I don't know if having a few miniature Tasselfoots would be any better than a numerical approach, but I guess I feel like the judging-the-judges shouldn't be completely objective because that overlooks the fact that judging is inherently subjective and, especially with the discrepancies in experience between these judges, I think that it's difficult to really guess where each of the judges are in their judging without more information or more direct comparisons.

[bmah]

Yep. The objectivity introduced by a numerical system just makes decisions a bit easier, but clearly has its own shortcomings. I don't expect to find a perfect system, so it's going to have to be an imperfect one with some manual patch-up fixes to tie up loose ends.

[dore]

Yeah. If there's one thing that should be changed, I think there should be at least a little gray area somehow.

[jimerax]

Even though there're things that could be improved but I don't find this system any worse than old system (too much work for judges, final decision was biased).

Putting secondary judges for borderline files would make the judgement quality better, it will be more complicated and slower though.

[TC_Halogen]

11ELEVEN: [+.], [+.], [+.] (12)

My fault: should be 13. Putting all of that stuff together was a bit of a strain on the eyes, sorry about that mistake. :/

[qqwref]

Some minor suggestions about the grading scheme:

- If 12 is going to be a rough cutoff for acceptance, that means that a single [?] (or worse) means a file gets rejected. So there isn't really any point in having [-] or [--] except to be personally mean to someone.

- Similarly, if [++] and [+] are worth the same amount, there isn't really any point to having [++] except to be personally nice to someone. The judging criteria say that if someone gives a file [++] it will almost certainly get good ratings from other judges, but this doesn't matter because we can *see* ratings from other judges.

- OK, now we know that some judges don't seem to give out the mid-range grades as much as they should, plus the ratings look similar enough that it is possible to get confused on what exactly people mean. I think this might be due to confusion since they look similar. So I suggest changing the 6 grades (5/5 to 0/5) from this:
[+], [+.], [+?], [?], [-], [--]
to the more clear:
[++], [+], [?], [-], [-], [-].

[jimerax]

Now I think judgement rating should be 5/5 - 1/5 instead of +?- things, since it looks like half of judges aren't accustomed to current ones (and got more inconsistent because of that). Of course it would give a bad effect if some judges only use + and - because they don't know how to use intermediate ratings.. ok for judges who are used to old one though.

++ and -- are the same with + and - score-wise but we will take judge's intention at least.

[dore]

Another random idea:

In the old system, the difference between + and +. was the difference between queue and conditional queue, usually. I took the distinction between the two was that + means that judge wants that file queued as is, and +. means that judge would like small changes but overall it's acceptable as is.

To me, the distinction between + and +. is much smaller than that of +. and +?. To me, +? was a file that had potential but needed changes to be acceptable.

It seems to me, then, that + and +. could be the same point value for acceptance.

+ = 4 points
+. = 4 points
+? = 3 points
? = 2 points
- = 1 point

Then you could just have a hard cutoff at 11+ = accept, <11 = reject, with maybe the exception that two +s is an auto-accept, regardless of the point value (so that a +,+,- or +,+,? still goes through). The judges could then confer and decide the difference between queue and conditional queue depending on whether the judges' notes agree or not.

just throwin around ideas to debate

[Korygon]

I was thinking potentially something like

+ and ++=3
+.=2.5
+?=2
?=1
-=0

With 8 or more being accept, and <8=rejected

[TC_Halogen]

why use a decimal to complicate things?

Simply put: do a 5 point scale, three judges, 12 point acceptance minimum with discretion used on files that fall below 12 points due to a single judge's difference in opinion. Forget all of the symbol nonsense.

[who_cares973]

lets use a 1-1000 scale

[Patashu]

I think that a judge should grade two things about a file: How good it is now, and how much it would be improved by the fixes suggested in the notes. This would disambiguate whether +. is 'not the best file but about as good as you'll get for the song' and 'fun, but with some tweaks it would be perfect, so I want these tweaks done'

Alternatively, EVERY queue is a conditional queue until the stepcharter writes in and says yes/no to every suggested thing in the batch notes, nominating either the current .sm or a modified .sm as the final version.

[qqwref]

Originally Posted by jimerax

++ and -- are the same with + and - score-wise but we will take judge's intention at least.

Well, has any file that would have been rejected (due to point count) been accepted because it had a ++? (And -- is still meaningless because giving a file a - already automatically rejects it with the current system, even if the other judges love it.)

Originally Posted by Patashu

I think that a judge should grade two things about a file: How good it is now, and how much it would be improved by the fixes suggested in the notes. This would disambiguate whether +. is 'not the best file but about as good as you'll get for the song' and 'fun, but with some tweaks it would be perfect, so I want these tweaks done'

Great idea. I don't think we need a 5-point scale, because if you give something 2 or less the actual rating doesn't matter much (a 2 is not really more likely to get accepted than a 1 or 0), so I think it would be good enough to have four levels that are essentially 5/5, 4/5, 3/5, and 0/5. In that case:
[++]=great, [+]=good, [?]=average, [-]=bad
and then add one ! for each additional point you think the stepartist could improve the chart with recommendations. So [+!] would be a good file that could be great with some fixes.

Then a file might need (say) 12-13 of 15 points in this scheme to make the real queue, but if it got up to 13 points including !'s (as 1 point each) then it would make the conditional queue.

[jimerax]

I still think +. (light accept) and + (strong accept) should be separated.

Originally Posted by Patashu

I think that a judge should grade two things about a file: How good it is now, and how much it would be improved by the fixes suggested in the notes. This would disambiguate whether +. is 'not the best file but about as good as you'll get for the song' and 'fun, but with some tweaks it would be perfect, so I want these tweaks done'

Yeah I know about this "perfectly stepped mediocre song" vs "roughly stepped awesome song" discussion.
Both could be 4/5(+.), but I think the latter really needs tweaks even if it's accepted.

In addition to normal rating, [accept /fix] in the comment could be useful for those (and occasionally those files will be in conditional queue).

Originally Posted by qqwref

Well, has any file that would have been rejected (due to point count) been accepted because it had a ++? (And -- is still meaningless because giving a file a - already automatically rejects it with the current system, even if the other judges love it.)

(+, +, -) isn't always a reject.
also - and -- matter mainly for submission limit reduction, or temporary ban of submission.
If a file get 3 "-"s (-- is counted as 2), the submitter should get some penalty for the submission.

[Patashu]

I can't think of any penalty that would stick as they could just submit under a pseudonym, except for a downloadable 'wall of shame' where examples of what not to do are held.