Batch judging suggestion

[leonid]

How about we follow the Official Rubik's Cube competition rule (Link Check 9f8)

Here's how it works:
Step 1: We want the average score of N judges on each submission.
Step 2: Get N+2 judges rate the batch.
Step 3: For each submission, omit the highest and the lowest scores and compute the average of the remaining scores.

Pros:
- Less bias
- Less drama (If a submission gets a low score with this method, it means that more than one judge gave a low score, thus preventing typical "omg my fiel almost made it through if it weren't for <Insert a name here>" drama)

Cons:
- Needs more judges

Thoughts?

[ninjaKIWI]

Fantastic idea reonid (o:

[jimerax]

We did a similar thing on hard song batch, and it was my first trial.

N = 5 required to do it properly, at least (pretty hard with group judgement).

[ninjaKIWI]

Might I propose ditching group judging, but at the same time to make up for it, maybe implement a batch total file limit?

[stargroup100]

Back at KBO, what we did was we outlined what we wanted to see in the game very very specifically, so when a dispute did occur, we could almost "objectively" determine whether the reasons stated were valid.

I don't know how you guys do your judging, but usually a disagreement within the judges is due to different standards. Standardizing what the you want to see in the game might eliminate a lot of the drama associated with the judging.

[qqwref]

Originally Posted by stargroup100

I don't know how you guys do your judging, but usually a disagreement within the judges is due to different standards. Standardizing what the you want to see in the game might eliminate a lot of the drama associated with the judging.

Yeah, this. I've tried to tune my judging standards to FFR - comparing to the kind of files in FFR rather than the kind I see in SM, considering comboability when I normally wouldn't, putting less emphasis on careful sync because of 30 fps, and so on. I think putting something vaguely like this in writing would help reduce feelings of unfairness, as a low grade would specifically mean "this chart is not suitable for FFR" rather than the "I don't like this file/song/cut/stepartist/difficulty" that we sometimes see now.

[rushyrulz]

this has been suggested before and the main argument was that 2 judges' notes were getting thrown out for nothing. Boo hoo, this would be an excellent system if judges weren't so stubborn.

[hi19hi19]

Originally Posted by rushyrulz

this has been suggested before and the main argument was that 2 judges' notes were getting thrown out for nothing. Boo hoo, this would be an excellent system if judges weren't so stubborn.

The notes aren't getting thrown out, the ratings are. The notes will still there and people can see the most glowing praise and the most scathing review in text, but the ratings wouldn't count.
idk why the judges would care to be honest. I wouldn't...

[foilman8805]

Sometimes you're statistically relevant, other times you're not -- and if you take it from that perspective, the rating is still important because you don't know when or where you'll be relevant/irrelevant. Though the sample size of judges is small, this is effectively eliminating outliers, which is a very common statistical practice. It doesn't nullify a review as it's still intact with the judge's thoughts/concerns and the artist gets to read them (like hi19 said). A pool of 5 judges is the bare minimum to make any kind of statistical judging worthwhile, imo. If at all possible, the number of judges should be increased.

[stargroup100]

This isn't the World Series of FFR Charting, you don't really need 13095 judges to figure out whether a chart is appropriate for a game.

[who_cares973]

Lets make a world series of ffr charting

I liked the turnout for the hard batch judging method and I think we should stick to it

[stargroup100]

It's just that I find it ironic that you guys take this so seriously and yet you don't take it seriously.

Like, if you're serious about judging enough that you have to have a minimum N number of judges with organized comments and a point/qualification system, it seems to make sense that you really should standardize your criteria and objectively figure out what you're aiming for in this game.

[ScarletSky]

Ummmmm yea, I'ma stay outa this one.

[who_cares973]

The thing is St no one wants to hurt anyone's feeling. People want the batch to be open to anyone and any kind of file. The problem with this though is that people use their own criteria which is sometimes higher than others which leads to the judgements we see. I agree though there should be something to go off of it would resolve a lot of the problems. More on this when I'm not on my phone

[foilman8805]

Originally Posted by stargroup100

This isn't the World Series of FFR Charting, you don't really need 13095 judges to figure out whether a chart is appropriate for a game.

Don't be a ****ing shithead. You know what I'm trying to say. 2 or 3 more judges is adequate.

[who_cares973]

The problem is finding those qualified to judge

[rushyrulz]

cross post from batch discussion thread in march

Originally Posted by rushyrulz

I do think that subjectivity plays a bigger role in the judging process than just the fun factor. If a file is technically correct with halfway decent PR there is no way in hell it should get a difference between the highest and lowest grades, greater than 2. I can see someone getting 5/x/3/x, but 5/x/2/x?? Come on, something isn't right here. 5/5/1/5 is out of the question, One judge shouldn't have the power to TRUMP reject all the other judges in the team, that's just insane.

When I see these score breakdowns, it sometimes leads me to believe that either some of the judges are unqualified to judge, or unable to judge with objectivity. Sure different judges have different opinions on the 'no-no' level of some sections of files, but I sometimes think that giving a rating of 2 solely because of a slight sync problem is a tad overboard. You've trump rejected a file because someone didn't hit F10/F11 enough.

So my honest opinion? Yes this system needs an overwork. Since we only have 12 judge slots, I don't think doing 2 teams would be the best option because of the overload of files to judge. 4 judge teams of 3 would give each judge even more trump power than they have currently. The two alternatives I have in mind would be:

a. Get more judges

This might not be a possibility seeing as how it's hard enough as is to get everyone to finish on time. (see: Patashu) Although I will admit, the current judge team is doing a really good job of getting notes done on time.

The more probable alternative would be:

b. Toss out the oddball score and make an accept 12 or 13/15

4 judges will still be in a team, just the rating that is most different from the others will be tossed out, resulting in a 15 point system. For example:

kommisar works really hard and makes the most badass file FFR has ever seen. The batch notes come out and he gets [++] [+] [+.] [?]. Under the current judge system, this file would not get queued because it only totals to 16/20. BUT if you toss out the odd 2/5 score, the total would be 14/15; A passing score. A different example:

Dr4g0nSly40r347 throws some piece of hindu throat singing crap together in half an hour. The notes come out and he gets [-] [?] [?] [+.]. While this would be rejected under both systems, the odd +. would be thrown out making for a more accurate depiction of how much of a piece of crap the file actually was. (9/20 vs. 5/15)




TL;DR: Current system gives too much power to the individual judge, we should throw out the most odd score to make a 'majority' type decision rather than a unanimous one.

[ScarletSky]

Problem I see with that idea then becomes: why have that extra judge there to begin with. I'm sure he/she doesn't want to feel like just because they felt strongly (justified or not) about a file, that they're opinion doesn't matter anymore just because of that.

Edit~
Plus, it sounds eerily similar to the OP's post. But instead of singling out 2 judges, it's just one.

[who_cares973]

That shouldn't matter. The review wouldn't deleted or ignored. People would need to get over themselves if this is the case. I think most of the judges would be on board with the new judging system that jx created

[ScarletSky]

I like the new system JX is trying to implement myself, but I do agree that for it to work efficiently we need more qualified judges to help as well.