Queue/Batch Discussion Thread v2

[Anaru]

Originally Posted by dag12

Or just be more rigorous with judging. Tougher judging means higher quality files and fewer number of submissions due to fewer SA points.

You mean, absolutely no chance for newb simfile artists to ever get a foothold in the FFR community?

New simfile artists = naturally worse files = less points = less submissions per batch = less motivation, etc..

[leonid]

An idea: keep the batch open all the time. Accept all submissions, put them into a judge queue, and periodically judge 50-ish (or less with a shorter period) files, first-come first-served.

[who_cares973]

I like Leonids idea

[nois-or-e]

//back-paged

Originally Posted by jimerax

- setting some all-through judges (judges who can judge all files).

I thought the idea of having a judge team was to avoid having a 'tass' that would be the be-all and end-all of what got into ffr?

IMO, regardless of how talented this person would be, there would be a distinction between what they like/don't like that will trickle into what files get into ffr and what don't.


However, if there are such people that would take on such a position (I know that rebirth/hi19 have expressed interest), perhaps a permanent judge team could be assigned to fast-track the judge system? For example, get hi19/rebirth/carlos & bmah to fast-track judging (only using names for the purpose of an example), so that for every 40 files that hit the batch email, they judge and write notes, send [x] amount of files to the queue and proceed to wait for the batch email to hit 40 submissions again. Have a 5th man to take the place of whichever of these 4 would happen to have a file of their own under judgement. Change the cap to a monthly basis, so that your current cap would = the amount of files you were allowed to submit per month.

Purely typing from the top of my head here, but a reasonable suggestion if there were people willing (and capable) to undertake such a task. We could potentially have a batch every few weeks with a week of judging, and slow down people that rush their charts for the sake of meeting a deadline, and raising the quality a fraction as a side effect.

[Anaru]

Originally Posted by leonid

An idea: keep the batch open all the time. Accept all submissions, put them into a judge queue, and periodically judge 50-ish (or less with a shorter period) files, first-come first-served.

The thing is, I think the judges would be overwhelmed by the number of files.

I'm sure MANY, MANY people have well over 10, heck, even 50 files ready for submission. I think the current system is fine, except that the batch needs to open more often. I mean, come on. who_cares973 has like 60 files in his folder. -_-;

Also, having a batch open all the time would drastically DROP the quality of files.
People realize that batches do not open frequently, and only get to submit X number of files.
Thus, they must put their utmost effort into stepping songs, otherwise their X submissions that they're given every X months is wasted.

If the batch was open all the time, people wouldn't feel as pressured to step well.

[who_cares973]

Lol out of those 60 wips only like 6 are for ffr

[leonid]

Originally Posted by Anaru

The thing is, I think the judges would be overwhelmed by the number of files.

I'm sure MANY, MANY people have well over 10, heck, even 50 files ready for submission. I think the current system is fine, except that the batch needs to open more often. I mean, come on. who_cares973 has like 60 files in his folder. -_-;

Also, having a batch open all the time would drastically DROP the quality of files.
People realize that batches do not open frequently, and only get to submit X number of files.
Thus, they must put their utmost effort into stepping songs, otherwise their X submissions that they're given every X months is wasted.

If the batch was open all the time, people wouldn't feel as pressured to step well.

There are always pros and cons; of course some tweaking is needed
One way to prevent submission spam is to restrict the number of files from the same person in the judge queue that haven't been judged yet: if your SAP is 4 and there are already four of your files in the judge queue, you are not allowed to submit before one of them gets judged.
Speaking of the pressure, I think people would feel more pressed if they have a deadline.

[iironiic]

I feel that the current judging system is extremely exclusive (break up to teams and judge a specific third of the batch). I know there is a new ten-point system proposed recently and I like that so far. I think to further make this system effective, each judge should make contact with each other about the files they are all judging (not simultaneously). That way, this method can avoid skewed ratings and contradictory commentaries on a particular file. It will also enlighten judges to view files differently and to consider that as they give a more accurate and deserved rating.

Downside: time commitment. Everyone, including myself, is busy these days haha.

EDIT: This might be a second silly idea, but to give judges more incentive to judge files seriously, maybe implement a judgment rating for each person? If someone is consistently giving incorrect ratings on a file, lower the rating to the point where JX can fire Judge A. Just a thought.

With this idea, I think anyone who is interested in judging can take advantage of this to see if judging is the right thing for them. After all, we are in a high demand for judges.

[Rebirth0]

My suggestion:

At the start of every month, open submissions for the first half of the month. 15 days. After 15 days, close it, and judges take the next two weeks to judge and submit notes. There can be two sets of 3 judges. One set judges files one month, the other does the next month. Everybody only does work 2 out of 8 weeks. If someone wants to go through every batch as an additional judge, let them.

4 or 5 songs per week all released on the same day, including one new skill token or regular token every week. I will even come up with them if I have to. It isn't hard. The token can be released every Sunday while everything else can be released some other day.

Submission caps can stay the same or become smaller, depending on how these submissions would go. If the quality of files gets lower with more submission time, then set a hard limit to what is ACCEPTED, not SUBMITTED. 30%? 20%? It shouldn't be a number, it should be a percentage if there is a limit at all.

I am ready to start working with this site again as whatever needed. As a judge, absolutely. FFR has well over 1.5 million users so obviously, with a huge community comes a huge workload. Deal with it. What is happening now isn't making anyone happy in the long run. We want people who will stick around and contribute eventually.

[00Razor00]

Rebirth0, the voice of reason.

[qqwref]

I feel like having a batch open for only a few days encourages people to rush files and get them in as quickly as possible, rather than having people take their time to get a good chart they are happy with. (This is bad because people rarely fix up their charts.) Of course, as we've seen, it also prevents certain people from sending in files meant for FFR just because they are too late, thus leading to people being unhappy with the entire system. I'd be in favor of leonid's approach, or else, just leaving the batch up for a set amount of time (a week, say) and not worrying about how many files are in it. If one person can judge 25 or so files in one day, I'm sure they will be able to handle 50 or more files in their group, given that they have a few weeks to get it done.

But hey, do whatever you want. I'm not the one in charge.

[bmah]

I can't review 25 files in a day for the work schedule I have.

[hi19hi19]

Just saying that if leonid's system was put in place with no checks, I would be very happy.
And the judges would be very overloaded.

Honestly, as happy as I would be, I do think it needs -some- restrictions if just for the crazies like me.

[i love you]

Originally Posted by qqwref

I feel like having a batch open for only a few days encourages people to rush files and get them in as quickly as possible, rather than having people take their time to get a good chart they are happy with. (This is bad because people rarely fix up their charts.) Of course, as we've seen, it also prevents certain people from sending in files meant for FFR just because they are too late, thus leading to people being unhappy with the entire system. I'd be in favor of leonid's approach, or else, just leaving the batch up for a set amount of time (a week, say) and not worrying about how many files are in it. If one person can judge 25 or so files in one day, I'm sure they will be able to handle 50 or more files in their group, given that they have a few weeks to get it done.

But hey, do whatever you want. I'm not the one in charge.

I have to agree with this. Not much to say really.

[Anaru]

leonid's post makes good sense, but we only have so many judges..

Even with the current system of reviewing files, it takes absolutely forever for batches to get judged. :/ But reviewing files on a weekly basis (who knows how many files they'll get.. 50? 100?) might be overwhelming.


Or, nominate some more judges.

[hi19hi19]

Hmm, batch pic?
I realize I'm being that guy, but hey someone's gotta be him

[who_cares973]

Originally Posted by Anaru

leonid's post makes good sense, but we only have so many judges..

Even with the current system of reviewing files, it takes absolutely forever for batches to get judged. :/ But reviewing files on a weekly basis (who knows how many files they'll get.. 50? 100?) might be overwhelming.


Or, nominate some more judges.

i really doubt there would be 50+ files in the batch every month. maybe for the first month or two but after that it would slow down especially with a submission cap like leonid said

[Anaru]

Originally Posted by who_cares973

i really doubt there would be 50+ files in the batch every month. maybe for the first month or two but after that it would slow down especially with a submission cap like leonid said

Really? I'm pretty sure there'd be more than 50 people submitting files every month.. I mean, we have 1.7 million registered users right? (Even if there's only 100k active or whatever..)

[bmah]

Most people on this site are players, not steppers.

[jimerax]

FYI, judgement process should be at "sustainable" pace. A system that lasts for only a few months is useless.

By "batch open all the time" syetem, it will be complicated to manage and we don't have a method for controling submission rate.

Judges who have time and judgement speed can judge files in all groups (If we have 50 files/group and 3 groups, 150), and Judges who don't can only judge files in one group (50). This is what I wanted to say.

ps batch pic.
some files were removed due to submission cap, or timing. 106 files, still.