Auto chart generator achieves 60% predicting accuracy on existing challenging+ charts

xxbidiao · 03-27-2017, 02:59 PM

https://arxiv.org/abs/1703.06891

Need more results to say whether this is outstanding, but extremely interesting.

**Untimely Friction** · 03-27-2017, 05:54 PM

Are you one of the contributors? Wheres 60% from and how does one do suectivity accurately, I'm curious of a little more before I go clicking around more, such as where you got that info. Im no neural network expert but 60% is only 10% more than 50 at which Id assume its still random 50/50, with a sample size that happened to be 60% "accurate", where other sets could be 30, 40, 80, random is random

Edit: tl;dr more info?

xxbidiao · 03-27-2017, 06:03 PM

Quote:

Originally Posted by Untimely Friction

Are you one of the contributors? Wheres 60% from and how does one do suectivity accurately, I'm curious of a little more before I go clicking around more, such as where you got that info. Im no neural network expert but 60% is only 10% more than 50 at which Id assume its still random 50/50, with a sample size that happened to be 60% "accurate", where other sets could be 30, 40, 80, random is random

Edit: tl;dr more info?

No, I'm not a contributor, but I happens to work on very similar topics. And well yeah, it depends on how you treat that 60% accuracy. This is just a preview and we may have to wait for a few months before all details are available.

**Untimely Friction** · 03-27-2017, 06:13 PM

Hmm now I need to figure out what to do with a .tex, I've been interested in convolutional and non neural networks but havent had a topic of interest to dive in, I'll be watching this thanks.

lurker · 03-28-2017, 01:04 AM

there's already a thread about this here
but the primary thing to take into account here is that this is specifically about pad play (ddr/itg/etc) and definitely not ffr

**Mourningfall** · 03-28-2017, 01:38 AM

Quote:

Originally Posted by lurker

this is specifically about pad play (ddr/itg/etc) and definitely not ffr

ie, not relevant and never going to have any idea of charting techniques that actually require a level of skill on keyboard to play.

it seems like a time-sink in all honesty, they should do something more constructive with their skills rather than designing something thats never going to reach full potential

like actually just make the charts themselves

DaBackpack · 03-28-2017, 02:14 AM

The main take-away here, as stated in the paper, isn't so much that "hey guys this is actually going to help DDR developers/artists", but rather "hello fellow scientists, here is a huge fucking corpus of annotated data, use this for your research purposes in designing intelligent music information retrieval systems"

Quote:

Originally Posted by Mourningfall

ie, not relevant and never going to have any idea of charting techniques that actually require a level of skill on keyboard to play.

it seems like a time-sink in all honesty, they should do something more constructive with their skills rather than designing something thats never going to reach full potential

like actually just make the charts themselves

there's no reason at all why this can't be applied to keyboard charts

the corpus they used was from ITG, so the techniques the LSTMs learned were about ITG. This research would theoretically be extensible to keyboard charts if they had a large enough corpus

and in the time it would take a human to produce a chart, the LSTM/CNN method would produce thousands of charts, if it had enough songs to work with

the tradeoff is that most of those charts would suck major ass, but this is far from useless tbh

EDIT: I don't really make FFR charts, and I'm assuming that there is no fundamental difference between charting for these different games, a difference that stops neural networks from learning patterns altogether

if there is, placing my foot in my mouth

Quote:

Originally Posted by Untimely Friction

Are you one of the contributors? Wheres 60% from and how does one do suectivity accurately, I'm curious of a little more before I go clicking around more, such as where you got that info. Im no neural network expert but 60% is only 10% more than 50 at which Id assume its still random 50/50, with a sample size that happened to be 60% "accurate", where other sets could be 30, 40, 80, random is random

Edit: tl;dr more info?

There are two steps that are explored here:

1) Given a song, where do we place the notes? (on this quarter notes, eighth note, etc)
2) Given a spot in the song to place notes, what notes do we place there? (Left, right, up, down, etc)

For the first one, you can expect a baseline of 50%: either one time-step in the song gets a note, or it doesn't. The 60% might not seem that impressive, but it IS better than the baseline, which is still pretty good.

For the second one, you can expect a baseline of 100/(C(4, 4) + C(4, 3) + C(4, 2) + C(4, 1)) = 100/15 = 6.66% per placement. (After all, each placement can have Left, Right, Up, Down, Left and Up, Left and Down, etc)

I might be misinterpreting something here because tired, so somebody can correct me if I'm wrong, but 50% for this part is pretty damn good.

**Mourningfall** · 03-28-2017, 09:21 AM

Quote:

Originally Posted by DaBackpack

EDIT: I don't really make FFR charts, and I'm assuming that there is no fundamental difference between charting for these different games, a difference that stops neural networks from learning patterns altogether

if there is, placing my foot in my mouth

Oh cool thanks.

**Dinglesberry** · 03-28-2017, 11:02 AM

Personally I want it to make really shitty minijack infested charts, they would be terrible but it would be fun to play

choof · 03-28-2017, 11:29 AM

Quote:

Originally Posted by DaBackpack

there's no reason at all why this can't be applied to keyboard charts

the corpus they used was from ITG, so the techniques the LSTMs learned were about ITG. This research would theoretically be extensible to keyboard charts if they had a large enough corpus

and in the time it would take a human to produce a chart, the LSTM/CNN method would produce thousands of charts, if it had enough songs to work with

the tradeoff is that most of those charts would suck major ass, but this is far from useless tbh

EDIT: I don't really make FFR charts, and I'm assuming that there is no fundamental difference between charting for these different games, a difference that stops neural networks from learning patterns altogether

if there is, placing my foot in my mouth

there are pretty large differences in making charts for pad and kb play
1 - pad patterning is somewhat restricted due to the physical aspect of the playstyle, and kb patterning is not
2 - patterning approaches are extremely varied in kb play and a fully functional tool would not only need to be technically accurate, but would also need to be artistic as well

I'm sure someone else can expand on these points further, I'm busy atm

this is a really neat idea but it would probably only ever work for 4k pad play.

DaBackpack · 03-28-2017, 12:52 PM

Quote:

Originally Posted by Mourningfall

Oh cool thanks.

It's going to be difficult to respond to this because I think there are some irreconcilable differences in opinions here, but I'm going to try anyway because I want to clarify some misconceptions

1) Research is iterative, and results like this do not mean that this is ready to replace human beings (spoiler alert, it will never be ready to replace human beings). What is promising here is that with this preliminary research shows strong results. And yes, these are strong results.

2) The paper clearly indicated that the method was actually better at predicting higher difficulty charts, which is at least in part a consequence of higher volume of steps.

3) "It's never gonna happen, machines lack imagination dude"
This really pisses me off, tbh, because it's so closed-minded. It's a common human arrogance that "humans are the only creative things out there." Can you define what creativity is? What imagination is? I strongly doubt that you can conjure a definition of either of those terms that is exclusively human-centric, because cognitive scientists and AI researchers have tried for years to classify, operationally, why humans are in monopoly of either of these things. This isn't really central to your argument, but I want to make clear that this particular statement is not as "obvious" or "trivial" as you are making it seem.

4) "But I'll take your word that 50%'s "pretty damn good." you've played these games enough to know that fucking up 1 in every 2 notes is no big deal."

The "50%" here is the ability of the system predict one particular step artist's charts for one particular song. I can see where you're coming from here, the ability to emulate the timing of one particular human being 50% of the time is not great for your purposes right now, but it is incredibly short-sighted of you to say that "it's never gonna happen", which is an expression you used multiple times. Refer to point 1.

5) @everyone who asked, I realize there are different stepping patterns between different games. What I'm asking is whether there is a particular reason why it is not possible to teach somebody how to step for keyboard games.

Presumably, there are a set of "rules" and "heuristics" for good tracks. People talk about good and bad tracks all the time. People here have some understanding of "when" and "where" to place steps. People can learn how to make good tracks. Ergo, there is a logic, or a set of logics, that human authors rely on when making tracks. This is what the system is learning. This is not different from DDR/ITG. It makes no fundamental difference that keyboard is more complex. (Until we get results for such an experiment, that is.)

It's like training for a 10k vs training for a marathon. The training process is not fundamentally different. The latter is harder and requires more resources and time, but you are still running.

It is not the case that this is the difference between training for a marathon and training for a hot-dog eating contest.

Anyway, sorry for the emotional response, but I feel an obligation to set the record straight about things

-- an AI PhD student

**DossarLX ODI** · 03-28-2017, 02:02 PM

DaBackpack, I think the short-sighted comments result from people not seeing the long-term potential for this project. Machine Learning and AI are rapidly growing fields and this project is an example of a program that recognizes patterns based on what it detects.

I can imagine this project being posted in an ITG players group and receiving judgmental responses about the program making "poorly patterned" or "bad" charts because that's the general attitude of that community -- if you don't do something a certain way, it's unplayable and sucks.

**rushyrulz** · 03-28-2017, 03:05 PM

Quote:

Originally Posted by DaBackpack

It is not the case that this is the difference between training for a marathon and training for a hot-dog eating contest.

I think this is actually the better analogy.

Being a god at ITG doesn't make you proficient at FFR/SM, and vice-versa.

DaBackpack · 03-28-2017, 03:28 PM

Quote:

Originally Posted by rushyrulz

I think this is actually the better analogy.

Being a god at ITG doesn't make you proficient at FFR/SM, and vice-versa.

This analogy was meant for the chart authoring process, not playing

a better analogy would be

"Writing in English and writing in French are different tasks, but the task -writing- are the same."

"Writing in English and shitting on the carpet are different tasks, and are not related."

**rushyrulz** · 03-28-2017, 04:03 PM

Ok. Being a god at charting for SM/FFR doesn't make you proficient at charting for ITG and vice-versa.

**Mourningfall** · 03-28-2017, 04:09 PM

Sorry to be short-sighted but this is like easter bunny, untill I see a giant talking rabbbit shitting chocolate, I'm never going to believe it exists. Sorry if that's ignorance, but ayyy I really do want this project to prove me wrong and rub it in my face for being unopen to new possibilities.

MinaciousGrace · 03-28-2017, 04:14 PM

Quote:

Originally Posted by DaBackpack

Presumably, there are a set of "rules" and "heuristics" for good tracks.

I don't want to get too far into this discussion- in part because I don't know much about pad charting. From what I understand pad charting has historically been more or less linear. If there is a prominent sound, there is a note. Want a harder chart? Step more sounds. Beyond this the major considerations are how the patterns choreograph the movement of the player during the chart.

The playing metagame enforces the charting meta and vice versa. I've heard recently that more what kb players would consider to be "technical" charts are being produced and played. If you don't know what I mean by that then I guess that sort of underscores one of the major disconnects between pad and kb charting.

Anyway the point is the decision making process in creating pad files tends to be less focused on a correlation between the music and the placed notes rather than a correlation between the placed notes and the playability thereof. So the machine learning process isn't learning how to express music. It's learning how to identify prominent peaks in waveforms and use them to create pad-playable patterns within a very restrictive scope.

Keyboard charting has its own set of concerns governing "playability", however as a general proposition the variety of patterns and types of patterns that are playable outstrips pad by a huge margin. Combine this with the fact that keyboard players are- by pure physical nature- able to hit more notes and you end up with a situation in which a huge number of decisions must be made concerning factors that pad charting doesn't even consider. It's easier to explain with an example.

In kb charting layering is the process by which multiple different sounds that fall on the same note are represented through chords. It is the driving force behind "jumpstream". Pitch relevancy (pr for short) is a generally observed guideline that as notes/tones ascend in pitch there should be a matching "rise" of the notes across the columns, so, 1234. Descending pitch should be represented as 4321. However we can't do this infinitely because we only have four columns. Suppose we have 8 notes placing a descending scale. We can wrap it around using 43214321. Large differentials in pitch can be represented by traversing more columns with each successive note. Instead of 4321 we might have 4213. You get the idea.

Now suppose we have a song with snares and bass hits alternating on 8ths with a 16th melody. We'll have double notes on 4ths and 8ths and all 16ths will be single notes. Snares tend to have higher pitches than bass hits so when layering onto the melody we'll prefer to place the associated note further to the right for snares and further to the left for the bass hits. Following pr in placing layered jumps doesn't exactly always follow this rule, though it often can and will. If you have a high note in the melody accompanied with a snare hit you'll probably place down a [34], and if you have a bass, a [14]. In this each note in the chord is placed according to its individual pitch relevancy respective to the previous 16th note, but you may also place chords according to the pitch relevancy of the chord relative to the last chord placed on the 8th.

Now, there's another problem, and this is where keyboard playability comes into play. There are very few melodies with accompanying percussion that will produce 100% strict pr adherence that do not form 16th jacks (44 for example). You can get this if you have an ascending melody on which you have a snare on an 8th, for example, 12[34]4. The ascending melody follows pr rules and the snare placement also follows pr rules, but following both rules results in something awkward and frequently avoided (but not always). Suppose we add a new rule that dictates we don't form 16th jacks within vanilla jumpstream. Moving the note to 1 would be the most compliant with pr.

Suppose the next note is a large jump in pitch, one that could easily justify moving three rows to the left. Now we have 12[34]14 and the issue is it plays the same as if the next note in the sequence were a reversal in the pitch change and slightly lower in pitch than the previous note. Strict adherence to PR would dictate we cross back across the columns and place the note on 4 as well. This would produce the same pattern despite two radically different melodies. Some artists would strictly adhere to pr, while others would forego pr rules and emphasize the pitch change by using 12[34]13, or a two column jump.

This is a stylistic distinction (a trait of the specific stepartist in question) that produces two different patterns that play differently and call attention to different aspects of the music. Is it more important to call the player's to the major pitch change? Is it more important to remain faithful to pr? If we call attention to the major pitch change will we detract from a similar technique used to highlight something else more interesting in the music? Which way will it play better? Can we teach a computer to appreciate subtle nuances in note placement based on what the stepartist intends to draw the player's attention to? Can we teach a computer how to make decisions on how to express what it hears (sees) in the music?

Is it worth asking these questions before we can programmatically determine the criterion upon which these decisions must be made? It needs to know what is a bass hit, what is a snare hit, how much the pitch in the melody changes. What if the percussion also varies in pitch, does it heed pr in the interest of staying faithful, does it ignore it in favor of a different set of rules that produces something more fun?. Can it decide whether to pr chords based on previous chords or individual notes within the chords?

Suppose intense vocals obscure parts of the melody or less affirmatively hit drums. Can it identify obscured bass drum hits? Obscured melody notes? If it can't, can we teach it to interpolate and assume there must be one there, and place a note anyway? Do we change the entire schema of the section to follow the vocals and not the melody and percussion? Can we teach a computer to do that?

Supposing we could use machine learning to teach a program to do all these things. Can we teach it to be consistent in its decision making across an entire chart? Can we teach it to make exceptions to its consistency? To make structured variations, and then to deviate from them? That's a lot of what music is, really. Deviation of variation. People have been trying to get computers to compose music for as long as computers have existed. Obviously they didn't have the benefit of machine learning and the modern computing power we have today, but we still haven't succeeded. Anyone familiar with music theory knows how formulaic, how mathematical it is. You can try to reduce it to saying it's just a bunch of rules governing what instruments get played at what points in time. We can all look at generic pop songs and think to ourselves that it could be spit out of a program, but we haven't been able to do that, because it's more than that- or at least, if you disagree with that sentiment, it's more rules than we've been able to handle thus far.

Most of what I've discussed concerns a singular example of a singular rule in keyboard charting which is heeded, partially heeded, or flat out ignored depending on the context of the chart and how it has progressed during construction, the charting style of the author, the difficulty and playability of the patterns it produces and or whether or not the stepartist just chooses to apply variation in charting technique within the same file, or variation of what specific sounds they are charting. This is all after the stepartist has made decisions on how they wish to stylistically approach a piece of music to begin with. If music is an expression, a chart is an expression of an interpretation of that expression. This is all before the stepartist has a chance to review their work, to playtest it, to make adjustments based on feedback. To gauge whether or not that feedback is good.

I'm not saying it's impossible to auto generate keyboard charts that could pass as human generated (or if that bar is too high, simply "worthy of being played"), and in fact I think that it is a possible eventuality, but the decision making tree involved in technical keyboard charting exponentially dwarfs that of pad charting (insofar as I understand the latter, in the context of keyboard dump files there is no debate). To equate the two is, in my opinion, naive. If I'm wrong and it is fair to equate the two, then the project is going to shortly run into insurmountable walls and observe logarithmic gains in accuracy over time as it has to learn and interleave an ever expanding set of rules and exceptions.

It may be within the narrow context of generating levels in a video game (interesting side thought, do you think it possible for a program to use machine learning, absorb every platformer game ever produced, and then use that information to auto generate levels for humans to play? For an entire platformer to have 100% autogenerated levels?) to go with music but you're still boiling down to the fundamental problem of "how do we get computers to think like humans?".

03-27-2017, 02:59 PM	#1
xxbidiao FFR Player Join Date: Dec 2012 Posts: 14	Auto chart generator achieves 60% predicting accuracy on existing challenging+ charts https://arxiv.org/abs/1703.06891 Need more results to say whether this is outstanding, but extremely interesting.

03-27-2017, 05:54 PM	#2
Untimely Friction D6 Challeneged Join Date: Aug 2012 Age: 31 Posts: 1,267	Re: Auto chart generator achieves 60% predicting accuracy on existing challenging+ ch Are you one of the contributors? Wheres 60% from and how does one do suectivity accurately, I'm curious of a little more before I go clicking around more, such as where you got that info. Im no neural network expert but 60% is only 10% more than 50 at which Id assume its still random 50/50, with a sample size that happened to be 60% "accurate", where other sets could be 30, 40, 80, random is random Edit: tl;dr more info? Last edited by Untimely Friction; 03-27-2017 at 05:54 PM..

03-27-2017, 06:13 PM	#4
Untimely Friction D6 Challeneged Join Date: Aug 2012 Age: 31 Posts: 1,267	Re: Auto chart generator achieves 60% predicting accuracy on existing challenging+ ch Hmm now I need to figure out what to do with a .tex, I've been interested in convolutional and non neural networks but havent had a topic of interest to dive in, I'll be watching this thanks.

03-28-2017, 01:04 AM	#5
lurker ur worst nitemare Join Date: Jan 2003 Age: 32 Posts: 1,626	Re: Auto chart generator achieves 60% predicting accuracy on existing challenging+ ch there's already a thread about this here but the primary thing to take into account here is that this is specifically about pad play (ddr/itg/etc) and definitely not ffr __________________ some feathery f*k Last edited by lurker; 03-28-2017 at 01:06 AM..*

03-28-2017, 11:02 AM	#9
Dinglesberry longing Join Date: Dec 2007 Location: Ontario, Canada Posts: 2,680	Re: Auto chart generator achieves 60% predicting accuracy on existing challenging+ ch Personally I want it to make really shitty minijack infested charts, they would be terrible but it would be fun to play

03-28-2017, 04:03 PM	#15
rushyrulz Digital Dancing! Join Date: Feb 2006 Location: 80 billion club, NE Age: 31 Posts: 12,980	Re: Auto chart generator achieves 60% predicting accuracy on existing challenging+ ch Ok. Being a god at charting for SM/FFR doesn't make you proficient at charting for ITG and vice-versa. __________________

03-28-2017, 04:09 PM	#16
Mourningfall ⇐⇓⇑⇒ Join Date: Jan 2011 Location: Paradise Posts: 1,627	Re: Auto chart generator achieves 60% predicting accuracy on existing challenging+ ch Sorry to be short-sighted but this is like easter bunny, untill I see a giant talking rabbbit shitting chocolate, I'm never going to believe it exists. Sorry if that's ignorance, but ayyy I really do want this project to prove me wrong and rub it in my face for being unopen to new possibilities.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)