Dance Dance Convolution : Input raw audio - output simfiles

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • tunsz555
    FFR Player
    • Apr 2009
    • 117

    #1

    Dance Dance Convolution : Input raw audio - output simfiles

    A recent paper named Dance Dance Convolution is out. You might be interested on reading the whole thing here if you like neural nets : https://arxiv.org/pdf/1703.06891.pdf.



    But for short, neural nets is a thing which can decide based on what it had learned. (We designed the net and let it learn, but we have no idea what happens inside that it is able to deduce correct answer given a new input.)

    This NN will automatically detects where to put steps (analyzing "onset" in an audio file) on an unseen audio. (Shazam for example, could detect what song it is because it already learned about the song, that's not unseen)

    The learning material is 133 ITG files with full difficulties, 8 different note charters.

    Then it will decide which note to put at that point in time. The resulting simfile is in this video.



    Luckily it is not that intelligent to do crossovers or gallops XD
    Last edited by tunsz555; 03-23-2017, 08:23 PM.
    I'm actually a Sargon/5argon
  • blanky!
    Banned
    • Jan 2015
    • 164

    #2
    Re: Dance Dance Convolution : Input raw audio - output simfiles

    This
    Might be very sick


    In all honesty this is probably something we've all thought about but never really taken into serious consideration because of the sheer complexity in making such a program. But it's here now!

    Comment

    • blanky!
      Banned
      • Jan 2015
      • 164

      #3
      Re: Dance Dance Convolution : Input raw audio - output simfiles

      From the website:
      "The network that predicts step timings has no concept of rhythm or tempo. It simply answers the following question 100 times a second: should there be a step here? We map these to step charts by creating measures with 192 steps at 125BPM. We will release a script soon allowing you to manually set the tempo/offset of a chart to clean things up a little. For now, turn off colored note skins to avoid confusion."

      It looks like the remnants of FFR's antiquated system have washed out on the shores of San Diego.

      Comment

      • Wayward Vagabond
        Confirmed Heartbreaker
        FFR Simfile Author
        • Jul 2012
        • 5866

        #4
        Re: Dance Dance Convolution : Input raw audio - output simfiles

        this is neat kind of reminds me of dancing monkey

        Comment

        • DaBackpack
          ~ お ま ん こ ~
          • Mar 2014
          • 918

          #5
          Re: Dance Dance Convolution : Input raw audio - output simfiles

          my adviser sent this to me for lab research today lmao

          ~ S C I E N C E ~


          Originally posted by Moogy
          no one cares
          Originally posted by TWG Dan Hedgehog
          there are 743 matches for hedgehog suicide on deviantart
          that's kind of a sad statistic

          Comment

          • FightABunny
            FFR Player
            • Mar 2004
            • 51

            #6
            Re: Dance Dance Convolution : Input raw audio - output simfiles

            Looks like there is a working demo of Dance Dance Convolution out there!!

            Last edited by FightABunny; 03-23-2017, 01:55 AM.

            Comment

            • lurker
              ur worst nitemare
              FFR Simfile Author
              • Jan 2003
              • 1628

              #7
              Re: Dance Dance Convolution : Input raw audio - output simfiles

              hello duderino i am taking a class on stuff like this atm and i have a couple friends who are working on something similar
              beat detection is pretty decent and the patterns chosen are generally quite playable on pad (though on the song i tested i ended up with one 175 bpm 16th note minijack and a 32nd triplet)
              Last edited by lurker; 03-23-2017, 01:55 AM.
              some feathery f**k

              Comment

              • RenegadeLucien
                FFR Veteran
                • Jan 2016
                • 283

                #8
                Re: Dance Dance Convolution : Input raw audio - output simfiles

                I feel like this could be massively improved just by allowing the user to input the song's BPM if it is known, rather than try to shove the song through 125BPM which would inevitably cause nearly every arrow to be a few ms offbeat


                Comment

                • MinaciousGrace
                  FFR Player
                  • Dec 2007
                  • 4278

                  #9
                  this is actually a fairly neat and well written paper and if i cared more about pad i'd prolly take the time to scrutinize it further

                  just want to point out that this paper defines an increase in difficulty as a successive increase in notes per second relating to a successive decrease in sound audibility

                  ------

                  after throwing ghost love score at it a few times i am more impressed by the paper than the results

                  though i supposed it was designed for stepmanias musics and not, you know, real music

                  ------

                  ok you know what i bet i could make a reasonably playable dump file by throwing a song at it 10 times and overlapping the results
                  Last edited by Velocity; 03-23-2017, 08:04 AM.

                  Comment

                  • contrapatior
                    Banned
                    • Sep 2016
                    • 191

                    #10
                    Re: Dance Dance Convolution : Input raw audio - output simfiles



                    Comment

                    • shenjoku
                      Wubalubadubdub
                      • May 2005
                      • 1697

                      #11
                      Re: Dance Dance Convolution : Input raw audio - output simfiles

                      Originally posted by contrapatior
                      true art
                      Beautiful :')
                      boop

                      Comment

                      • lurker
                        ur worst nitemare
                        FFR Simfile Author
                        • Jan 2003
                        • 1628

                        #12
                        Re: Dance Dance Convolution : Input raw audio - output simfiles

                        Originally posted by MinaciousGrace
                        just want to point out that this paper defines an increase in difficulty as a successive increase in notes per second relating to a successive decrease in sound audibility
                        the first part (linking difficulty to nps) is a lot more accurate for pad charts anyway but the music connection is ??

                        Originally posted by MinaciousGrace
                        though i supposed it was designed for stepmanias musics and not, you know, real music
                        this is more a side effect of peak detection working far better on sounds with sharp attacks, as i've found that it tends to miss notes in synths quite often
                        Last edited by lurker; 03-23-2017, 05:20 PM.
                        some feathery f**k

                        Comment

                        Working...