Pop Music Transformer: Generating Music with Rhythm and Harmony

Paper (arXiv):  https://arxiv.org/abs/2002.00212

Code (GitHub):  https://github.com/YatingMusic/remi


We’ve developed Pop Music Transformer, a deep learning model that can generate pieces of Pop piano music of several minutes.  Unlike existing models for music composition, our model learns to compose music over a metrical structure defined in terms of bars, beats, and sub-beats.  As a result, our model can generate music with more salient and consistent rhythmic structure.


Here are nine pieces of piano performances generated by our model in three different styles.  While generating the music, our model takes no human input (e.g., prompt or chord progressions) at all.  Moreover, no post-processing steps are needed to refine the generated music.  The model learns to generate expressive and coherent music automatically.



From a technical point of view, the major improvement we have made is to invent and employ a new approach to represent musical content.  The new representation, called REMI (REvamped MIDI-derived events), provides a deep learning model more contextual information for modeling music than existing MIDI-like representation.   The new model can generate music with explicit harmonic and rhythmic structure, while allowing for expressive rhythmic freedom in music (e.g., tempo rubato).  While it can generate chord events and tempo changes events on its own, it also provides a mechanism for human users to control and manipulate the chord progression and local tempo of the music being generated as they wish.


The above figures show the piano-rolls of the generated piano music when we enforce some control signals over the chord and tempo. Figure (b) shows that the model selects chords that are harmonically close to F:minor when we prohibit it from generating F:minor. Figure (c) shows controlling the tempo events.


You can find more details in our paper (see the link below the title) and try the model yourself with the code we’ve released!  We provide not only the source code but also the pre-trained model for developers to play with.


More examples of the generated music can be found at:



Enjoy listening!


By: Yu-Siang Huang, Chung-Yang Wang, Wen-Yi Hsiao, Yin-Cheng Yeh and Yi-Hsuan Yang (Yating Music Team, Taiwan AI Labs)