Piano Genie

When playing you decide the timing of the notes and have some control over the “shape” of the melody: whether it should move up or down in pitch, and by how much. The Piano Genie AI will assist you by choosing which piano keys to play to make you sound more like a professional player.

Piano Genie was programmed without coding in any theoretical rules about harmony or composition. Its AI is based on a neural network that was trained with 1400 performances from the International Piano e-Competition. We can say that through this training it learned on its own, and by example, what piano music should sound like.

How it works

Neural Networks are AI systems inspired by biological brains. Similar to how groups of neurons work, they take an input (e.g. signals from the nervous system) and produce an appropriate output (e.g. activate a muscular response). The network learns to give “intelligent” responses through a training process: it receives many examples of input, has its performance evaluated on each, and then is given feedback so it can improve over time.

For a long time most AI systems required the programmer to precisely describe how to perform a task to the computer. Neural networks, on the other hand, are commonly used for tasks where describing the decision making process is not possible because of its complexity. They’re used for a very large variety of tasks such as image and voice recognition, playing games, developing drugs and vaccines, driving cars, or even creating music.

The Piano Genie AI was built by first creating two separate neural networks: an encoder and a decoder. The encoder receives a melody (a sequence of notes) represented as the numbers 1-88 (the keys of the piano) as input and produces a sequence with the same length but using only the numbers 1 through 8. The decoder reverses the process by taking this simplified sequence and outputting a melody that once again uses the different 88 notes a piano can produce. By connecting the encoder and decoder together the melody passes through a sort of funnel.

Of course it’s not always possible to perfectly recover the original melody once it passes through the encoder and decoder. With a full piano each note of a melody can be any of 88 available pitches. When the notes of a melody are encoded using only 8 symbols it means that after each note there are only eight possible ways to continue. We can decode it back to something that hopefully sounds similar, but will probably not be the same.

There’s not a unique way to encode the notes using 8 different values so they can later be decoded back and the creators of Piano Genie didn’t know which one was the best. This is the role of AI: by training them, the neural networks can find the best way to do this conversion.

Both the encoder and decoder networks in Piano Genie were trained simultaneously using the 1400 piano performances mentioned. The networks were evaluated by looking at how closely the output of the decoder matched the original melody. The encoder was also evaluated based on how similar the “shape” of the 8 note sequence was to the original (for instance, if the notes in the melody went up and then down, the numbers of the encoded sequence should go up and then down as well).

Once the encoder and decoder learned to perform both tasks well enough, the authors of Piano Genie took the decoder (by itself) and connected its input to the 8 colored on-screen buttons. When you play on Piano Genie’s keyboard, the decoder receives your sequence of key presses and tries to decode it into a piano melody. The decoder network is attempting to reconstruct a piano improvisation that actually never took place.

Why it works

We wouldn’t consider every possible combination of piano notes musical, just like not every random combination of letters is a sentence. Music has structure: it follows patterns, scales, motifs. When the encoder and decoder are trained, they learn how to compress music (that is, express it using a smaller amount of symbols) by finding the common structure and patterns in the examples we give them. They work with the musically interesting sequences and ignore the arbitrary ones.

When the decoder neural network is connected to the 8 Piano Genie buttons, it produces note sequences that share structure to the ones in the songs it was trained on.

Further experiments

Check out the settings button in Piano Genie. You’ll find options for using MIDI instruments for input or output. There’s also an option for using a Makey Makey kit to play the piano using everyday objects.

The rock band The Flaming Lips used Piano Genie on its live shows to play the piano by touching fruit in a bowl.

Some things to think about

• Could you recognize, just by listening to it, that a piece of music was created with the assistance of AI?
• Is something art because of how we respond to it, or because of the artistic intention of its creator?
• Piano Genie creates music similar to the performances it was trained on. If an AI created music that sounds nothing like what we know, would we accept it? Can the boundaries of art be pushed only by humans?

Links

The official Piano Genie page at Google Magenta
The original Piano Genie paper
The Piano Genie source code

  Text is available under the Creative Commons Attribution License.